Preflight

Webkit
If using Google Chrome, you will likely need the Dev channel to see all of the functionality in this presentation. If you are using Safari, you will likely need a nightly build of WebKit in order to see all of the functionality in this presentation.
Mozilla
You are running a Mozilla browser. While such browsers generally have excellent support for HTML5 features, this presentation has only been tested using WebKit browsers such as Google Chrome or Safari. You should still be able to navigate the slides by using left/right arrow keys, but will currently see display errors with regard to the 3d rendering of the slides and some demo content.
Other browser
You are running a browser that has not been tested with this presentation. You may not be able to run some or all of the samples listed here. While we want to add support for as many browsers as possible, currently we only support WebKit-based browsers such as Google Chrome or Safari.
Media Source API: supported
Media Source API: not supported
Your browser does not support the Media Source API. If using Google Chrome 17+, you need to enable the API in about:flags.
If things look good, press → to move on.
Welcome! (This field is for presenter notes and commentary.)
Press:
  • Space or ← / → to move around
  • Ctrl/Command / – or + to zoom in and out if slides don’t fit
  • N to show/hide speaker notes
  • H to highlight important elements in code snippets

HTML5 Video Specifications

Silvia Pfeiffer
Drupal Down Under
15th January 2012
10:45 - 11:30am

Slide Template by HTML5 WOW
Who am I?

@silviapfeiffer
  • W3C and WHATWG video standards
  • HTML5 Accessibility Task Force
  • Google video accessibility contractor
  • Mozilla video accessibility contractor
  • Video research at CSIRO
What we'll be looking at...
  • 1. Video/Audio Element
    • Fallback
    • Encoding
    • Media Element JS API
    • Custom Controls
  • 2. Drupal Modules
    • VideoJS
    • JW Player
    • MediaFront
  • 3. Video Accessibility
    • WebVTT and Captions
    • Audio Descriptions
    • Sign Language
    • Transcripts
  • 4. WebRTC
    • Live video: HTTP adaptive streaming
    • Device input: MediaStream
    • Communication: PeerConnection

Video/Audio Element

How to publish an audio file?
<audio controls>
  <source src="nyan_cat.ogg" type="audio/ogg">
  <source src="nyan_cat.mp3" type="audio/mp3">
</audio>
How to publish a video file?
<video controls poster="nyan_cat.png">
  <source src="nyan_cat.mp4" type="video/mp4">
  <source src="nyan_cat.webm" type="video/webm">
</video>
Video files work in <audio> also
<audio controls poster="nyan_cat.png">
  <source src="nyan_cat.mp4" type="video/mp4">
  <source src="nyan_cat.webm" type="video/webm">
</audio>
How to fall back gracefully?
<video controls poster="nyan_cat.png">
  <source src="nyan_cat.mp4" type="video/mp4">
  <source src="nyan_cat.webm" type="video/webm">
  <p>Download Nyan Cat video from here:</p>
  <ul>
    <li><a href="nyan_cat.webm">WebM format></li>
    <li><a href="nyan_cat.mp4">MP4 format></li>
  </ul>
</video>
How to fall back to Flash or other plugin?
<video controls poster="nyan_cat.png">
  <source src="nyan_cat.mp4" type="video/mp4">
  <source src="nyan_cat.webm" type="video/webm">
    <object classid='clsid:D27CDB6E-AE6D-11cf-96B8-444553540000'>
      <param name='movie'           value='player.swf'>
      <param name='allowfullscreen' value='true'>
      <param name='flashvars'       value='file=video.mp4'>
      <embed src='player.swf'
             allowscriptaccess='always'
             allowfullscreen='true'
             flashvars="file=video.mp4"
      />
    </object>
</video>
How to encode video / audio for HTML5?
Use FFMpeg (command-line) or Miro Video Converter (GUI) to transcode.
  • Video formats:
    • Safari & IE9: MPEG-4
    • Chrome, Opera & FF: WebM / Theora
  • Audio formats:
    • Safari & IE9: MP3, AAC, (WAV)
    • Chrome, Opera & FF: Vorbis, WAV
Transcoding examples
# MP4 -> WebM
ffmpeg -i video.mp4 video.webm

# MP3 -> Ogg Vorbis
ffmpeg -acodec vorbis -i audio.mp3 audio.ogg

# WebM -> Ogg Vorbis
ffmpeg -vn -acodec vorbis -i video.webm audio.ogg
Media and the JavaScript API
video = document.getElementsByTagName("video")[0];

// functions            // catching events
video.load();           onloadstart
video.play();           onplay
video.pause();          onpause

// reading properties
video.currentSrc
video.currentTime
video.duration
video.volume
video.muted
Media and the JavaScript API
Custom player using JS and CSS

Drupal HTML5 video modules

VideoJS by zencoder
Home Page: http://videojs.com/
Drupal Page: http://drupal.org/project/videojs
Sponsored by: heidisoft
Contains:
  • embed code
  • JS library: video.js
  • CSS skin: video-js.css
  • Flowplayer as fallback
Works with video module for upload, transcoding, thumbnailing: http://drupal.org/project/video
Video intro: http://www.modulesunraveled.com/video
JW Player by Jeroen Wijering
Home Page: http://www.longtailvideo.com/players/jw-flv-player/
Drupal Page: http://drupal.org/node/1213496
Contains:
  • JW embedder
  • Uses SWF tools
  • JS library: jwplayer.js
  • CSS skin: video-js.css
  • provides caption support
Works with video module for upload, transcoding, thumbnailing: http://drupal.org/project/video
MediaFront
Home Page: http://www.mediafront.org/
Drupal Page: http://drupal.org/project/mediafront
Sponsored by: Alethia Inc.
Contains:
  • uses OSMplayer for flash fallback
  • uses JQuery
  • uses JQuery UI ThemeRoller
Documentation: http://www.mediafront.org/documentation

Video Accessibility

Accessibility requirements of
native transport controls
  • Keyboard accessible
    • tab accessible play, transport bar and volume buttons
    • space bar for play/pause toggle
    • left/right arrow for navigation
    • up/down arrow for volume
  • Screenreader accessible
    • labels on buttons
player controls with play, transport bar, and volume button highlighted

Opera: keyboard control and tab access

Native controls in other browsers
  • IE9 and Firefox
    • Keyboard control
    • No tab access
    • No screenreader exposure
  • Chrome and Safari
    • No keyboard access to controls
    • No tab access
    • No screenreader exposure
Accessible custom controls: YouTube's HTML5 player

Join http://www.youtube.com/html5


HTML5 introduces markup for captions and subtitles
<video controls poster="video.png">
  <source src="video.mp4" type="video/mp4">
  <source src="video.webm" type="video/webm">

  <track kind="captions"  srclang="en" src="video_cc_en.vtt">

  <track kind="subtitles" srclang="de" src="video_sub_de.vtt">

</video>

Associate caption files with a video through markup.

HTML5 introduces WebVTT for caption authoring

New caption file format

Example: arduino-en.vtt

WEBVTT

1
00:00:13.000 --> 00:00:16.100
I heard about this <c.red.caps>arduino</c> project,
and I saw it online - 

2
00:00:16.100 --> 00:00:20.100
- and I said '<b>Wow!</b> a lot of people are starting to talk about this.
I should <i>check it out</i>!'
WebVTT in HTML5

Polyfill Captionator.JS by Christopher Giffard
WebVTT features
  • text formatting
  • viewport positioning
  • internationalization
  • CSS styling override ::cue
  • paint-on captions

More in WebVTT talk

WebVTT captions:
Status of implementation in browsers
  • IE10 <track> preview released
  • WebKit / Chrome dev are testing <track> element
  • Firefox and Opera working on it
  • JavaScript polyfill as bridging solution: http://captionatorjs.com/
  • WebVTT work: W3C Text Tracks Community Group
Markup for descriptions in text
<video controls poster="video.png">
  <source src="video.mp4" type="video/mp4">
  <source src="video.webm" type="video/webm">

  <track kind="descriptions"  srclang="en" src="video_desc_en.vtt">

</video>
Using WebVTT for description authoring

video_audesc_en.vtt

WEBVTT

1
00:00:00.000 --> 00:00:05.000
The orange open movie project presents

2
00:00:05.010 --> 00:0:12.000
Introductory titles are showing on the background of a water pool with fishes
swimming and mechanical objects lying on a stone floor.

3
00:00:12.010 --> 00:00:14.800
title: elephants dream
WebVTT descriptions in HTML5
WebVTT descriptions: Status of Implementation
  • No native browser implementations yet
  • Browser accessibility APIs: aria-live
  • Video pausing required for extended cues
Other Accessibility Features in Development
  • DAISY style video navigation
  • Full text transcripts
  • Sign Language video tracks
  • Audio Description audio tracks

WebRTC: Video Conferencing on the Web

HTTP Adaptive Streaming

What is HTTP Adaptive Streaming?

Can be used for live streaming.

Safari supports: HLS for MPEG-4 in M3U8 manifests

MPEG proposed DASH manifest format

HTTP Adaptive Streaming for Web Developers

Proposal by Google

Extends the audio and video interfaces

partial interface HTMLMediaElement {
  // URL passed to src attribute to enable the media source logic.
  readonly attribute [URL] DOMString webkitMediaSourceURL;

  // append bytes from webkitMediaSourceURL
  bool webkitSourceAppend(in Uint8Array data);
};
HTTP Adaptive Streaming for Web Developers
video.addEventListener('webkitsourceopen', function(e) {
  fetch_header(this, playlist);
  fetch_chunk(this, playlist);
}, false);

video.src = video.webkitMediaSourceURL;

var current_cluster = 0;
function fetch_chunk(video, playlist) {
  var start_bytes = playlist.clusters[current_cluster].offset;
  var range_req = 'bytes=' + start_bytes + '-';
  var xhr = new XMLHttpRequest();
  xhr.open('GET', playlist.filename, true);
  xhr.setRequestHeader("Range", range_req);
  xhr.onload = function(e) {
    video.webkitSourceAppend(new Uint8Array(this.response));
  };
  xhr.send();
}
Example playlist file
{"total_size": 2348204,
 "height"    : 270, 
 "width"     : 480, 
 "duration"  : 33003.0,
 "clusters"  :
     [{"timecode":     0, "offset":    4444},
      {"timecode":   315, "offset":   10989},
      {"timecode":  4309, "offset":  222621},
      ...
      {"timecode": 32298, "offset": 2336319}], 
  "filename" : "test.webm"}

More details

Experimental Implementation in Chrome 17+

Activate through chrome://flags/

Requirements for Video Conferencing
  • access to local devices: camera, microphone
    • navigator.getUserMedia()
  • connecting to remote peers
    • PeerConnection API
  • accepting data from local or remote peer
    • MediaStream API
  • displaying from local or remote peer
    • createObjectURL
Rough example code snippet
var mirror = document.getElementsByTagName('video')[0];
var peer = document.getElementsByTagName('video')[1];
var connection = new PeerConnection("STUN stun.l.google.com:19302", onSignal);

// get local video and audio
navigator.getUserMedia("video,audio", gotStream);

function gotStream(stream) {
  // mirror video locally
  mirror.src = URL.createObjectURL(stream);
  // start sending video
  connection.addStream(stream);
}

// peer starts sending video
connection.onaddstream = function (event) {
  // show video locally
  peer.src = URL.createObjectURL(event.stream);
};
There's more...

BlobCallback

SignallingCallback

RTCWeb Offer/Answer Protocol

Javascript Session Establishment Protocol

captureStream

Canvas recording

Web Workers for media

Web Audio API

Audio Data API

MediaController API

Stream mixing

...but that's enough for now.

Questions?

@silviapfeiffer