Lets face it, Apple really stuck us in a bad situation.  While HTML5 is really slick on the desktop (in supported browser with supported file types), its pretty clunky on iOS.  And Flash is ALSO really slick on the desktop, but doesn’t work at all on iOS.

So what we are left with are 2 things that work really well on the desktop, but no common thing that works well on Android, iOS, and the desktop.

What can we do?

Native iOS

Simply the best option on iOS now is to sign up for a developer’s license and make an app.  iOS offers two different video player frameworks MPMoviePlayer and AVPlayer.  MPMoviePlayer is the simplest to get started with, and offers you playback through the Quicktime UI.  AVPlayer is harder to get started with and you are forced to create your own UI, but you have more control over video playback.

AIR for Mobile

What may be the best solution is to build a Flash player with HLS streams on iOS and normal HTTP or RTMP streams on Android or the desktop.  While it’s true that Flash isn’t supported on iOS, Flash Builder 4.5 can compile your existing project out to iOS and Android applications.  All Flash video is supported in AIR, Mobile AIR and the web Flash player – except for iOS.  So you’ll need the two different streams.

Corona and other 3rd Party Packagers

Corona seems to be a very popular way to create cross platform mobile apps these days.  There are other packagers as well like Titanium, Appcelerator and more.  Last I looked they don’t handle video, or don’t handle video very well.  Probably the best is Corona which does allow video to play, but it needs to launch the video in a modal window or fullscreen, and you have little control over it. As cool as these solutions are, I don’t believe that they handle video all that well.

Before we conclude this series, I should mention something nobody likes – but they are an absolute necessity if content producers are going to put their video online in the first place.

Those things are ads and reporting.

Content owners want to monetize their video, and be able to track many aspects of it’s usage to make better decisions about video you’re watching.

With Hulu, Netflix, and many networks putting first run television online it seems that web video is gaining on what we watch on cable TV, and we may see a day soon when the web is significant enough to eat into cable tv’s revenue model such that we’ll networks and content producers who want to thrive can’t afford to just be on one or the other.

As ad revenue from cable tv and network broadcasts are shifted, owners will want to make sure this money is JUST shifted and not lessened.  So even today, folks are experimenting with longer ad breaks to see if they can get away with the same video experience you see on TV.

To accomplish this, we need some good 3rd party libraries from ad companies and reporting companies, as well as decent support on the actual device, browser, or plugin to facilitate these advanced needs.  If we can’t come up with creative solutions to accommodate ads and reporting, companies will simply not come – and this can make or break a platform’s usefulness for serving video.

3rd Party Library maturity

I’ve been working with Flash video extensively for a few years now, and in that time I’ve seen 3rd party frameworks and libraries delivered to us from various ad and reporting companies (many revisions in fact).  Flash seems to have a lock on maturity of 3rd party libraries for handling these things.  Flash has almost maniacal support for every aspect of playing video – something you won’t see from HTML5, or even iOS (from what I’ve seen so far).  In truth, this is why Flash is difficult to work with for video newcomers.  But, at the same time it offers extreme control to allow almost anything we need.  We can offer midrolls, prerolls, postrolls, companions, overlays and more.  We can also track video buffering events, stream transitions, and many more minute details.

HTML5 video, on the other hand doesn’t have this level of control – especially in iOS given the playback obstacles we’ve discussed.  Given this – many 3rd party libraries are still figuring out how to offer the same level of ads and reporting we can obtain from Flash players.

One company I work with just recently started offering midroll ads, where before their Javascript/HTML5 video solution wasn’t mature enough to do this.  They seem to be coming along just fine, but won’t be caught up to Flash anytime soon!  See part 6 for my experiences working with mid roll ads.

Advertising with MP4 – At Least that’s Easy!

I’ve talked about iOS streaming for long-form content.  It uses Apple’s HLS, and is pretty much incompatible with Flash or Silverlight.  As a result, content owners will need to completely re-encode their streams for Apple devices – that is IF they want it to be streaming.

Many advertisers these days, however, will serve up their ad creatives as mp4 files encoded with h264.  This works on Flash, and it ALSO works on iOS, Chrome, and IE.  This means that if these advertisers wanted to use their same ads on iOS, they’re free to do so without have to re-encode the files!

Companion Ads and Banners

The same can’t be said for companion ads, however.  Companion ads are static or animated images that you see outside the video player area (usually in another div on the page).  The problem is that these ads, in my experience, aren’t static images and usually animated Flash.  This means that advertisers and content producers need to work together to create new companion creatives and serve them up appropriately for Flash or iOS.

So we talked about how Apple supports both progressive and streaming video with their HTML5 video tag.  We also briefly discussed how Flash Media Server serves up its OWN streams.  What about Apple’s HTTP Live Streaming?  What kind of magic is that?

Turns out that its quite simple.  Take a video file, after encoding it for HLS you’ll end up with a directory of files.  First off there is a table of contents file.  This table of contents files points to MORE table of contents files, and these point to video file segments.

The video segments are most likely h264 encoded segments.  Each one has a “ts” file extension (myfile.ts).  ”TS” stands for transport segment.  Each segment will most likely be small, about a megabyte or so and contain several seconds worth of video (this is entirely left up to the encoder settings).

The table of contents files sound a little hacky, but they work well.  Apple has gone ahead and adopted m3u8 playlist files for streaming video.  What’s that you ask?  What’s m3u8?  Well, if you crack open Winamp or iTunes, you can create a music playlist.  Both m3u and pls files are pretty standard for holding playlists of songs.  The m3u8 spec is an enhanced version of this (and still used for music playlists!). It makes sense though, you’re really just assembling a playlist of video segments.

I also mentioned playlists in playlists didn’t I?  Well, the master m3u8 file will typically hold several different bitrate references (for if a user wants high definition, low definition, or anything in between).  This file will hold references to other m3u8 files and list what their bitrates are in bytes per second.

The reference m3u8 files will be the ones that actually list the segments, examples are listed below.

First is an m3u8 file referencing OTHER m3u8 files for the different bitrates:

#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000

http://ALPHA.mycompany.com/lo/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000

http://BETA.mycompany.com/lo/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000

http://ALPHA.mycompany.com/md/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000

Next up is an example of the actual m3u8 file that holds all of our file references.
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:10, no desc
fileSequence0.ts
#EXTINF:10, no desc
fileSequence1.ts
#EXTINF:10, no desc
fileSequence2.ts
#EXTINF:10, no desc
fileSequence3.ts
#EXTINF:10, no desc
fileSequence4.ts
#EXTINF:10, no desc
…..
#EXT-X-ENDLIST

Live Playback Limits on HLS

In the example above, notice the line #EXT-X-ENDLIST.  This command marks the end of the file.  This an important piece of info!  It tells Quicktime that there will be no more segments after this, and what we have in the file is the entire stream.

What if this line didn't exist?  Well, it turns out that the absence of this line indicates to Quicktime that the m3u8 file is a live stream.  In this regard, the file has no end.  With a VOD (video on demand) stream, Quicktime only needs to query this table of contents once, and the player is secure in it's knowledge that there is nothing else to load.  For a live stream, the m3u8 file is repeatedly queried to find out what the latest segments are so the player can keep up with the most recent segments.

This is a nifty little system -  but there's a problem.  Typically, a live event will allow the end user to watch in a 30 minute or so sliding window.  The user can't seek backwards more than 10 minutes or so.  These older segments are dropped from the m3u8 file such that the m3u8 file only contains a handful of segments.

The problem is if you need a long stream.  My last project had a need to run clips from a large 14 hour stream.  When playing through HTML5 on a web page, the stream broke down after it grew to 40 minutes or so.  The problem was that the growing number of entries referencing transport segments made the file grow exceptionally large after a while.  If I recall correctly, a 10 hour stream's m3u8 file was around 600kb for me. Multiply that by 3 for the other bitrates I had.  So eventually, Quicktime was constantly querying over a megabyte of playlist info - and it had trouble keeping up with playback!

I came up with the solution of having our encoding team put a fake end on the m3u8 file even though it was live.  This caused the file to only be loaded once even though the live stream was continually playing.  It doesn't work for a lot of situations since you're not able to play the most recent stuff since you loaded the file, but it worked for our specific needs with our site!

Miraculously, it seems that this is only a problem in HTML5, and not a problem if you're just running Quicktime natively or in an app.

Other Streaming Services

I don't have any experience with Silverlight, but IIS's Smooth Streaming functions pretty much the same way.  Instead of an m3u8 file, Smooth Streaming has an XML format for it's table of contents.

We discussed how Flash Media Server worked in a previous post in this series - but to recap, it can generate these segments on the fly and serve them over RTMP, HTTP and for Flash or iOS.

Playback in Flash doesn't have the same length issues that iOS HTML5 has for live streams.  It plays things back fairly well!  HLS and SmoothStreaming though, have a pretty important feature on Flash - bitrate switching happens automatically between the server and the player.  You don't need to program anything extra in your code to make bitrate switching work - and it does work very smoothly!  You'd be hard pressed to notice the transition.

Flash recently implemented a smooth stream switchover as well.  You as a developer, must measure bandwidth and make the switch yourself (though OSMF will do this automatically for you).  Once you call the "play2" command, it will START making the switch, but it will wait until the next video keyframe to make the switch so its as seamless as HLS and Smooth Streaming make it.

We’ve discussed in part 4 of this series that Flash can get a little complicated.  It’s true, but it mostly boils down to the different video formats you can play.  Let’s talk about those formats.

Progressive Video

If you are clueless about what the difference is between streaming and progressive video – chances are the video you know and love is said to be “progressive”.  What does this mean?

Well, if you hit a single video file on a server somewhere and start downloading it, you can actually watch the video before it finishes downloading.  You are using it “progressively” in that you are watching it while download is in progress.  This happens in much the same way as if you had an image loading on a slow connection and you see it start at the top and fill in as it gets loaded.

One of the problems I’ve seen from progressive video files (at least when playing from Flash) is when the MOOV Atom is placed at the end of the file by the video encoder.  Think of the MOOV Atom as a block of data in the file that holds info about your video…like duration, bitrate, frame rate, encoding specs, etc.   If this is place at the end of the file, the player doesn’t know the details of your video file, and won’t play it UNTIL it gets this block of data.  Since it’s at the end of the file, users need to wait until the entire video file is downloaded so that the video player knows exactly what it’s playing.

Streaming Video

So, speaking progressively, video is one large video file that you download and cache to your computer (or at least that’s whats happening behind the scenes in your browser) for playback.

On the other side of the coin, we have streaming playback.  Typically a streaming file will be served from some sort of media playback server as a lightweight container file that serves as a table of contents for many smaller video segment files.  The lightweight container file will be loaded once up front (or continually if it is often updated like when you are connected to a live stream).  Your player will then be smart enough to know what segment files are needed to playback video.  It will go out and grab these segments, and then throw them away when you’re done.  A player will keep a “buffer” that may or may not be able to be set by the player of how many seconds of video to keep (and therefore how many video segments).  If you have video buffered, you can easily seek back and forth to these points instantaneously.  If on the other hand, you seek outside the buffer, the player will have to read the table of contents container and lookup which video segments to fetch and buffer for you.

The Flash Media Server will hide all the complexities from you, though.  Usually, you’ll just connect to a path that starts with rtmp:// and ends with myfile.mp4.  For all YOU know it’s just a file, not chunked up into tiny segments.  And actually it sort of IS one big file.  Flash Media Server will just tranform this file into a stream for you on demand, so its easy to manage and still works like a stream.

Flash Media Server, in recent days,  has actually started supporting streams over HTTP for Flash and iOS.   This means that if your company doesn’t want to deal with serving stuff over RTMP, you can just use HTTP streams and Flash 10.1 (and OSMF if you don’t want to program support yourself).  You can use your same server and files to send a stream to iOS!  Kind of a timesaver if you need to support both, instead of managing all 2 separate servers/encoders.

Use Cases for Each

Why use streaming vs progressive?  Well the major reason is what we in the business call “long-form video”.  Long form video is typically 20 minutes to a few hours long, while  ”short-form video” is under a minute or a few minutes long.  YouTube clips are perfect examples of short form video.  And they are also decent examples of why the overhead of having a streaming service and player is completely unnecessary.  If it’s only a few megabytes, then it really doesn’t take long to load and cache the whole video does it?  And in the end, when you close your browser you’ll only have a few MB from it in your cache or in your memory.

Long-form, on the other hand, can be several hundred megabytes or even a couple of gigabytes depending how long the video is and what quality it was encoded at.  Not only will you be left with a behemoth of a file in your RAM or cache, but it you can’t seek forward to skip over pieces until that much has been downloaded.

Another benefit of streaming is the ability to switch between bitrates or cameras.  If your video player thought that it was having difficulty keeping up with your HD file, ideally, you’ll switch to a lower resolution video.  It’s REALLY difficult and pointless to do this on a progressive file.  You’d have to latch on to a different file, and then seek to the position you left off at….which means you’d have to progressively download the file up to the point where you left off which could take several minutes!  With a stream, all you have to do is tell the server to start sending you different file segments.  It will do so, and the player will never be the wiser, cause all it sees is new segments coming in – it doesn’t really matter if they are from a different source file.

So that coupled with a server’s ability to offer DRM is a huge reason why TV shows and movies are all streaming.  While there are many DRM solutions available, one fact is that when you segment all your files like this (even without DRM), its a pain in the butt to download and assemble them all if you were so inclined to steal content.  Flash Media Server obfuscates the stream even more by serving it over RTMP.  Since it’s an entirely different protocol than HTTP, most standard web dev tools won’t even see the files come through and its even more difficult to steal!

Contrast this with YouTube.  Most of the time, I can go in and open up my Chrome developer tools, watch a big file come in, and copy the source URL.  I can then download the file and rename to a mp4 or flv file and then play it on my desktop.  BAM!  I just stole content and it took 30 seconds.

Something in between (pseudo-streaming)

You may have noticed recently in YouTube, where it shows you a red line in your progress bar to indicate your downloaded buffer.  In years past, you couldn’t seek beyond this.  Well, now you can!  How is this possible?