javascript « Benjamin Farrell

C Web Run, Run Web Run

After a long vacation, i’m finally back from Adobe MAX and California. Going to MAX certainly gave a lot of perspective on things.

Adobe’s keynote around Flash and HTML5 seemed to be a hot button for some people. Flex wasn’t mentioned, and Flash was named “the console of the web”. This seemed to draw ire from people who know Flash is more than games.

I actually think the verbage is just fine. Flash is the box in your living room that runs nifty stuff. It does applications, video, games, and all the cool stuff that is net connected but isn’t a “web page”.

Obviously, Flash does RIA applications because it does everything else. It’s a little unfortunate that RIAs aren’t part of the overall message, but to be realistic, when normal people talk RIAs, they are thinking of what the HTML/JS/CSS stack can do easily—and if you want to claim that Flex should be used where HTML/JS/CSS can be easily used, you are fighting a losing battle these days.

For the rest of us technically adept people, we know in which situations Flash can or can’t be used. It’s unfortunate it’s been reduced to “web standards” versus “the console,” but Adobe had to put it in terms the world could understand. I think they threw in the towel on a small battle to win the larger war to keep Flash going strong.

This is all interesting, but it’s not my takeaway from MAX. Adobe gives us a stack for the console: Flex/AIR/Flash. They also give us a stack for standards-based Web: JQuery/HTML/PhoneGap. The untold story taking shape in the creative space is low level and close to the metal, as they say. C++ is heading to the creative space and the Web. It is slowly taking shape as the technology that can be injected into the Web stack, the Flash stack, and the creative expression stack.

Prior to MAX, I had been looking into OpenFrameworks (OF). It’s a C++ library that allows people to creatively express themselves using C++. This is significant because C++, in the past, was seen as one of those daunting languages that nobody wanted to touch because it was too complicated and reserved for computer rocket scientists. But OF aims to make C++ approachable for creative types and allows them to push graphics around and do some cool things like face tracking, object detection, and more.
Basically, anything creative that Flash or Javascript is too slow to handle, C++ can do and OF is your path to get there.

Sure, OF is significant, but it’s just part of the story. Adobe has another project in labs called Alchemy. They recently announced in a MAX session that it would be getting a lot more attention…official product support attention! You may remember Alchemy as a MAX demo a few years back. The source code for Quake was compiled from C++ through Alchemy and used as a Flash-compiled library. The end result was that a Quake level was cross compiled and playable in Flash (and this was before Stage3D).

Alchemy is used to compile existing C++ libraries or just code you’d like to run that goes too damn slow in Flash/Actionscript. This opens up a world of possibilities for near native performance. In fact, remember that Photoshop Touch demo you saw in the Day 1 keynote? That was built with AIR. Still, Actionscript is just a shade too slow to run Photoshop’s advanced image manipulation algorithms. Instead, the developers simply leveraged Photoshop’s existing C++ libraries and plugged them into AIR/Flash.

That story was the same as the one I overheard from Ben Garney of PushButton Labs when talking about his game development work. He coded up a library in C++, which could be targeted at normal native usage. When needed in Flash, the library can be simply compiled with Alchemy for usage in Flash.

The benefit is immense here—incredible performance boosts and the ability to leverage the same code everywhere that runs C++.

But the Flash story doesn’t stop there. AIR 3.0 allows native extensions on both Desktop or mobile applications. Whether you use C++ to create a windows DLL or use the NDK to create an Android extension, it doesn’t matter. Objective-C for iOS might be a slighly different flavor…but Apple isn’t known for playing nice.

The benefits of Flash are huge, but what about beyond Flash?

Google Chrome is part of the same story with their “Native Client.” Just recently, Google gave us the ability to leverage C++ code in their Web browser for more power and speed. Meanwhile, projects like Emscripten will compile C++ to Javascript. While Emscripten doesn’t equate to a speed boost, you can leverage exisiting C++ libraries. This is important because C++ has been around for decades, and someone before you may have already solved your problem.

Where is all of this going? Well, I conjecture that we will end up doing the meaty bits of our apps/applications/Webpages with C++. We’ll share the same C++ libraries across Flash, AIR, Desktop, Mobile, and Web. We’ll then layer a thin client on top whether or not we write it in Javascript, MXML, Actionscript, Dart, HTML, etc. This layer will be visual, with the bottom C++ layer used for perfomance.
I may be completely wrong about this prediction. Even if I am, the current creative ways to express ourselves with C++ give us normal folk a definite inroad into learning the most popular language on the planet.

In other words, if you take the opportunity now to learn C++, what could possibly go wrong?!

09Sep

HTML5 vs Flash Video: Choose Wisely Part 10 – Other Options

Lets face it, Apple really stuck us in a bad situation. While HTML5 is really slick on the desktop (in supported browser with supported file types), its pretty clunky on iOS. And Flash is ALSO really slick on the desktop, but doesn’t work at all on iOS.

So what we are left with are 2 things that work really well on the desktop, but no common thing that works well on Android, iOS, and the desktop.

What can we do?

Native iOS

Simply the best option on iOS now is to sign up for a developer’s license and make an app. iOS offers two different video player frameworks MPMoviePlayer and AVPlayer. MPMoviePlayer is the simplest to get started with, and offers you playback through the Quicktime UI. AVPlayer is harder to get started with and you are forced to create your own UI, but you have more control over video playback.

AIR for Mobile

What may be the best solution is to build a Flash player with HLS streams on iOS and normal HTTP or RTMP streams on Android or the desktop. While it’s true that Flash isn’t supported on iOS, Flash Builder 4.5 can compile your existing project out to iOS and Android applications. All Flash video is supported in AIR, Mobile AIR and the web Flash player – except for iOS. So you’ll need the two different streams.

Corona and other 3rd Party Packagers

Corona seems to be a very popular way to create cross platform mobile apps these days. There are other packagers as well like Titanium, Appcelerator and more. Last I looked they don’t handle video, or don’t handle video very well. Probably the best is Corona which does allow video to play, but it needs to launch the video in a modal window or fullscreen, and you have little control over it. As cool as these solutions are, I don’t believe that they handle video all that well.

09Sep

HTML5 vs Flash Video: Choose Wisely Part 9 – The Verdict

So what wins? HTML5 video or Flash? What should you use, what’s better?

The answer to this is that it depends on your project and what platforms you are targeting.

Use HTML5 video for simple playback

For short form video that contains no in-stream advertisements (that’s fancy talk for pre, post, and midrolls) its extremely easy to use HTML5 now. If your content is at most a few minutes long and you want it to work on any browser and any device you should use HTML5 with a Flash fallback (or vice versa).

Your content should be MP4/h264 so that it will work on iOS, Flash, Chrome, and IE9. If you feel feisty, you can make a copy of your content as OGG to play in Firefox, but you can easily handle Firefox with Flash video.

Use Flash for in-depth Ad and Tracking support

Adding ads can be a little trickier. It’s much easier to do this with Flash, but if your content is short, all you’ll most likely need is a short pre-roll. If doing this with HTML5, the experience may be a bit clunky, but at least it will work on the iPad.

It stinks a little more as you add tracking (or other features). Do you want to duplicate your programming efforts in both Flash and Javascript?

One suggestion might be to implement a Flash player with no control bar and to only handle simple playback of video. Do all your graphics, reporting and ad logic in Javascript, and just use the Flash player in the same way you’d use a video tag. In this way, you’re using two things that are pretty interchangable.

All in all, you’ll be most successful if you implement these features in Flash as Flash has the most support for this type of thing. HTML5 is certainly possible, but hard – you’ll need to power through it, especially if you need it to work on iOS.

Use HTML5 for iOS
If your only options are Flash or HTML5, obviously you should use HTML5 for playback since Flash isn’t supported!

Use Flash for Long-Form Streaming Content

This is a hard truth – HTML5 can’t do streaming. As such, it can’t do long-form content without completely bogging down end user’s browsers.

For playback on Android devices, you can use your same Flash player for long-form streaming playback (just make sure to make it mobile and touch friendly).

For iOS, you’re pretty much out of luck. You can run streaming long form content through iOS, but unfortunately that same content won’t work on normal browsers. If you want playback on both desktop browsers and iOS devices, you’ll need to double your efforts and develop players for each scenario.

09Sep

HTML5 vs Flash Video: Choose Wisely Part 8 – Advertising and Reporting

Before we conclude this series, I should mention something nobody likes – but they are an absolute necessity if content producers are going to put their video online in the first place.

Those things are ads and reporting.

Content owners want to monetize their video, and be able to track many aspects of it’s usage to make better decisions about video you’re watching.

With Hulu, Netflix, and many networks putting first run television online it seems that web video is gaining on what we watch on cable TV, and we may see a day soon when the web is significant enough to eat into cable tv’s revenue model such that we’ll networks and content producers who want to thrive can’t afford to just be on one or the other.

As ad revenue from cable tv and network broadcasts are shifted, owners will want to make sure this money is JUST shifted and not lessened. So even today, folks are experimenting with longer ad breaks to see if they can get away with the same video experience you see on TV.

To accomplish this, we need some good 3rd party libraries from ad companies and reporting companies, as well as decent support on the actual device, browser, or plugin to facilitate these advanced needs. If we can’t come up with creative solutions to accommodate ads and reporting, companies will simply not come – and this can make or break a platform’s usefulness for serving video.

3rd Party Library maturity

I’ve been working with Flash video extensively for a few years now, and in that time I’ve seen 3rd party frameworks and libraries delivered to us from various ad and reporting companies (many revisions in fact). Flash seems to have a lock on maturity of 3rd party libraries for handling these things. Flash has almost maniacal support for every aspect of playing video – something you won’t see from HTML5, or even iOS (from what I’ve seen so far). In truth, this is why Flash is difficult to work with for video newcomers. But, at the same time it offers extreme control to allow almost anything we need. We can offer midrolls, prerolls, postrolls, companions, overlays and more. We can also track video buffering events, stream transitions, and many more minute details.

HTML5 video, on the other hand doesn’t have this level of control – especially in iOS given the playback obstacles we’ve discussed. Given this – many 3rd party libraries are still figuring out how to offer the same level of ads and reporting we can obtain from Flash players.

One company I work with just recently started offering midroll ads, where before their Javascript/HTML5 video solution wasn’t mature enough to do this. They seem to be coming along just fine, but won’t be caught up to Flash anytime soon! See part 6 for my experiences working with mid roll ads.

Advertising with MP4 – At Least that’s Easy!

I’ve talked about iOS streaming for long-form content. It uses Apple’s HLS, and is pretty much incompatible with Flash or Silverlight. As a result, content owners will need to completely re-encode their streams for Apple devices – that is IF they want it to be streaming.

Many advertisers these days, however, will serve up their ad creatives as mp4 files encoded with h264. This works on Flash, and it ALSO works on iOS, Chrome, and IE. This means that if these advertisers wanted to use their same ads on iOS, they’re free to do so without have to re-encode the files!

Companion Ads and Banners

The same can’t be said for companion ads, however. Companion ads are static or animated images that you see outside the video player area (usually in another div on the page). The problem is that these ads, in my experience, aren’t static images and usually animated Flash. This means that advertisers and content producers need to work together to create new companion creatives and serve them up appropriately for Flash or iOS.

09Sep

HTML5 vs Flash Video: Choose Wisely Part 7 – What’s Happening on the Server Side

So we talked about how Apple supports both progressive and streaming video with their HTML5 video tag. We also briefly discussed how Flash Media Server serves up its OWN streams. What about Apple’s HTTP Live Streaming? What kind of magic is that?

Turns out that its quite simple. Take a video file, after encoding it for HLS you’ll end up with a directory of files. First off there is a table of contents file. This table of contents files points to MORE table of contents files, and these point to video file segments.

The video segments are most likely h264 encoded segments. Each one has a “ts” file extension (myfile.ts). ”TS” stands for transport segment. Each segment will most likely be small, about a megabyte or so and contain several seconds worth of video (this is entirely left up to the encoder settings).

The table of contents files sound a little hacky, but they work well. Apple has gone ahead and adopted m3u8 playlist files for streaming video. What’s that you ask? What’s m3u8? Well, if you crack open Winamp or iTunes, you can create a music playlist. Both m3u and pls files are pretty standard for holding playlists of songs. The m3u8 spec is an enhanced version of this (and still used for music playlists!). It makes sense though, you’re really just assembling a playlist of video segments.

I also mentioned playlists in playlists didn’t I? Well, the master m3u8 file will typically hold several different bitrate references (for if a user wants high definition, low definition, or anything in between). This file will hold references to other m3u8 files and list what their bitrates are in bytes per second.

The reference m3u8 files will be the ones that actually list the segments, examples are listed below.

First is an m3u8 file referencing OTHER m3u8 files for the different bitrates:

#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000

http://ALPHA.mycompany.com/lo/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000

http://BETA.mycompany.com/lo/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000

http://ALPHA.mycompany.com/md/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000

Next up is an example of the actual m3u8 file that holds all of our file references.
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:10, no desc
fileSequence0.ts
#EXTINF:10, no desc
fileSequence1.ts
#EXTINF:10, no desc
fileSequence2.ts
#EXTINF:10, no desc
fileSequence3.ts
#EXTINF:10, no desc
fileSequence4.ts
#EXTINF:10, no desc
…..
#EXT-X-ENDLIST

Live Playback Limits on HLS

In the example above, notice the line #EXT-X-ENDLIST. This command marks the end of the file. This an important piece of info! It tells Quicktime that there will be no more segments after this, and what we have in the file is the entire stream.

What if this line didn't exist? Well, it turns out that the absence of this line indicates to Quicktime that the m3u8 file is a live stream. In this regard, the file has no end. With a VOD (video on demand) stream, Quicktime only needs to query this table of contents once, and the player is secure in it's knowledge that there is nothing else to load. For a live stream, the m3u8 file is repeatedly queried to find out what the latest segments are so the player can keep up with the most recent segments.

This is a nifty little system - but there's a problem. Typically, a live event will allow the end user to watch in a 30 minute or so sliding window. The user can't seek backwards more than 10 minutes or so. These older segments are dropped from the m3u8 file such that the m3u8 file only contains a handful of segments.

The problem is if you need a long stream. My last project had a need to run clips from a large 14 hour stream. When playing through HTML5 on a web page, the stream broke down after it grew to 40 minutes or so. The problem was that the growing number of entries referencing transport segments made the file grow exceptionally large after a while. If I recall correctly, a 10 hour stream's m3u8 file was around 600kb for me. Multiply that by 3 for the other bitrates I had. So eventually, Quicktime was constantly querying over a megabyte of playlist info - and it had trouble keeping up with playback!

I came up with the solution of having our encoding team put a fake end on the m3u8 file even though it was live. This caused the file to only be loaded once even though the live stream was continually playing. It doesn't work for a lot of situations since you're not able to play the most recent stuff since you loaded the file, but it worked for our specific needs with our site!

Miraculously, it seems that this is only a problem in HTML5, and not a problem if you're just running Quicktime natively or in an app.

Other Streaming Services

I don't have any experience with Silverlight, but IIS's Smooth Streaming functions pretty much the same way. Instead of an m3u8 file, Smooth Streaming has an XML format for it's table of contents.

We discussed how Flash Media Server worked in a previous post in this series - but to recap, it can generate these segments on the fly and serve them over RTMP, HTTP and for Flash or iOS.

Playback in Flash doesn't have the same length issues that iOS HTML5 has for live streams. It plays things back fairly well! HLS and SmoothStreaming though, have a pretty important feature on Flash - bitrate switching happens automatically between the server and the player. You don't need to program anything extra in your code to make bitrate switching work - and it does work very smoothly! You'd be hard pressed to notice the transition.

Flash recently implemented a smooth stream switchover as well. You as a developer, must measure bandwidth and make the switch yourself (though OSMF will do this automatically for you). Once you call the "play2" command, it will START making the switch, but it will wait until the next video keyframe to make the switch so its as seamless as HLS and Smooth Streaming make it.

09Sep

HTML5 vs Flash Video: Choose Wisely Part 6 – Apple’s HTML5 vs Normal HTML5

If you are interested in pursuing HTML5, I imagine the reason why you are doing so is to support iOS. As we all know, iOS doesn’t support Flash, so to play video we’ll need to take advantage of HTML5 video.

HTML5 is fairly normal via Safari on OSX. Safari is just another browser, however the video tag is handled by Quicktime. It looks like we’re trading one plugin for the other here!

There’s probably nothing wrong with using Quicktime here in OSX. If Apple chooses to use one of their oldest video technologies to playback video in their browser on their OS, no big deal! It really probably a matter of semantics anyway – could you consider the h264 decoder in Chrome a plugin? Yes and no – its a fine line.

On Windows, however, if you use Safari, you’ll need the Quicktime plugin installed! So, here we’re REALLY talking about a plugin. In this situation its both a plugin AND HTML5 at the same time.

As it is Quicktime, we can use the video tag to play whatever Apple can normally play with Quicktime. In fact, we can play Apple’s HTTP Live Streaming (HLS). I think this leads to a great deal of confusion and is a pretty big bait and switch on Apple’s part.

First they tell us “who needs Flash?! Its slow, buggy, and HTML5 can replace it!”. Then they live stream their developer event in HTML5, but surprise! Only Apple’s brand of HTML5 can stream it, because it’s really Quicktime under the hood. Even more than that, streaming video isn’t even part of the HTML5 spec! So, it should be quite awhile before we’ll see streaming content come to browsers natively.

Some Apple playback quirkiness

I really haven’t seen much in terms of weirdness on Safari OSX. I haven’t had to play with it much, but the video tag works pretty much as advertised here (since it’s webkit, it’s just like using Chrome in fact).

But that’s not really what matters – on OSX you have the option of Flash too. On iOS, you either need to go HTML5, or write an app. There’s no other option!

So, how does the video tag fare? There are some major things here to be aware of. Apple has decided that we can’t be trusted with the autoplay feature of the video tag, so they have disabled it. This means that you cannot automatically play a video after a page loads. There needs to be some sort of user intervention here to kick off the video. This will usually involve a play button overlaid on top of the video that the user clicks to start playback. It’s a little heavy handed in my opinion, but the reasoning behind this is to not let malicious people use your data for things that you don’t want – like ads.

Unfortunately, this prevents some major barriers for advanced video applications. Something as simple as playing midroll ads (ads that appear in the middle of content at a specified ad break) is made complicated.

Midroll Ads

See, one problem with midroll ads is that you must stop your content, flip over to the ad, and then after the ad, resume content again. I’ve done this in Flash and am currently doing this in iOS/Objective-C. In my experience, the best way to do this is to use two separate video objects. With your main content in one video object, when an ad comes on, you can pause this and spawn a new video object overlaying it on top of your main content. Play your ad in the new video object, and then when you’re done, kill the object and resume your main video.

With this game plan, you don’t need to suffer through a rebuffer of your main video content after your ad. If using the same video object, you’d need to not only load the video again, but automatically seek to the spot right after the ad break.

Using two video objects like this isn’t really feasible. For each video tag you use, you’d need to tell the user to manually click play on each one. When playing an ad – you’re adding insult to injury like this!

Furthermore, another quirk I’ve seen is when you hide a video tag on iOS using CSS and either hidden or display:none. This consistently crashed my iPad browser when I attempted this. So maintaining 2 video tags, but hiding one isn’t an option!

In essence, midroll ads seem to be stretching the capabilities of HTML5 on iOS!

Automatically Seeking

Yet another quirk I’ve seen is attempting to automatically seek into a stream a certain number of seconds (which you’d need to do when returning from an ad, or just playing a clip of a larger video). The problem is that when you initially load a video resource, it can take some time before you can interact with it and tell it to seek.

The initial run up to playback includes the video object becoming ready, the asset being loaded enough to start playback, and finally as more of the asset is loaded, and finally it becomes seekable. The problem is that this process can take a second or so. In the MEANTIME you can’t interact with the video and tell it to seek. You just have to sit there and watch it playback a second or half second of video. What should be an automatic seek becomes an automatic seek with a small video glitch at the beginning.

Player Chrome

Your customers like customization don’t they? Some would like some nifty custom graphics to show up wouldn’t they? A custom lime-green playback button to match their brand maybe? It’s completely possible on the desktop with HTML5 – simply hide the controls and create your own, positioning them over the video with CSS. The problem is that this is a bit problematic in iOS. See – in my experience, you can’t overlay things on top of a playing video. Well actually you can – but the video window will suck in all your mouseclicks. This means that while you can overlay your own controls on iOS, they probably won’t function. Nothing can obscure the video and be perfectly functional!

Furthermore, you always get the same UI when you run either on an iPhone or an iPad in fullscreen mode. You get Quicktime’s native controls – your page is gone (in the background) and you are left with nothing but the video playing in the Quicktime application. Fortunately enough, Javascript can still operate in the background and control your video.

Video API Events

I’ve also noticed that events can be a little unreliable on iOS, like listening for “onCanPlay” or when the player is ready. I’ve had to setup a timer that fires every half second or so to check for readiness and playback state in my own project to accommodate this problem.

08Sep

HTML5 vs Flash Video: Choose Wisely Part 5 – Streaming vs Progressive

We’ve discussed in part 4 of this series that Flash can get a little complicated. It’s true, but it mostly boils down to the different video formats you can play. Let’s talk about those formats.

Progressive Video

If you are clueless about what the difference is between streaming and progressive video – chances are the video you know and love is said to be “progressive”. What does this mean?

Well, if you hit a single video file on a server somewhere and start downloading it, you can actually watch the video before it finishes downloading. You are using it “progressively” in that you are watching it while download is in progress. This happens in much the same way as if you had an image loading on a slow connection and you see it start at the top and fill in as it gets loaded.

One of the problems I’ve seen from progressive video files (at least when playing from Flash) is when the MOOV Atom is placed at the end of the file by the video encoder. Think of the MOOV Atom as a block of data in the file that holds info about your video…like duration, bitrate, frame rate, encoding specs, etc. If this is place at the end of the file, the player doesn’t know the details of your video file, and won’t play it UNTIL it gets this block of data. Since it’s at the end of the file, users need to wait until the entire video file is downloaded so that the video player knows exactly what it’s playing.

Streaming Video

So, speaking progressively, video is one large video file that you download and cache to your computer (or at least that’s whats happening behind the scenes in your browser) for playback.

On the other side of the coin, we have streaming playback. Typically a streaming file will be served from some sort of media playback server as a lightweight container file that serves as a table of contents for many smaller video segment files. The lightweight container file will be loaded once up front (or continually if it is often updated like when you are connected to a live stream). Your player will then be smart enough to know what segment files are needed to playback video. It will go out and grab these segments, and then throw them away when you’re done. A player will keep a “buffer” that may or may not be able to be set by the player of how many seconds of video to keep (and therefore how many video segments). If you have video buffered, you can easily seek back and forth to these points instantaneously. If on the other hand, you seek outside the buffer, the player will have to read the table of contents container and lookup which video segments to fetch and buffer for you.

The Flash Media Server will hide all the complexities from you, though. Usually, you’ll just connect to a path that starts with rtmp:// and ends with myfile.mp4. For all YOU know it’s just a file, not chunked up into tiny segments. And actually it sort of IS one big file. Flash Media Server will just tranform this file into a stream for you on demand, so its easy to manage and still works like a stream.

Flash Media Server, in recent days, has actually started supporting streams over HTTP for Flash and iOS. This means that if your company doesn’t want to deal with serving stuff over RTMP, you can just use HTTP streams and Flash 10.1 (and OSMF if you don’t want to program support yourself). You can use your same server and files to send a stream to iOS! Kind of a timesaver if you need to support both, instead of managing all 2 separate servers/encoders.

Use Cases for Each

Why use streaming vs progressive? Well the major reason is what we in the business call “long-form video”. Long form video is typically 20 minutes to a few hours long, while ”short-form video” is under a minute or a few minutes long. YouTube clips are perfect examples of short form video. And they are also decent examples of why the overhead of having a streaming service and player is completely unnecessary. If it’s only a few megabytes, then it really doesn’t take long to load and cache the whole video does it? And in the end, when you close your browser you’ll only have a few MB from it in your cache or in your memory.

Long-form, on the other hand, can be several hundred megabytes or even a couple of gigabytes depending how long the video is and what quality it was encoded at. Not only will you be left with a behemoth of a file in your RAM or cache, but it you can’t seek forward to skip over pieces until that much has been downloaded.

Another benefit of streaming is the ability to switch between bitrates or cameras. If your video player thought that it was having difficulty keeping up with your HD file, ideally, you’ll switch to a lower resolution video. It’s REALLY difficult and pointless to do this on a progressive file. You’d have to latch on to a different file, and then seek to the position you left off at….which means you’d have to progressively download the file up to the point where you left off which could take several minutes! With a stream, all you have to do is tell the server to start sending you different file segments. It will do so, and the player will never be the wiser, cause all it sees is new segments coming in – it doesn’t really matter if they are from a different source file.

So that coupled with a server’s ability to offer DRM is a huge reason why TV shows and movies are all streaming. While there are many DRM solutions available, one fact is that when you segment all your files like this (even without DRM), its a pain in the butt to download and assemble them all if you were so inclined to steal content. Flash Media Server obfuscates the stream even more by serving it over RTMP. Since it’s an entirely different protocol than HTTP, most standard web dev tools won’t even see the files come through and its even more difficult to steal!

Contrast this with YouTube. Most of the time, I can go in and open up my Chrome developer tools, watch a big file come in, and copy the source URL. I can then download the file and rename to a mp4 or flv file and then play it on my desktop. BAM! I just stole content and it took 30 seconds.

Something in between (pseudo-streaming)

You may have noticed recently in YouTube, where it shows you a red line in your progress bar to indicate your downloaded buffer. In years past, you couldn’t seek beyond this. Well, now you can! How is this possible?

It’s what we call “pseudo-streaming”. Basically it acts like a progressive download, but with trickery it can have the main benefit of streaming which is to seek anywhere you want in the file regardless of how much downloaded.

It’s do-able because many servers (even if just a simple Apache server running PHP) allow the browser request to specify a byte range. Typically when you make a file request from your browser you ask for the file, and it ASSUMES you want all the bytes of the file starting at the beginning of the file. What pseudo streaming does is to ask for the file, but ask for the bytes at a certain position in the file and not to start at the beginning.

What’s Possible in HTML5 vs Flash

Pseudo Streaming is actually very popular these days because the HTML5 video tag supports it natively. To use it, you as a developer don’t have to do a thing beyond allowing your user to seek (as well as having a server that supports byte ranges). Contrast this with Flash – where you’d need to jump through hoops to get this working. Flash doesn’t support byte ranges with video or audio, which means you’d have to load the bytes MANUALLY, and then append the bytes onto the active video stream. Not only does this suck to code, but appending bytes to a stream wasn’t even supported until Flash Player 10.1.

Of course, progressive is supported in both HTML5 and Flash. Flash supports a few different encoding types including mp4/h264 just like HTML5 normally does (depending on the browser).

Here’s the bombshell though…HTML5 does not support actual streaming, and there is no official proposed spec for it! I bet you don’t believe me because of a certain large, fruity corporation. It’s true though! More on that in the next part. But the long story short here, is that I can’t imagine recommending HTML5 for long form content, doing 2 hour movies with a progressive download would be beastly (note that there may be clever ways, I just don’t know what they would be beyond artificially breaking up the video files and leaving gaping holes in playback while loading the next segment).

07Sep

HTML5 vs Flash Video: Choose Wisely Part 3 – Extra Help with HTML5

Filed in html5 | javascript | ui | video | web Leave a comment

In part two we discussed a good deal of how to use the video tag and how (if we’re ambitious) we can code up our own controls. We also discussed the fact that the HTML5 video tag might not be an option if you’re using an older browser. And maybe Firefox/OGG video isn’t part of your workflow. In either case, you need to opt out of HTML5 and opt into Flash (or vice versa – whatever floats your boat).

Luckily there are a few projects to help out with that!

Modernizr

If simple feature checking is your game, Modernizr is a pretty big help. Simply use Modernizr.video in your Javascript to detect if you can play video. Modernizr.video.ogg, Modernizr.video.webm, Modernizr.video.h264 will do well to test if you can support the various formats.

Not Modernizr

If you’d rather not use Modernizr, you can easily instantiate a new, empty video tag in your document (attached to nothing). If the video tag object you’ve created support the “canPlayType” method, then you have a legitimate video tag!

For example:

video = document.createElement('video');
video.canPlayType('video/mp4');

No Javascript

You can also take the no Javascript route. Kroc Camen setup a nice bit of HTML that does “detection” in a Javascriptless way. And then Jonathan Neal took it one step further and made a nice little “Video For Everybody Generator” application based on the “Video For Everybody” idea.

http://sandbox.thewikies.com/vfe-generator/

The basic idea is that you”ll hit the video source tags first, and if successful will start video playback on the given source. If you make it to the Flash object/embed tag, you’ve failed all HTML5 tests and you get Flash playback!

Video Frameworks

The simple stuff leads into the heavyweights. The Javascript frameworks. The “Adobe Video Widget” is as good a place as any to start. You can grab it from the Adobe Widget Browser and add it to Dreamweaver….drag and drop style!

This widget is indicative of quite a number of frameworks out there. Depending on your preference, it will prioritize HTML5 and fall back to Flash. Even though it’s branded Adobe, the Javascript player and Flash player are both created by a company called Kaltura.

The interesting thing about a lot of these frameworks, including Kaltura, is that when you peruse the source code it seems obvious that they intend you to use these frameworks as sort of a slapdash HTML5 player which falls back to Flash. In my opinion, many of these frameworks aren’t friendly to you digging around and extending things in major ways.

I had previously scouted a few different player libraries for a project – here’s my rundown of findings:

VideoJS - A free/open source player in active development. It doesn’t rely on any 3rd party libraries (like jQuery). It has an easy to extend framework, and gives you several video event listeners that you can tap into. It seems to work on all the major platforms (desktop and mobile). Skinning is pure CSS which makes things easy for separation of functionality and design
Open Video Player – Free and open source (must retain license info). The framework is quite extensible, though I did find a bug that I was able to patch. The project seems minimally active. It relies on jQuery, and the for custom skinning it relies on Javascript and CSS.
Projekktor - Open source, but if you modify the project you are expected to submit your contributions back to the project. The project relies on jQuery and skinning can be done via Javascript, CSS, and custom HTML tags.
MediaElement - Open source, and it would be swell if you link back to the project if you use it. It relies on jQuery and uses CSS for custom skins. The only problem with this project was that I felt that the Javascript API was a little confusing and hard to extend.
JWPlayer - The JW Player is probably the most popular given it’s VERY popular flash player. Unfortunately it’s not free if using commerically, and even worse the cost depends on how many sites you launch this on. It does rely on jQuery, and you can create custom skins with CSS and also purchase skins.
Sublime Video Player - Another pay player. The price is based on overall pageviews as they appear to do all the player hosting on their site. It doesn’t seem to rely on a framework, and for custom skinning they do have a CMS.

Given all these options – VideoJS seemed the very best for my purposes. I didn’t want to fallback to Flash since I was doing an iPad site and this was easy to turn off. I had a fair number of custom features I wanted to implement as well, and it was very easy to tap into it’s underlying framework and add new features like advertising and reporting.

What I really dug was the easy of skinning. It comes with a base video skin along with 3 skins which extend the base skin. It was quite easy to skin how I wanted!

All in all thought, you probably won’t need to go into the level of customization that I needed, so whatever works easiest for your purposes is best – but it seemed to me that VideoJS was helpful right out of the box!

04Sep

Live Instrumentation in Flash Part 7 – Beyond Flash

At the heart of all our audio generation lies one simple Flash feature – the sample data event.

Can we find things outside of Flash that will let us do similar things?

The answer is….of course! People have been writing desktop audio and music synthesizer software for years. Flash and AIR offers some decent functionality on the desktop, but writing something in C++ would blow Flash away of course.

The reason is that Flash has only been in the generative audio game for a few years, while people writing C++ have been at this for decades. As such, they have access to ready made filters, libraries, etc.

Take, for example, Node Beat – written with OpenFrameworks

http://forum.openframeworks.cc/index.php?topic=6070.0

Objective-C for doing iPhone development has the nice benefit of access to core libraries that Apple OSX Cocoa developers have had access to for years. The basic AudioUnit in Cocoa or Objective-C gives you the low level access you need to make things like Mobile Synth:

http://code.google.com/p/mobilesynth/

Or even a simple tone generator:

http://cocoawithlove.com/2010/10/ios-tone-generator-introduction-to.html

Basically though, you’ll find that anything that gives you low-level audio access will give you what you need. Speed is the key here, number crunching needs to be done in an instant. Flash can keep up OK, and gives you the bare minimum of what you’d need to generate sound.

What about Javascript? Well, sort of! The Web Audio API has been defined and implemented on the developer version of Chrome and there’s also a Audio Data API draft in Firefox. The basics are the same as Flash, but many more goodies are offered to help you along.

So, in the end we need speed and some basic API hooks. On the web, Flash seems the most reliable, but it sounds like Chrome and Firefox are nipping at Flash’s heels with some very cool stuff coming up!

And in the mobile world, just like the desktop world, it seems this sort of base level audio access is standard as well.

28Aug

HTML5 vs Flash Video: Choose Wisely

The title of this blog post is ALSO the title of an upcoming talk I’m giving at NCDevCon (http://ncdevcon.com/). I’ve decided that instead of starting out in power point, I’d write this up as a blog post and think about how to present this stuff on a big screen later. I do this, not only to have a nice blog post about this stuff, but because I hate looking at power presos on my computer – and I’d rather have something better to share with folks after the presentation.

I’ll write this up as a series of posts with this as the main index page. So below are the main topics in this series! I’ll fill them in with links as I progress, so stay tuned!

The War of the Buzzwords!