HTML5 vs Flash Video: Choose Wisely Part 10 – Other Options

Filed in development | flash | flash/flex | flex | html5 | ios | javascript | ui | video | web Leave a comment

Lets face it, Apple really stuck us in a bad situation.  While HTML5 is really slick on the desktop (in supported browser with supported file types), its pretty clunky on iOS.  And Flash is ALSO really slick on the desktop, but doesn’t work at all on iOS.

So what we are left with are 2 things that work really well on the desktop, but no common thing that works well on Android, iOS, and the desktop.

What can we do?

Native iOS

Simply the best option on iOS now is to sign up for a developer’s license and make an app.  iOS offers two different video player frameworks MPMoviePlayer and AVPlayer.  MPMoviePlayer is the simplest to get started with, and offers you playback through the Quicktime UI.  AVPlayer is harder to get started with and you are forced to create your own UI, but you have more control over video playback.

AIR for Mobile

What may be the best solution is to build a Flash player with HLS streams on iOS and normal HTTP or RTMP streams on Android or the desktop.  While it’s true that Flash isn’t supported on iOS, Flash Builder 4.5 can compile your existing project out to iOS and Android applications.  All Flash video is supported in AIR, Mobile AIR and the web Flash player – except for iOS.  So you’ll need the two different streams.

Corona and other 3rd Party Packagers

Corona seems to be a very popular way to create cross platform mobile apps these days.  There are other packagers as well like Titanium, Appcelerator and more.  Last I looked they don’t handle video, or don’t handle video very well.  Probably the best is Corona which does allow video to play, but it needs to launch the video in a modal window or fullscreen, and you have little control over it. As cool as these solutions are, I don’t believe that they handle video all that well.

HTML5 vs Flash Video: Choose Wisely Part 9 – The Verdict

Filed in flash | flash/flex | flex | html5 | ios | javascript | ui | video | web Leave a comment

So what wins?  HTML5 video or Flash?  What should you use, what’s better?

The answer to this is that it depends on your project and what platforms you are targeting.

Use HTML5 video for simple playback

For short form video that contains no in-stream advertisements (that’s fancy talk for pre, post, and midrolls) its extremely easy to use HTML5 now.  If your content is at most a few minutes long and you want it to work on any browser and any device you should use HTML5 with a Flash fallback (or vice versa).

Your content should be MP4/h264 so that it will work on iOS, Flash, Chrome, and IE9.  If you feel feisty, you can make a copy of your content as OGG to play in Firefox, but you can easily handle Firefox with Flash video.

Use Flash for in-depth Ad and Tracking support

Adding ads can be a little trickier. It’s much easier to do this with Flash, but if your content is short, all you’ll most likely need is a short pre-roll.  If doing this with HTML5, the experience may be a bit clunky, but at least it will work on the iPad.

It stinks a little more as you add tracking (or other features).  Do you want to duplicate your programming efforts in both Flash and Javascript?

One suggestion might be to implement a Flash player with no control bar and to only handle simple playback of video.  Do all your graphics, reporting and ad logic in Javascript, and just use the Flash player in the same way you’d use a video tag.  In this way, you’re using two things that are pretty interchangable.

All in all, you’ll be most successful if you implement these features in Flash as Flash has the most support for this type of thing.  HTML5 is certainly possible, but hard – you’ll need to power through it, especially if you need it to work on iOS.

Use HTML5 for iOS
If your only options are Flash or HTML5, obviously you should use HTML5 for playback since Flash isn’t supported!

Use Flash for Long-Form Streaming Content

This is a hard truth – HTML5 can’t do streaming.  As such, it can’t do long-form content without completely bogging down end user’s browsers.

For playback on Android devices, you can use your same Flash player for long-form streaming playback (just make sure to make it mobile and touch friendly).

For iOS, you’re pretty much out of luck.  You can run streaming long form content through iOS, but unfortunately that same content won’t work on normal browsers.  If you want playback on both desktop browsers and iOS devices, you’ll need to double your efforts and develop players for each scenario.

HTML5 vs Flash Video: Choose Wisely Part 8 – Advertising and Reporting

Filed in development | flash | flash/flex | flex | html5 | ios | javascript | ui | video | web Leave a comment

Before we conclude this series, I should mention something nobody likes – but they are an absolute necessity if content producers are going to put their video online in the first place.

Those things are ads and reporting.

Content owners want to monetize their video, and be able to track many aspects of it’s usage to make better decisions about video you’re watching.

With Hulu, Netflix, and many networks putting first run television online it seems that web video is gaining on what we watch on cable TV, and we may see a day soon when the web is significant enough to eat into cable tv’s revenue model such that we’ll networks and content producers who want to thrive can’t afford to just be on one or the other.

As ad revenue from cable tv and network broadcasts are shifted, owners will want to make sure this money is JUST shifted and not lessened.  So even today, folks are experimenting with longer ad breaks to see if they can get away with the same video experience you see on TV.

To accomplish this, we need some good 3rd party libraries from ad companies and reporting companies, as well as decent support on the actual device, browser, or plugin to facilitate these advanced needs.  If we can’t come up with creative solutions to accommodate ads and reporting, companies will simply not come – and this can make or break a platform’s usefulness for serving video.

3rd Party Library maturity

I’ve been working with Flash video extensively for a few years now, and in that time I’ve seen 3rd party frameworks and libraries delivered to us from various ad and reporting companies (many revisions in fact).  Flash seems to have a lock on maturity of 3rd party libraries for handling these things.  Flash has almost maniacal support for every aspect of playing video – something you won’t see from HTML5, or even iOS (from what I’ve seen so far).  In truth, this is why Flash is difficult to work with for video newcomers.  But, at the same time it offers extreme control to allow almost anything we need.  We can offer midrolls, prerolls, postrolls, companions, overlays and more.  We can also track video buffering events, stream transitions, and many more minute details.

HTML5 video, on the other hand doesn’t have this level of control – especially in iOS given the playback obstacles we’ve discussed.  Given this – many 3rd party libraries are still figuring out how to offer the same level of ads and reporting we can obtain from Flash players.

One company I work with just recently started offering midroll ads, where before their Javascript/HTML5 video solution wasn’t mature enough to do this.  They seem to be coming along just fine, but won’t be caught up to Flash anytime soon!  See part 6 for my experiences working with mid roll ads.

Advertising with MP4 – At Least that’s Easy!

I’ve talked about iOS streaming for long-form content.  It uses Apple’s HLS, and is pretty much incompatible with Flash or Silverlight.  As a result, content owners will need to completely re-encode their streams for Apple devices – that is IF they want it to be streaming.

Many advertisers these days, however, will serve up their ad creatives as mp4 files encoded with h264.  This works on Flash, and it ALSO works on iOS, Chrome, and IE.  This means that if these advertisers wanted to use their same ads on iOS, they’re free to do so without have to re-encode the files!

Companion Ads and Banners

The same can’t be said for companion ads, however.  Companion ads are static or animated images that you see outside the video player area (usually in another div on the page).  The problem is that these ads, in my experience, aren’t static images and usually animated Flash.  This means that advertisers and content producers need to work together to create new companion creatives and serve them up appropriately for Flash or iOS.

HTML5 vs Flash Video: Choose Wisely Part 7 – What’s Happening on the Server Side

Filed in development | flash | flash/flex | flex | html5 | ios | javascript | ui | video | web Leave a comment

So we talked about how Apple supports both progressive and streaming video with their HTML5 video tag.  We also briefly discussed how Flash Media Server serves up its OWN streams.  What about Apple’s HTTP Live Streaming?  What kind of magic is that?

Turns out that its quite simple.  Take a video file, after encoding it for HLS you’ll end up with a directory of files.  First off there is a table of contents file.  This table of contents files points to MORE table of contents files, and these point to video file segments.

The video segments are most likely h264 encoded segments.  Each one has a “ts” file extension (myfile.ts).  ”TS” stands for transport segment.  Each segment will most likely be small, about a megabyte or so and contain several seconds worth of video (this is entirely left up to the encoder settings).

The table of contents files sound a little hacky, but they work well.  Apple has gone ahead and adopted m3u8 playlist files for streaming video.  What’s that you ask?  What’s m3u8?  Well, if you crack open Winamp or iTunes, you can create a music playlist.  Both m3u and pls files are pretty standard for holding playlists of songs.  The m3u8 spec is an enhanced version of this (and still used for music playlists!). It makes sense though, you’re really just assembling a playlist of video segments.

I also mentioned playlists in playlists didn’t I?  Well, the master m3u8 file will typically hold several different bitrate references (for if a user wants high definition, low definition, or anything in between).  This file will hold references to other m3u8 files and list what their bitrates are in bytes per second.

The reference m3u8 files will be the ones that actually list the segments, examples are listed below.

 

First is an m3u8 file referencing OTHER m3u8 files for the different bitrates:

#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000

http://ALPHA.mycompany.com/lo/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000

http://BETA.mycompany.com/lo/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000

http://ALPHA.mycompany.com/md/prog_index.m3u8

#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000

Next up is an example of the actual m3u8 file that holds all of our file references.
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:10, no desc
fileSequence0.ts
#EXTINF:10, no desc
fileSequence1.ts
#EXTINF:10, no desc
fileSequence2.ts
#EXTINF:10, no desc
fileSequence3.ts
#EXTINF:10, no desc
fileSequence4.ts
#EXTINF:10, no desc
…..
#EXT-X-ENDLIST

Live Playback Limits on HLS

In the example above, notice the line #EXT-X-ENDLIST.  This command marks the end of the file.  This an important piece of info!  It tells Quicktime that there will be no more segments after this, and what we have in the file is the entire stream.

What if this line didn't exist?  Well, it turns out that the absence of this line indicates to Quicktime that the m3u8 file is a live stream.  In this regard, the file has no end.  With a VOD (video on demand) stream, Quicktime only needs to query this table of contents once, and the player is secure in it's knowledge that there is nothing else to load.  For a live stream, the m3u8 file is repeatedly queried to find out what the latest segments are so the player can keep up with the most recent segments.

This is a nifty little system -  but there's a problem.  Typically, a live event will allow the end user to watch in a 30 minute or so sliding window.  The user can't seek backwards more than 10 minutes or so.  These older segments are dropped from the m3u8 file such that the m3u8 file only contains a handful of segments.

The problem is if you need a long stream.  My last project had a need to run clips from a large 14 hour stream.  When playing through HTML5 on a web page, the stream broke down after it grew to 40 minutes or so.  The problem was that the growing number of entries referencing transport segments made the file grow exceptionally large after a while.  If I recall correctly, a 10 hour stream's m3u8 file was around 600kb for me. Multiply that by 3 for the other bitrates I had.  So eventually, Quicktime was constantly querying over a megabyte of playlist info - and it had trouble keeping up with playback!

I came up with the solution of having our encoding team put a fake end on the m3u8 file even though it was live.  This caused the file to only be loaded once even though the live stream was continually playing.  It doesn't work for a lot of situations since you're not able to play the most recent stuff since you loaded the file, but it worked for our specific needs with our site!

Miraculously, it seems that this is only a problem in HTML5, and not a problem if you're just running Quicktime natively or in an app.

Other Streaming Services

I don't have any experience with Silverlight, but IIS's Smooth Streaming functions pretty much the same way.  Instead of an m3u8 file, Smooth Streaming has an XML format for it's table of contents.

We discussed how Flash Media Server worked in a previous post in this series - but to recap, it can generate these segments on the fly and serve them over RTMP, HTTP and for Flash or iOS.

Playback in Flash doesn't have the same length issues that iOS HTML5 has for live streams.  It plays things back fairly well!  HLS and SmoothStreaming though, have a pretty important feature on Flash - bitrate switching happens automatically between the server and the player.  You don't need to program anything extra in your code to make bitrate switching work - and it does work very smoothly!  You'd be hard pressed to notice the transition.

Flash recently implemented a smooth stream switchover as well.  You as a developer, must measure bandwidth and make the switch yourself (though OSMF will do this automatically for you).  Once you call the "play2" command, it will START making the switch, but it will wait until the next video keyframe to make the switch so its as seamless as HLS and Smooth Streaming make it.

HTML5 vs Flash Video: Choose Wisely Part 6 – Apple’s HTML5 vs Normal HTML5

Filed in html5 | ios | javascript | ui | video | web Leave a comment

If you are interested in pursuing HTML5, I imagine the reason why you are doing so is to support iOS.  As we all know, iOS doesn’t support Flash, so to play video we’ll need to take advantage of HTML5 video.

HTML5 is fairly normal via Safari on OSX.  Safari is just another browser, however the video tag is handled by Quicktime.  It looks like we’re trading one plugin for the other here!

There’s probably nothing wrong with using Quicktime here in OSX.  If Apple chooses to use one of their oldest video technologies to playback video in their browser on their OS, no big deal!  It really probably a matter of semantics anyway – could you consider the h264 decoder in Chrome a plugin?  Yes and no – its a fine line.

On Windows, however, if you use Safari, you’ll need the Quicktime plugin installed!  So, here we’re REALLY talking about a plugin.  In this situation its both a plugin AND HTML5 at the same time.

As it is Quicktime, we can use the video tag to play whatever Apple can normally play with Quicktime.  In fact, we can play Apple’s HTTP Live Streaming (HLS).  I think this leads to a great deal of confusion and is a pretty big bait and switch on Apple’s part.

First they tell us “who needs Flash?!  Its slow, buggy, and HTML5 can replace it!”.  Then they live stream their developer event in HTML5, but surprise!  Only Apple’s brand of HTML5 can stream it, because it’s really Quicktime under the hood.  Even more than that, streaming video isn’t even part of the HTML5 spec!  So, it should be quite awhile before we’ll see streaming content come to browsers natively.

Some Apple playback quirkiness

I really haven’t seen much in terms of weirdness on Safari OSX.  I haven’t had to play with it much, but the video tag works pretty much as advertised here (since it’s webkit, it’s just like using Chrome in fact).

But that’s not really what matters – on OSX you have the option of Flash too.  On iOS, you either need to go HTML5, or write an app.  There’s no other option!

So, how does the video tag fare?  There are some major things here to be aware of.  Apple has decided that we can’t be trusted with the autoplay feature of the video tag, so they have disabled it.  This means that you cannot automatically play a video after a page loads.  There needs to be some sort of user intervention here to kick off the video.  This will usually involve a play button overlaid on top of the video that the user clicks to start playback.  It’s a little heavy handed in my opinion, but the reasoning behind this is to not let malicious people use your data for things that you don’t want – like ads.

Unfortunately, this prevents some major barriers for advanced video applications. Something as simple as playing midroll ads (ads that appear in the middle of content at a specified ad break) is made complicated.

Midroll Ads

See, one problem with midroll ads is that you must stop your content, flip over to the ad, and then after the ad, resume content again.  I’ve done this in Flash and am currently doing this in iOS/Objective-C.  In my experience, the best way to do this is to use two separate video objects.  With your main content in one video object, when an ad comes on, you can pause this and spawn a new video object overlaying it on top of your main content.  Play your ad in the new video object, and then when you’re done, kill the object and resume your main video.

With this game plan, you don’t need to suffer through a rebuffer of your main video content after your ad.  If using the same video object, you’d need to not only load the video again, but automatically seek to the spot right after the ad break.

Using two video objects like this isn’t really feasible.  For each video tag you use, you’d need to tell the user to manually click play on each one.  When playing an ad – you’re adding insult to injury like this!

Furthermore, another quirk I’ve seen is when you hide a video tag on iOS using CSS and either hidden or display:none.  This consistently crashed my iPad browser when I attempted this.  So maintaining 2 video tags, but hiding one isn’t an option!

In essence, midroll ads seem to be stretching the capabilities of HTML5 on iOS!

Automatically Seeking

Yet another quirk I’ve seen is attempting to automatically seek into a stream a certain number of seconds (which you’d need to do when returning from an ad, or just playing a clip of a larger video).  The problem is that when you initially load a video resource, it can take some time before you can interact with it and tell it to seek.

The initial run up to playback includes the video object becoming ready, the asset being loaded enough to start playback, and finally as more of the asset is loaded, and finally it becomes seekable.  The problem is that this process can take a second or so.  In the MEANTIME you can’t interact with the video and tell it to seek.  You just have to sit there and watch it playback a second or half second of video.  What should be an automatic seek becomes an automatic seek with a small video glitch at the beginning.

Player Chrome

Your customers like customization don’t they?  Some would like some nifty custom graphics to show up wouldn’t they?  A custom lime-green playback button to match their brand maybe?  It’s completely possible on the desktop with HTML5 – simply hide the controls and create your own, positioning them over the video with CSS.  The problem is that this is a bit problematic in iOS.  See – in my experience, you can’t overlay things on top of a playing video.  Well actually you can – but the video window will suck in all your mouseclicks.  This means that while you can overlay your own controls on iOS, they probably won’t function.  Nothing can obscure the video and be perfectly functional!

Furthermore, you always get the same UI when you run either on an iPhone or an iPad in fullscreen mode.  You get Quicktime’s native controls – your page is gone (in the background) and you are left with nothing but the video playing in the Quicktime application.  Fortunately enough, Javascript can still operate in the background and control your video.

Video API Events

I’ve also noticed that events can be a little unreliable on iOS, like listening for “onCanPlay” or when the player is ready.  I’ve had to setup a timer that fires every half second or so to check for readiness and playback state in my own project to accommodate this problem.

HTML5 vs Flash Video: Choose Wisely Part 5 – Streaming vs Progressive

Filed in development | flash | flash/flex | flex | html5 | ios | javascript | ui | video | web Leave a comment

We’ve discussed in part 4 of this series that Flash can get a little complicated.  It’s true, but it mostly boils down to the different video formats you can play.  Let’s talk about those formats.

Progressive Video

If you are clueless about what the difference is between streaming and progressive video – chances are the video you know and love is said to be “progressive”.  What does this mean?

Well, if you hit a single video file on a server somewhere and start downloading it, you can actually watch the video before it finishes downloading.  You are using it “progressively” in that you are watching it while download is in progress.  This happens in much the same way as if you had an image loading on a slow connection and you see it start at the top and fill in as it gets loaded.

One of the problems I’ve seen from progressive video files (at least when playing from Flash) is when the MOOV Atom is placed at the end of the file by the video encoder.  Think of the MOOV Atom as a block of data in the file that holds info about your video…like duration, bitrate, frame rate, encoding specs, etc.   If this is place at the end of the file, the player doesn’t know the details of your video file, and won’t play it UNTIL it gets this block of data.  Since it’s at the end of the file, users need to wait until the entire video file is downloaded so that the video player knows exactly what it’s playing.

Streaming Video

So, speaking progressively, video is one large video file that you download and cache to your computer (or at least that’s whats happening behind the scenes in your browser) for playback.

On the other side of the coin, we have streaming playback.  Typically a streaming file will be served from some sort of media playback server as a lightweight container file that serves as a table of contents for many smaller video segment files.  The lightweight container file will be loaded once up front (or continually if it is often updated like when you are connected to a live stream).  Your player will then be smart enough to know what segment files are needed to playback video.  It will go out and grab these segments, and then throw them away when you’re done.  A player will keep a “buffer” that may or may not be able to be set by the player of how many seconds of video to keep (and therefore how many video segments).  If you have video buffered, you can easily seek back and forth to these points instantaneously.  If on the other hand, you seek outside the buffer, the player will have to read the table of contents container and lookup which video segments to fetch and buffer for you.

The Flash Media Server will hide all the complexities from you, though.  Usually, you’ll just connect to a path that starts with rtmp:// and ends with myfile.mp4.  For all YOU know it’s just a file, not chunked up into tiny segments.  And actually it sort of IS one big file.  Flash Media Server will just tranform this file into a stream for you on demand, so its easy to manage and still works like a stream.

Flash Media Server, in recent days,  has actually started supporting streams over HTTP for Flash and iOS.   This means that if your company doesn’t want to deal with serving stuff over RTMP, you can just use HTTP streams and Flash 10.1 (and OSMF if you don’t want to program support yourself).  You can use your same server and files to send a stream to iOS!  Kind of a timesaver if you need to support both, instead of managing all 2 separate servers/encoders.

Use Cases for Each

Why use streaming vs progressive?  Well the major reason is what we in the business call “long-form video”.  Long form video is typically 20 minutes to a few hours long, while  ”short-form video” is under a minute or a few minutes long.  YouTube clips are perfect examples of short form video.  And they are also decent examples of why the overhead of having a streaming service and player is completely unnecessary.  If it’s only a few megabytes, then it really doesn’t take long to load and cache the whole video does it?  And in the end, when you close your browser you’ll only have a few MB from it in your cache or in your memory.

Long-form, on the other hand, can be several hundred megabytes or even a couple of gigabytes depending how long the video is and what quality it was encoded at.  Not only will you be left with a behemoth of a file in your RAM or cache, but it you can’t seek forward to skip over pieces until that much has been downloaded.

Another benefit of streaming is the ability to switch between bitrates or cameras.  If your video player thought that it was having difficulty keeping up with your HD file, ideally, you’ll switch to a lower resolution video.  It’s REALLY difficult and pointless to do this on a progressive file.  You’d have to latch on to a different file, and then seek to the position you left off at….which means you’d have to progressively download the file up to the point where you left off which could take several minutes!  With a stream, all you have to do is tell the server to start sending you different file segments.  It will do so, and the player will never be the wiser, cause all it sees is new segments coming in – it doesn’t really matter if they are from a different source file.

So that coupled with a server’s ability to offer DRM is a huge reason why TV shows and movies are all streaming.  While there are many DRM solutions available, one fact is that when you segment all your files like this (even without DRM), its a pain in the butt to download and assemble them all if you were so inclined to steal content.  Flash Media Server obfuscates the stream even more by serving it over RTMP.  Since it’s an entirely different protocol than HTTP, most standard web dev tools won’t even see the files come through and its even more difficult to steal!

Contrast this with YouTube.  Most of the time, I can go in and open up my Chrome developer tools, watch a big file come in, and copy the source URL.  I can then download the file and rename to a mp4 or flv file and then play it on my desktop.  BAM!  I just stole content and it took 30 seconds.

Something in between (pseudo-streaming)

You may have noticed recently in YouTube, where it shows you a red line in your progress bar to indicate your downloaded buffer.  In years past, you couldn’t seek beyond this.  Well, now you can!  How is this possible?

It’s what we call “pseudo-streaming”.  Basically it acts like a progressive download, but with trickery it can have the main benefit of streaming which is to seek anywhere you want in the file regardless of how much downloaded.

It’s do-able because many servers (even if just a simple Apache server running PHP) allow the browser request to specify a byte range.  Typically when you make a file request from your browser you ask for the file, and it ASSUMES you want all the bytes of the file starting at the beginning of the file.  What pseudo streaming does is to ask for the file, but ask for the bytes at a certain position in the file and not to start at the beginning.

What’s Possible in HTML5 vs Flash

Pseudo Streaming is actually very popular these days because the HTML5 video tag supports it natively.  To use it, you as a developer don’t have to do a thing beyond allowing your user to seek (as well as having a server that supports byte ranges).  Contrast this with Flash – where you’d need to jump through hoops to get this working.  Flash doesn’t support byte ranges with video or audio, which means you’d have to load the bytes MANUALLY, and then append the bytes onto the active video stream.  Not only does this suck to code, but appending bytes to a stream wasn’t even supported until Flash Player 10.1.

Of course, progressive is supported in both HTML5 and Flash.  Flash supports a few different encoding types including mp4/h264 just like HTML5 normally does (depending on the browser).

Here’s the bombshell though…HTML5 does not support actual streaming, and there is no official proposed spec for it!  I bet you don’t believe me because of a certain large, fruity corporation.  It’s true though!  More on that in the next part.  But the long story short here, is that I can’t imagine recommending HTML5 for long form content, doing 2 hour movies with a progressive download would be beastly (note that there may be clever ways, I just don’t know what they would be beyond artificially breaking up the video files and leaving gaping holes in playback while loading the next segment).

HTML5 vs Flash Video: Choose Wisely Part 4 – How to: Basic Flash Playback

Filed in development | flash | flash/flex | ui | Uncategorized | video | web Leave a comment

The interesting thing about Flash is that before some recent projects, it was much harder to get a basic player going.  Video playback is composed of three basic elements: the NetStream, the NetConnection, and the Video object.

To start playback, you’d need to create a NetConnection, connect to a server if streaming or null if a normal progressive file, start a new NetStream with the NetConnection, and finally attach a the NetStream to a video object.

connection = new NetConnection();
connection.connect(null);
stream = new NetStream(connection);
stream.client = this;
video = new Video();
video.attachNetStream(stream);
stream.play("myfile.mp4");
addChild(video);

Not only that, but you’ll need to add the video to the stage, and manage the aspect ratio of the video properly.  You’ll also need to publish a SWF, and embed it on the page.

It’s a heck of a lot more complicated than a simple HTML video tag, isn’t it?

 

Adobe’s Open Source Media Framework

A couple of years ago, Adobe set out to make things less complicated – to create a framework that can simplify video playback as well as provide an easy plugin architecture for ads, reporting, and more.  This framework can handle any video type that Flash can handle without you needing to worry about what type of video you’re playing.

Let’s check out how to get a simple video up and running:

playerSprite = new MediaPlayerSprite();
playerSprite.media = new VideoElement( new URLResource( "myvideo.mp4" ) );
addChild( playerSprite );

You’ll still need to compile and embed on a page as well.  This is only 3 lines of code as opposed to the 8 without OSMF.  Its not necessarily the amount of lines of code I’m concerned with, it’s that to really know what’s going on, you need to know what the NetStream, NetConnection, and Video objects all do.  It can take a bit of research to put all those pieces together!  In other words, OSMF is more intuitive to me – you create a playback sprite, and then assign some media to it

The extra good thing is that we can handle most file types.  We’ll get back to this in a bit, but as you use streaming media, worry about hooking up a control bar, things get exponentially more difficult.  OSMF makes this nice and tidy for you.  I must warn you though, as you start doing weird things with your video playback, you’ll need to learn the language and architecture of this framework.  Normally though, things are quite simple to use through the framework’s API.

Strobe and Flash Media Playback

Can it get easier maybe?  Answer is yes!  All levels of ease seem to be taken care of – from the insanely detailed and comprehensive to having a ready made SWF pre-compiled for you and ready to use!

The next level here is the Strobe Media Playback project.  This project is already compiled and ready to go…ready to be embedded on a page.  It comes with some external configuration and external skinning options if you need to change anything around.  In addition to the compiled SWF, it comes with all the source code to the player as well – so if there’s anything you need to change with code, you’re welcome to make the change and compile again!

Can it get easier?  Yes!  There’s Flash Media Playback.  If you follow the link, Adobe brands it as “Instant Video Playback on your Website”.  Flash Media Playback is an extension of Strobe Media Playback – however, you don’t download any files.  Adobe hosts the player, you simply grab the embed code and perhaps make your own configuration file that you point it too to handle the customization aspects you need.  Adobe offers a configuration and test playback page for you to get started.

Other 3rd Party Players

There are plenty of players out there in the same vein as Flash Media Playback, and most have probably been around longer!  The most popular ones that I’m aware of are JWPlayer, FlowPlayer, and the BrightCove (which has an entire video publishing platform)!

Typically you’ll embed these on your page and pass in one or several “flashvars” to tell it what video to play.  I don’t have much experience with these 3rd party players, but most cost money if you’d like to use them commercially.

 

As you can see, using video in Flash can range from drop dead simple to mind boggingly complex.  More on why this is in the next part of this series.  What’s good is that there are several projects you can take advantage of to get simple playback on your website with no or minimal Flash experience….without even having to buy or touch a compiler even!

HTML5 vs Flash Video: Choose Wisely Part 3 – Extra Help with HTML5

Filed in html5 | javascript | ui | video | web Leave a comment

In part two we discussed a good deal of how to use the video tag and how (if we’re ambitious) we can code up our own controls.  We also discussed the fact that the HTML5 video tag might not be an option if you’re using an older browser.  And maybe Firefox/OGG video isn’t part of your workflow.  In either case, you need to opt out of HTML5 and opt into Flash (or vice versa – whatever floats your boat).

Luckily there are a few projects to help out with that!

Modernizr

If simple feature checking is your game, Modernizr is a pretty big help.  Simply use Modernizr.video in your Javascript to detect if you can play video.  Modernizr.video.ogg, Modernizr.video.webm, Modernizr.video.h264 will do well to test if you can support the various formats.

Not Modernizr

If you’d rather not use Modernizr, you can easily instantiate a new, empty video tag in your document (attached to nothing).  If the video tag object you’ve created support the “canPlayType” method, then you have a legitimate video tag!

For example:

video = document.createElement('video');
video.canPlayType('video/mp4');

No Javascript

You can also take the no Javascript route.  Kroc Camen setup a nice bit of HTML that does “detection” in a Javascriptless way.  And then Jonathan Neal took it one step further and made a nice little “Video For Everybody Generator” application based on the “Video For Everybody” idea.

http://sandbox.thewikies.com/vfe-generator/

The basic idea is that you”ll hit the video source tags first, and if successful will start video playback on the given source.  If you make it to the Flash object/embed tag, you’ve failed all HTML5 tests and you get Flash playback!

Video Frameworks

The simple stuff leads into the heavyweights.  The Javascript frameworks.  The “Adobe Video Widget” is as good a place as any to start.  You can grab it from the Adobe Widget Browser and add it to Dreamweaver….drag and drop style!

This widget is indicative of quite a number of frameworks out there.  Depending on your preference, it will prioritize HTML5 and fall back to Flash.  Even though it’s branded Adobe, the Javascript player and Flash player are both created by a company called Kaltura.

The interesting thing about a lot of these frameworks, including Kaltura, is that when you peruse the source code it seems obvious that they intend you to use these frameworks as sort of a slapdash HTML5 player which falls back to Flash.  In my opinion, many of these frameworks aren’t friendly to you digging around and extending things in major ways.

I had previously scouted a few different player libraries for a project – here’s my rundown of findings:

  • VideoJS - A free/open source player in active development.  It doesn’t rely on any 3rd party libraries (like jQuery).  It has an easy to extend framework, and gives you several video event listeners that you can tap into.  It seems to work on all the major platforms (desktop and mobile).  Skinning is pure CSS which makes things easy for separation of functionality and design
  • Open Video Player   – Free and open source (must retain license info).  The framework is quite extensible, though I did find a bug that I was able to patch.  The project seems minimally active.  It relies on jQuery, and the for custom skinning it relies on Javascript and CSS.
  • Projekktor - Open source, but if you modify the project you are expected to submit your contributions back to the project.  The project relies on jQuery and skinning can be done via Javascript, CSS, and custom HTML tags.
  • MediaElement - Open source, and it would be swell if you link back to the project if you use it.  It relies on jQuery and uses CSS for custom skins.  The only problem with this project was that I felt that the Javascript API was a little confusing and hard to extend.
  • JWPlayer - The JW Player is probably the most popular given it’s VERY popular flash player.  Unfortunately it’s not free if using commerically, and even worse the cost depends on how many sites you launch this on.  It does rely on jQuery, and you can create custom skins with CSS and also purchase skins.
  • Sublime Video Player - Another pay player.  The price is based on overall pageviews as they appear to do all the player hosting on their site.  It doesn’t seem to rely on a framework, and for custom skinning they do have a CMS.
Given all these options – VideoJS seemed the very best for my purposes.  I didn’t want to fallback to Flash since I was doing an iPad site and this was easy to turn off.  I had a fair number of custom features I wanted to implement as well, and it was very easy to tap into it’s underlying framework and add new features like advertising and reporting.
What I really dug was the easy of skinning.  It comes with a base video skin along with 3 skins which extend the base skin.  It was quite easy to skin how I wanted!
All in all thought, you probably won’t need to go into the level of customization that I needed, so whatever works easiest for your purposes is best – but it seemed to me that VideoJS was helpful right out of the box!

Live Instrumentation in Flash Part 5 – Voice Synthesis

Filed in development | flash/flex | flex | music | ui Leave a comment

So far, we’ve discussed how to create tones, notes, and chords.  We’ve also gone into several different algorithms to change how a digital instrument sounds.  Changing how our digital instrument sounds is a lot more than simply changing the algorithm that defines your sound wave!  You may have noticed that given the several algorithms we covered in part 2 of this series, the sounds didn’t change all that much.

What can we do to create some more variety?

Attack, Decay, Sustain, Release

In electronic synthesizers, there are the concepts of “attack”, “decay”, “sustain”, and “release” (commonly abbreviated ADSR).  The easiest way to explain this is to imagine yourself hitting a key on a piano.

The moment you hit the key, the string is hit by the hammer by the piano.  This initially creates a loud tone.  The initial tone is much louder than anything that happens moments after.  This initial loudness is called the “attack”.  As the initial loudness wears off, the sound draws down to the normal volume as you hold the piano key.  This drawdown is known as the “decay”, and for the rest of the duration that you’re holding the key is known as the sustain.

Once you release that piano key (on a real piano), there would be a small amount of sound coming out as the string is still vibrating.  This will eventually fade away – but this period is known as the “release” period.

Envelopes

Attack, Decay, Sustain, Release is a type of envelope.  At a basic level, an envelope is controls volume on a waveform as the cycles continue.  There are more types of envelopes, but the ADSR envelope is the most commonly known.

Some synthesizers break up the phases even more.  Consider the DAHDSR envelope.  This starts with a delay phase.  This is the phase before the attack – a phase where you don’t hear any audio, or much at all, in the initial lead up to the loudness that is the attack phase.

Then comes the “hold” phase.  This is meant to prolong the time between the attack and release.  It keeps the volume high!

And then the rest continues with the decay, sustain, and release.

Different types of synthesizers take different envelope approaches.  Some even let you create your own – however it seems that the simple ADSR is the most common.

Shortening the Sustain

One thing I’ve found is that a sustain that keeps going and going and going while you press a key, sounds kind of unnatural.  It might be appropriate for something like a pipe organ, but if you’re trying to replicate the percussiveness of a piano or a bass guitar pluck, you don’t want something that goes on forever.

Keeping it short and sweet is the perfect key to making something sound natural!  I’m still experimenting with envelopes to try to accomplish this effect without just cutting off the waveform.  Overall though, I’ve found that limiting a sustain to a few hundred milliseconds is the perfect solution to making a more natural sounding instrument – especially if you want something percussive like a bass pluck or piano key.

Harmonic Overtones and other Imperfections

Imperfections in the note quality are a good way to make a computer generated note sound more natural.  Perhaps the tone could be off a couple steps in the frequency, however there are other ways!

One such way is to give a note a harmonic overtone.  Remember back in part 4 when we went into note relations?  We talked about octaves, where an octave higher will produce the same key/tone, but is at a higher pitch.  An octave higher or lower is in perfect harmony with the root note.

With harmonic overtones, we layer several octaves on top of the note.  This gives a more natural sound.  Many instruments in the real world will have the first 2 overtones be in perfect harmony.  In this regard, starting at middle A of 440hz, we’d have overtones of 880hz, 1320hz, and 1760hz.  All these notes playing at the same time to produce something more natural!

Another factor in real world instruments is that these overtones (especially the third and fourth) may not be exactly the same key in an octave, not in absolute harmony.  These higher overtones might be a little sharp or a little flat.  These nuances contribute to the instruments unique sound.  In fact, if you’ve ever seen the flare on the end of the trumpet, it’s not to get more volume, but to correct the harmonic overtones to be closer to absolute harmony.

We can absolutely replicate this in our digital instruments!  We can mix harmonic overtones together when playing a single note, and even make variations to avoid absolute harmony and give our instruments some character.

Filters, Modulations, and more

We’ve only scratched the surface of creating new and exciting voices for our instruments.  We can pass our tone through a filter, do frequency modulation, and more.  I’m still exploring, so this is best left for another time!

Live Instrumentation in Flash Part 3 – Generating a Tone

Filed in flash | flash/flex | flex | music | ui | web Leave a comment

Basic Sine Wave Generation

OK!  Lets step into the wayback machine and go back to part 1 of this series.  We were talking about building audio samples with a byte array

 var bytes:ByteArray = new ByteArray();
for (var c:int = 0; c < 8192; c++) {
bytes.readFloat(number);
bytes.readFloat(number);
}

I never DID tell you guys what goes in number, did I?

Well, that’s where we can start making something listenable (and not the white noise we made before).

Lets make some changes:

for(var c:int = 0; c < durationinseconds*44100; c++) {
var number:Number = Math.sin(c * 2*Math.PI/44100 * 440);
bytes.readFloat(number);
bytes.readFloat(number);
}

OK, so what’s going on here? We’ve introduced, first of all, “durationinseconds”. We multiply by 44100. Since there are 44,100 samples per second, all we need to do is multiply by the number of seconds we want to produce, and generate that many samples.

Next up is to generate a sine wave. Why a sine wave? Well, think about your high school algebra class. There are all sorts of mathematic equations, and you can graph any of them. Most basic is a simple line.

a line - it starts infinitely low, and ends infinitely high, never repeating

The problem with a line, is that it keeps going up and up and up and up. That’s really not what we want here. What we want is something that goes high, comes back down, goes up again, and can do that forever.

We COULD utilize a little programming, and make a line, but at a set high point, reset and make it start at the low point again.

a line that cycles from going up and up, but resetting at 0 again

This is actually a good way to produce a sound, but lets talk sine waves for now.

The sine function to produce wave cycles is fantastic. The line can go up and down all day as you continually increment, but not only that, it’s curvy! So it goes up and down in a very smooth way.

Goes up and down repeatedly (forever) and is curvy

The thing about sine waves is that since they repeat themselves, they can ALSO be measured like a circle – in degrees. If you go 360 degrees, you’re right back where you started from. The bummer about trigonometry though, is that everything is measured in radians.

So 360 actually is equal to 2 Pi. 180 is Pi, and 90 is Pi/2. So above, we took 2 Pi and divided by 44100. We’re basically making a baseline here and saying that one full revolution through a sine wave is equivalent to one second. We multiply this by our iterator (c), to make the individual samples for each data point.

This would actually produce a SUPER low tone (I doubt you’d be able to hear it). I’m going to go ahead and ALSO multiply by 440 to produce something listenable.

Frequency and Amplitude

So let’s talk about what we did when we multiplied by 440. The effect we had was to take that super low tone (the one that takes a whole second to go a complete cycle) and make it go 440 cycles in one second.

How often something cycles is known as frequency. And that’s the thing about frequency in sound. If the frequency is too low, that is it doesn’t cycle fast enough, our brain doesn’t put together that its a repeating pattern, and it can’t latch on to the fact that its an audible tone. If too high, our ears can’t actually discern the signals either.

440 cycles is pretty much smack in the middle of what we are comfortable hearing. There’s a LITTLE more to this, and I’ll tell you more in part 4.

Anyway, all those cycles going so fast come together sounding like one tone. The more cycles per second, the higher the tone.

When you’ve heard about frequency, you’ve also heard about amplitude. In terms of audio, amplitude is just volume. It’s pretty easy. If you want a volume that’s 1/4 as loud, just multiply that “number” variable by .25 (OK I’m lying, volume is actually logarithmic, but I don’t feel like getting into that whole thing right now).

Here’s a nice little demo to allow you to play around with frequency and amplitude on a sin wave.

Different Types of Cycles

Like we discussed, sine waves are curvy… When we generate the tone, its sounds nice (if not a little whiny). There are other ways to go. I eluded to this before (with the line that we keep resetting in a cycle).
You can make things sound edgy too! Like you could make a square wave. I’m going to start giving examples now, and copy over some of what I have in the Flashamaphone project.

First of all though, lets simplify things and pop a bunch of math into a phase variable, like so:

var phase = c * 2*Math.PI/44100 * 440;

There! Now we don’t have to write all that stuff out each time. So let’s revisit how to do a sine wave:

// loop
number = Math.sin(phase);
// end loop

Goes up and down repeatedly (forever) and is curvy

Next, lets try a square wave

// loop
number = Math.floor(Math.sin(phase));
// end loop

It sounds very 8-bit and harsh.

At this point, I can start making up my own terminology – and I came up with a stepped wave. It’s half-way between harsh and smooth, between a sine wave and a square wave.

// loop
number = Math.floor(Math.sin(phase)*4)/8 - Math.floor(Math.cos(phase)*4)/8;
// end loop

I have more!

A Step wave?

// loop
number = Math.floor(Math.sin(phase)) - Math.floor(Math.cos(phase));
// end loop

Shark fin?

// loop
number = Math.cos(phase) - Math.floor(Math.sin(phase));
// end loop

Saw tooth?

// loop
number = phase - Math.floor(phase);
// end loop

a line that cycles from going up and up, but resetting at 0 again

Saw Sine?

// loop
number = Math.sin(phase) - Math.floor(Math.sin(phase));
// end loop

To listen to these examples, check out this demo!

That’s all I have in the Flashamaphone project.  We’ll see soon in part 5 of this series that there is much more to producing different types of sound than just the math to generate the cycle.  First though, jump on over to part 4, and we’ll talk music theory!

TOP