Live Instrumentation in Flash Part 7 – Beyond Flash

At the heart of all our audio generation lies one simple Flash feature – the sample data event.

Can we find things outside of Flash that will let us do similar things?

The answer is….of course!  People have been writing desktop audio and music synthesizer software for years.  Flash and AIR offers some decent functionality on the desktop, but writing something in C++ would blow Flash away of course.

The reason is that Flash has only been in the generative audio game for a few years, while people writing C++ have been at this for decades.  As such, they have access to ready made filters, libraries, etc.

Take, for example, Node Beat – written with OpenFrameworks


Objective-C for doing iPhone development has the nice benefit of access to core libraries that Apple OSX Cocoa developers have had access to for years.  The basic AudioUnit in Cocoa or Objective-C gives you the low level access you need to make things like Mobile Synth:

Or even a simple tone generator:

Basically though, you’ll find that anything that gives you low-level audio access will give you what you need.  Speed is the key here, number crunching needs to be done in an instant.  Flash can keep up OK, and gives you the bare minimum of what you’d need to generate sound.

What about Javascript?  Well, sort of!  The Web Audio API has been defined and implemented on the developer version of Chrome and there’s also a Audio Data API draft in Firefox.  The basics are the same as Flash, but many more goodies are offered to help you along.


So, in the end we need speed and some basic API hooks.  On the web, Flash seems the most reliable, but it sounds like Chrome and Firefox are nipping at Flash’s heels with some very cool stuff coming up!

And in the mobile world, just like the desktop world, it seems this sort of base level audio access is standard as well.

Live Instrumentation in Flash Part 5 – Voice Synthesis

So far, we’ve discussed how to create tones, notes, and chords.  We’ve also gone into several different algorithms to change how a digital instrument sounds.  Changing how our digital instrument sounds is a lot more than simply changing the algorithm that defines your sound wave!  You may have noticed that given the several algorithms we covered in part 2 of this series, the sounds didn’t change all that much.

What can we do to create some more variety?

Attack, Decay, Sustain, Release

In electronic synthesizers, there are the concepts of “attack”, “decay”, “sustain”, and “release” (commonly abbreviated ADSR).  The easiest way to explain this is to imagine yourself hitting a key on a piano.

The moment you hit the key, the string is hit by the hammer by the piano.  This initially creates a loud tone.  The initial tone is much louder than anything that happens moments after.  This initial loudness is called the “attack”.  As the initial loudness wears off, the sound draws down to the normal volume as you hold the piano key.  This drawdown is known as the “decay”, and for the rest of the duration that you’re holding the key is known as the sustain.

Once you release that piano key (on a real piano), there would be a small amount of sound coming out as the string is still vibrating.  This will eventually fade away – but this period is known as the “release” period.


Attack, Decay, Sustain, Release is a type of envelope.  At a basic level, an envelope is controls volume on a waveform as the cycles continue.  There are more types of envelopes, but the ADSR envelope is the most commonly known.

Some synthesizers break up the phases even more.  Consider the DAHDSR envelope.  This starts with a delay phase.  This is the phase before the attack – a phase where you don’t hear any audio, or much at all, in the initial lead up to the loudness that is the attack phase.

Then comes the “hold” phase.  This is meant to prolong the time between the attack and release.  It keeps the volume high!

And then the rest continues with the decay, sustain, and release.

Different types of synthesizers take different envelope approaches.  Some even let you create your own – however it seems that the simple ADSR is the most common.

Shortening the Sustain

One thing I’ve found is that a sustain that keeps going and going and going while you press a key, sounds kind of unnatural.  It might be appropriate for something like a pipe organ, but if you’re trying to replicate the percussiveness of a piano or a bass guitar pluck, you don’t want something that goes on forever.

Keeping it short and sweet is the perfect key to making something sound natural!  I’m still experimenting with envelopes to try to accomplish this effect without just cutting off the waveform.  Overall though, I’ve found that limiting a sustain to a few hundred milliseconds is the perfect solution to making a more natural sounding instrument – especially if you want something percussive like a bass pluck or piano key.

Harmonic Overtones and other Imperfections

Imperfections in the note quality are a good way to make a computer generated note sound more natural.  Perhaps the tone could be off a couple steps in the frequency, however there are other ways!

One such way is to give a note a harmonic overtone.  Remember back in part 4 when we went into note relations?  We talked about octaves, where an octave higher will produce the same key/tone, but is at a higher pitch.  An octave higher or lower is in perfect harmony with the root note.

With harmonic overtones, we layer several octaves on top of the note.  This gives a more natural sound.  Many instruments in the real world will have the first 2 overtones be in perfect harmony.  In this regard, starting at middle A of 440hz, we’d have overtones of 880hz, 1320hz, and 1760hz.  All these notes playing at the same time to produce something more natural!

Another factor in real world instruments is that these overtones (especially the third and fourth) may not be exactly the same key in an octave, not in absolute harmony.  These higher overtones might be a little sharp or a little flat.  These nuances contribute to the instruments unique sound.  In fact, if you’ve ever seen the flare on the end of the trumpet, it’s not to get more volume, but to correct the harmonic overtones to be closer to absolute harmony.

We can absolutely replicate this in our digital instruments!  We can mix harmonic overtones together when playing a single note, and even make variations to avoid absolute harmony and give our instruments some character.

Filters, Modulations, and more

We’ve only scratched the surface of creating new and exciting voices for our instruments.  We can pass our tone through a filter, do frequency modulation, and more.  I’m still exploring, so this is best left for another time!

Live Instrumentation in Flash Part 4 – A Little Music Theory

In the last post, I threw a number at you: 440.  This was the frequency I put in my code to make the resulting audio sample listenable.

Well, it was no accident.  440 cycles is actually a middle A.

A Brief History

It wasn’t always 440hz (hertz actually denotes cycles per second), but not for lack of trying!  Remember that we haven’t gone digital until the past 30 years or so, and industrial manufacturing is only 70-80 years old.  People have been making instruments and playing music on them for thousands of years.  Beethoven was playing the piano in the late 18th century.

When you hear Beethoven’s music recreated, you’re not hearing it exactly as he played it.  Yes, all the music is transcribed and is probably fairly true to the original.  However, tuning a piano in the 18th century wasn’t an exact science.  Beethoven’s middle A might not have exactly been 440hz.

While Beethoven was a German composer, Chopin was Polish.  Being in different countries, so far apart, Chopin probably had a different tuning than Beethoven.  If you were in the German audience listening to Beethoven, you might be accustomed to a middle A tuned around 338hz.  Whereas, if you traveled to Poland, and heard the 445hz middle A, you might think something was a bit off.  You weren’t used to the different tuning.

Recorded music and touring artists changed all this.  As records were being sold and artists toured, someone in Japan could listen to a record produced in Spain.  As folks started getting together from all over the world, a sort of standard tuning was made out of all of this.  When digital music and exact measurements of frequencies came along, we could finally settle on a specific standard that everyone could agree on.


The Math of Notes and Octaves

So far, we’ve only talked about middle A, or “A4″.  Since A4 is basically the center of it all, we can go from there.  First off, we can talk octaves.  Take our middle A, for example.  If you go down an octave, you land on another A….A3.  Basically, its the same tone but a lower frequency.

How, do we figure out the frequency?  Well this time its easy – no fancy trigonometry!  Just half the 440.  So A3 is 220.  A2 is 110.  A5, on the other hand, is double 440 (880hz).

Now, what’s between A4 and A5?  What’s between an octave?  Well, there are 12 notes between the two.  In music terminology going up 1 note is called a “half step”.  Going up 2 notes is called a “whole step”.

Starting at A4 (or any A):

  • A
  • A sharp/B flat
  • B
  • C
  • C sharp/D flat
  • D
  • D sharp/E flat
  • E
  • F
  • F sharp/G flat
  • G
  • G sharp/A flat

If we assume that we’re starting at A4, and going up 1/12 of an octave for each half step, there’s actually a mathematical formula to calculate the frequency!

That formula is: frequency = 440 * (Math.pow(2, indx/12));

You should ALSO note that the relation between half-steps.  If note X is a half step below note Y, note X is said to be Y “flat”.  Note Y is said to be X sharp.  This applies to any key (not just the black keys on the piano are sharps and flats, though it makes total sense to name them that way given their positions).

So in the Flashamaphone framework, I’ve gone ahead and started at middle A.  I’ve put each of the twelve notes into an array.  Whichever index we’re trying to access in represented by “indx” in the above formula. We subtract 12 from index for each octave below 4, and add twelve for each octave above 4.
What we have here, is a simple way to calculate frequency for a given note!  Pretty cool!

How Western music is different from others

Jjust like how Beethoven’s piano was tuned differently from Chopin’s, different cultures have different connotations for chords.  The most basic example is major vs minor.  In our western culture, we associate major keys with a happy sound, but minor keys are sad.  In other cultures, this doesn’t necessarily hold true.  In fact, Indian music has no concept of major vs minor….and so, no similar connotations.

What we’ll cover next with chord structures will be accurate for any culture.  However, what each chord construct invokes for a person in western civilization is completely different for other cultures.    We’ve already discussed that major vs minor doesn’t mean anything important for Indian music – but that doesn’t mean the construct doesn’t exist!  It’s all just different ways of describing the ways notes can make up a chord.

Chords Structures

Starting from the root, you can get all the notes in a “key signature” the same way from any root note.  A key signature groups notes together – and typically if you play only the notes in the key, your music won’t sound off-putting.

A major key is always comprised of the first, third, fifth, seventh, ninth, and eleventh half tones up from your root.

A minor key is used in western culture to make something sound sad or similar.  Likewise, its always comprised of the first, third, fourth, sixth, eighth, ninth, and eleventh half tones up from the root.

I take care of this in Flashamaphone by making the following arrays for you:

// major key
notesInKey.push( notesToIndex[0] );
notesInKey.push( notesToIndex[2] );
notesInKey.push( notesToIndex[4] );
notesInKey.push( notesToIndex[5] );
notesInKey.push( notesToIndex[7] );
notesInKey.push( notesToIndex[9] );
notesInKey.push( notesToIndex[11] )
// minor key
notesInKey.push( notesToIndex[0] );
notesInKey.push( notesToIndex[2] );
notesInKey.push( notesToIndex[3] );
notesInKey.push( notesToIndex[5] );
notesInKey.push( notesToIndex[7] );
notesInKey.push( notesToIndex[8] );
notesInKey.push( notesToIndex[10] );

The most common chord structure is a triad – this means that there are 3 notes in a chord.  The first note is a chord is the “root”.  A “C” chord will always have the “C” note as the root, in other words.

A typical triad would consist of the 1st, 3rd, and 5th notes in the key signature. There’s lots more chord types, which you can use Flashamaphone to generate for you, but that’s the basics!

Live Instrumentation in Flash Part 3 – Generating a Tone

Basic Sine Wave Generation

OK!  Lets step into the wayback machine and go back to part 1 of this series.  We were talking about building audio samples with a byte array

 var bytes:ByteArray = new ByteArray();
for (var c:int = 0; c < 8192; c++) {

I never DID tell you guys what goes in number, did I?

Well, that’s where we can start making something listenable (and not the white noise we made before).

Lets make some changes:

for(var c:int = 0; c < durationinseconds*44100; c++) {
var number:Number = Math.sin(c * 2*Math.PI/44100 * 440);

OK, so what’s going on here? We’ve introduced, first of all, “durationinseconds”. We multiply by 44100. Since there are 44,100 samples per second, all we need to do is multiply by the number of seconds we want to produce, and generate that many samples.

Next up is to generate a sine wave. Why a sine wave? Well, think about your high school algebra class. There are all sorts of mathematic equations, and you can graph any of them. Most basic is a simple line.

a line - it starts infinitely low, and ends infinitely high, never repeating

The problem with a line, is that it keeps going up and up and up and up. That’s really not what we want here. What we want is something that goes high, comes back down, goes up again, and can do that forever.

We COULD utilize a little programming, and make a line, but at a set high point, reset and make it start at the low point again.

a line that cycles from going up and up, but resetting at 0 again

This is actually a good way to produce a sound, but lets talk sine waves for now.

The sine function to produce wave cycles is fantastic. The line can go up and down all day as you continually increment, but not only that, it’s curvy! So it goes up and down in a very smooth way.

Goes up and down repeatedly (forever) and is curvy

The thing about sine waves is that since they repeat themselves, they can ALSO be measured like a circle – in degrees. If you go 360 degrees, you’re right back where you started from. The bummer about trigonometry though, is that everything is measured in radians.

So 360 actually is equal to 2 Pi. 180 is Pi, and 90 is Pi/2. So above, we took 2 Pi and divided by 44100. We’re basically making a baseline here and saying that one full revolution through a sine wave is equivalent to one second. We multiply this by our iterator (c), to make the individual samples for each data point.

This would actually produce a SUPER low tone (I doubt you’d be able to hear it). I’m going to go ahead and ALSO multiply by 440 to produce something listenable.

Frequency and Amplitude

So let’s talk about what we did when we multiplied by 440. The effect we had was to take that super low tone (the one that takes a whole second to go a complete cycle) and make it go 440 cycles in one second.

How often something cycles is known as frequency. And that’s the thing about frequency in sound. If the frequency is too low, that is it doesn’t cycle fast enough, our brain doesn’t put together that its a repeating pattern, and it can’t latch on to the fact that its an audible tone. If too high, our ears can’t actually discern the signals either.

440 cycles is pretty much smack in the middle of what we are comfortable hearing. There’s a LITTLE more to this, and I’ll tell you more in part 4.

Anyway, all those cycles going so fast come together sounding like one tone. The more cycles per second, the higher the tone.

When you’ve heard about frequency, you’ve also heard about amplitude. In terms of audio, amplitude is just volume. It’s pretty easy. If you want a volume that’s 1/4 as loud, just multiply that “number” variable by .25 (OK I’m lying, volume is actually logarithmic, but I don’t feel like getting into that whole thing right now).

Here’s a nice little demo to allow you to play around with frequency and amplitude on a sin wave.

Different Types of Cycles

Like we discussed, sine waves are curvy… When we generate the tone, its sounds nice (if not a little whiny). There are other ways to go. I eluded to this before (with the line that we keep resetting in a cycle).
You can make things sound edgy too! Like you could make a square wave. I’m going to start giving examples now, and copy over some of what I have in the Flashamaphone project.

First of all though, lets simplify things and pop a bunch of math into a phase variable, like so:

var phase = c * 2*Math.PI/44100 * 440;

There! Now we don’t have to write all that stuff out each time. So let’s revisit how to do a sine wave:

// loop
number = Math.sin(phase);
// end loop
Goes up and down repeatedly (forever) and is curvy

Next, lets try a square wave

// loop
number = Math.floor(Math.sin(phase));
// end loop

It sounds very 8-bit and harsh.

At this point, I can start making up my own terminology – and I came up with a stepped wave. It’s half-way between harsh and smooth, between a sine wave and a square wave.

// loop
number = Math.floor(Math.sin(phase)*4)/8 - Math.floor(Math.cos(phase)*4)/8;
// end loop

I have more!

A Step wave?

// loop
number = Math.floor(Math.sin(phase)) - Math.floor(Math.cos(phase));
// end loop

Shark fin?

// loop
number = Math.cos(phase) - Math.floor(Math.sin(phase));
// end loop

Saw tooth?

// loop
number = phase - Math.floor(phase);
// end loop
a line that cycles from going up and up, but resetting at 0 again

Saw Sine?

// loop
number = Math.sin(phase) - Math.floor(Math.sin(phase));
// end loop

To listen to these examples, check out this demo!

That’s all I have in the Flashamaphone project.  We’ll see soon in part 5 of this series that there is much more to producing different types of sound than just the math to generate the cycle.  First though, jump on over to part 4, and we’ll talk music theory!