REVIEW: MP3 Comparison Tests

[This article was published on TapeOp’s website. Whatever it took to publicly humiliate the competition in print, Larry Crane didn’t have, but he still had the nuts to say it somewhere.]

All Rights Reserved © 2006 Thomas W. Day

A while back, I read an editorial in Mix Magazine that claimed to “prove once and for all” that MP3 audio compression was low-fi.  The author was a Mac user, so maybe he can’t be faulted for the many flaws in his test procedure, but be that as it may be, I read between his lines and found a few statements that I couldn’t believe.  His procedure was incredibly flawed and rigged to make MP3 files test poorly, even if the file format and compression were perfect.  His built-in bias was so strong that it was obvious how the test would turn out, even before the second sentence.  This author claimed to hear a “robust difference signal,” proving that the MP3 format was defective and high distortion.  I listen to MP3s, mostly in my car and I was suspicious of that “robust difference signal.” 

First, probably because of the limitations of his computer equipment and/or software, the author took an audio sample, inverted the phase of that sample, played the two (in-phase and out-of-phase together), and found that the resulting signal was the expected nothing.  The result was a near-perfect cancellation, proving that the two samples were equal, with practically non-existent distortion.  Here is where the example died a non-scientific death.  The author converted the phase-inverted signal to an MP3 format and, because his software could only play AIFF wave files, converted the MP3 back to AIFF and ran the above test again. 

Back in my audio manufacturing employment history, I built an ABX test rig for my employer and did a ton of ABX testing on anyone I could con into submitting to the ABX protocol.  Mostly, we learned what the audiologist discovered at the 1985 AES Convention in LA; that most people involved in professional audio are “functionally deaf” or, at least, “hearing impaired.”  Instead of spending our time trying to figure out what subtle differences in equipment were audible, we discovered that drastic defects in the signal path went undetected by most of our listeners.  I went so far as to install defective active components (ICs producing as much as 10% THD) into the signal path of otherwise identical pieces of equipment and found that an alarming number of audio professionals were unable to hear the difference in a reasonably good listening environment.  On the other hand, one of my own employees was able to hear signal differences that my test equipment (which was Audio Precision’s finest of the time) barely identified as measurable (except for substantial very-low-frequency phase differences that I’m still hard-pressed to believe explained the listening test results).   With that history behind me, I decided to replicate the magazine test myself, using a Windows-based software (Adobe Audition) which doesn’t have the limitation of only being able to play one file format simultaneously.  This eliminates the multiple conversion errors from the author’s test and makes the test more of an apple-to-apple test.

I picked three MP3 formats, 128Kbps, 192Kbps, and 320Kbps, constant bitrate, with CRC checksums and a pair of 44kHz, 16-bit CD recordings (“Afternoon” from Pat Metheny Group, Speaking of Now, and L’Adoration de la Terre from Telarc’s Cleveland Orchestra recording of the Stravinsky The Rite of Spring) for the test.  I, first, copied a section of the music and made an inverse-phase mono copy of that section.  In Audition’s Multitrack View, I inserted the two sections into a pair of tracks and compared the resulting signal; or lack of signal.  The two WAV copies exactly cancelled, both according to my ears and Audition’s metering system.  Figure 1 (from the Afternoon recording) displays a section of the original signal from the recording that I will use to display the results of the signal-canceling comparison tests.

Figure 1: The original signal, with peaks approaching 0dB.

I tried both Windows’ Media Player v9.0 and dBpowerAMP Music Converter™ for the creation of my MP3 files (all made from the reversed-phase WAV file with an already established signal accuracy).  I believe I saw a very slight reduction in distortion using the 320Kbps Music Converter™ version of the Media Player output, so I created the rest of my MP3 samples using that program.  I inserted the 3 MP3 inverse-phase signals into a 5 channel Multitrack file and listened to the resulting output, comparing each MP3 to the original in-phase WAV file. 

The first problem you will discover in performing this test is that the MP3 converters both added a short leader (approximately 50mS) to the files.  Ignoring this anomaly produces the “robust” distortion difference signal that started this investigation for me.  Time-aligning my files took some time, but produced a much more believable result from the comparisons. 

 

Figure 2: 12 8Kbps distortion result waveform

The 128Kbps compression produced a tinny output, with a little low end and a sound quality that was obviously distortion components. The resulting distortion component waveform is pictured in Figure 2.  At this point in the original recording, the peaks are touching 0dB, so the peak distortion output was approximately 25dB below the peak recording signal level.

Figures 3 and 4 picture the results of repeating this test with the 192Kbps and 320Kbps MP3 samples.  The distortion components of the 320Kbps compression sample are 33-36dB below the original signal.  Those residual signal values roughly translate to 5% THD for 128Kbps, 3% THD, for 192Kbps, and less than 1% THD for 320Kbps MP3 samples.  I’ll agree that these are substantial distortion values, but “robust” is not how I’d describe the resultant signal and I question the ability of most professionals to clearly hear the difference signal in an ABX environment.  After all, I’ve simulated much worse distortion components that appeared to have been inaudible.

Figure 3: 192Kbps distortion result waveform

Regardless of my test results, I recommend that you try a similar test with material of your own choosing and in a controlled listening environment.  Personally, I discount the results of any listening test that doesn’t live up to the rigor of the ABX protocol.  You can believe any fantasy you like, though.  Part of what makes working in audio so entertaining is the delusions under which we labor and the resulting, sometimes silly, products produced to cater to those illusions. 

 

Figure 4: 320Kbps distortion result waveform

I do, however, intensely suspect the opinions of someone who claims that consumer cassettes were musical and that MP3 reproduction systems are deficient in comparison.  Compared to FM radio, a high resolution MP3 is practically pristine.  Pop recordings are often so distorted that the minimal harmonic addition low-fi 64Kbps compression introduces can do no more harm to what’s left of the musical content.  If we’re not going to complain about these traditional high distortion delivery systems, where is our credibility regarding new technology?

Most analog consoles don’t produce 40dB cancellation artifacts when one channel is beat against another in producing the side component in a Mid-Side microphone signal.  Analog recording systems are far from capable of producing this level of signal uniformity. 

Dis’ing our customers’ sonic standards, because they are listening to a technology that has an economic impact on our industry, is dishonest, unbelievable and ineffective.  They know the MP3 files they listen to are higher fidelity than past and current commercially delivered formats and once they suspect the industry is lying about quality, what else do we have to offer? 

Technorati Tags: audio recording,distortion,mp3,analysis

ARTICLE: Loud Noises

All Rights Reserved © 2004 Thomas W. Day

"I'm busy," was my first excuse.  "Too many things to do, too little time, no money, my teeth itch," were among the later excuses I used to avoid getting involved.  None of those excuses worked.  He needed help and I was the only guy he knew who he would trust with his first big gig as a sound company.  He kept at me.  My friend was the proud owner of a startup business and his situation was the usual startup position of low cash and large expenses, this wouldn't be a paying gig for me.  Being self-employed and in the middle of my busiest part of the season, I do my best to avoid non-paying gigs. 

I worked, for a long time, for a pro-sound equipment producer.  During that period, I had the opportunity to slither into a fair number of high-end gigs, where I got to do monitors and FOH for some of my old musical heroes. This backstage pass got me into those opportunities without the credentials or the paid dues to be in that position.  I was the FOH and monitor engineer for one of the best horn bands I've ever heard, short of JB and Co.  So, I've done what I wanted to do in live music and it's been fun, thanks for the memories.  But, no thanks, I don't need any new ones. 

Three decades ago, I moved from doing live music to recorded music in one short moment of clarity.  I was a recording engineer, live engineer, and musician, at the time, and had fallen between bands and in the midst of business building.  One evening, after a particularly wonderful day in the studio, a friend called to say he was putting together an OMB (original music band, as opposed to the very financially successful cover bands he'd done in the past) and wanted me and my tunes to be part of the group. 

Still flushed from the day's recording work, it only took me a moment to respond, "What you're asking is do I want to work twenty hours a week, along with my day gig, putting together an act so that I can drive five-hundred miles a day to lug four tons of equipment up two flights of stairs and set it up while you (the drummer) putz around pretending that you're tuning your snare and the rest of the band primps their big hair, so that I can play my songs in front of a crowd of drunks who continuously bawl 'Freebird' for three hours until it's mercifully over and I get to tear it all down, pack it up, and drive to the next hellhole where I can do it all again?  Is that about it?"

"Something like that," he mumbled. 

"I think I'll pass," I answered.

We exchanged myths about where our latest project would be taking us and he went on to find another victim.  His victim was mutual friend who went on to riches and fame as a one-hit wonder bandmember and, as far as I know, the two of them are still touring fairs and clubs and are happy to be doing it.  I managed to nearly go broke with my recording studio/sound company and ended up where I am today through a completely random route. 

We ran into each other again, about fifteen years later, when one of his bands was recording two of my tunes.  As an afterthought, his management company decided they should get my permission to include the songs in an album.  Since they'd already recorded the songs and had pressed and distributed a few thousand media copies of the record, they thought it would be a formality.  Ah, the music industry; the soul of integrity.

Shift forward again, thirty years later, an old friend is trying to drag me back into the world of back-breaking labor and loud noises, if just for one night, and I'm not particularly tempted.  In the past few years, I've done an occasional live engineering gig and I've usually been disappointed with the experience.  Being a type-A, semi-perfectionist, anally retentive mental case, the sloppiness of live music in mediocre venues doesn't appeal to me.  Acoustically, the experience is disappointing.  Musically, the lack of control and perfection irritates me.  The big plus for most folks who love doing live music, the audience reaction, is a fat zero for me.  I do music for myself, outside opinions hold about as much water for me as a rusted sieve.  When the crowd cheers, howls, claps, or farts, I can't hear what I'm doing at the board and it irritates me to lose control for even a few seconds.  Every once in a while, all of that gets cancelled by the quality of the act and the complexity of the job, but not often enough to make me want to do it more than once a every year or ten. 

Have I made it clear that I didn't want to fly anywhere to be a part of putting together, testing, and running a sound system?  In the end, my friend found the cash to fly me to the show and the gig landed on a weekend where my schedule had a hole.  I was out of excuses and in the air in a post-9/11 world where flying is less fun than it has ever been.

He met me at the airport and, during the drive to his facility, we talked about the work we'd be doing the following day.  We ended up doing a little of that work before giving up for the night and he dropped me off at the hotel   Five hours later, we're on the way to his shop to load gear.  Sometime about now, he learned that the promoter had copped-out and opted to save a few bucks by not hiring the union guys to let us into the auditorium after noon.  We were going to have less than three hours to unload, setup, integrate the rented components, troubleshoot, and sound check a new and fairly large professional sound system. 

At least, we thought, the contract stated that we wouldn't be "allowed" to take part in the unloading of the truck (union rules), so we'd have some time to start wiring the system up as it was delivered to the stage.  On my way up the loading ramp, however, the union supervisor said, "If you want all this stuff on stage before the soundcheck, you better give us a hand with unloading."  So much for "extra time" we might have carved out of the idiotically tight schedule.

Two hours later (and twenty zillion minor hassles with unloading, available stage power, and equipment configuration), I'm firing up the mains and setting up the crossover and EQs.  One major piece of luck occurred at this point: one of my friend's potential future customers was on site for this stage and left before the headliner's crew got their hands on the gear.  Our initial sound check produced tons of volume (enough that I stuffed my ears with -25dB earplugs) and a tight, clean musical output.  The customer commented on the quality of the sound and said we had more than enough power for his facility and he left. 

An hour later, the headliner's FOH guy had "tweeked" the crossover until the subs dominated the system's output and bass definition was lost in the blubbering wind of 80Hz-and-below rubble.  (Ask me why I don't think live sound guys have ears and I'll take you to a concert.  Proof enough?  Ask me again and I'll take you to an audiologist, where a quick hearing test will eliminate all doubt.)  Finishing off the system's low frequency response, the FOH guy EQ’d out everything from 150Hz to 400Hz, with a few dips in the random lower midrange and his test music sounded like a 1960’s airport public address system.  He was a happy camper, as best I could tell. 

The band's monitor engineer added an equal demonstration of expertise by waving his SM58 inches from the monitors and asking for various frequency bands to be cut until he could no longer produce squeal.  Then the monitor volumes were increased until he got more squeals and had rolled off those tones.  When he was done, the monitor graphics had all cut everything from 300Hz up.  He might have asked for a shelving bass boost and got the same effect. 

Various other problems appeared as we went through the sound check and some of them were pretty serious.  We wrestled with bugs and glitches right up until the lights went down and the crowd came through the doors.  To protect the innocent, guilty, and random bystanders, I'll refrain from anything more specific than these descriptions of our tribulations, but everything (barely) short of burning down the house happened between the start of the sound check and the headliner's opening notes.  We did lose about an hour of troubleshooting time due to a false fire alarm that chased us from the building.  I used most of my experience as an engineer and all of my abilities as a politician from the moment we hit the loading ramp until we closed the truck door and drove away from the venue. 

I ran FOH for the opening acts and "accidentally" hit the main's EQ bypass switch for those acts.  In fact, I forgot to switch the EQ back on for the FOH engineer and he happily played with the knobs and buttons for the first half of his act, with the EQ out of the signal path.  Other than the huge sub crossover boost, the system sounded pretty decent flat.  As the volume rose through the show and hearing fatigue took its course, I continued to trade up my hearing protection until I was stuffing 34dB industrial foam ear protectors around my 25dB "musician's" protection.  My ears still rang the next morning and I'm way too old for that shit. 

While doing my bit as a customer service agent during the main act, I watched the FOH guy fool with a delay line.  While he had a short, medium, and long tap assigned to the output, the short delay seemed to occupy most of his attention.  He rolled it about 5mS above and below 45mS, varying it with the song being played and some inner cue, but never seemed to notice that the delay had no signal on its input indicator.  That particular aux group had its group volume control half way up and none of the individual channels were assigned to the group.  The same story applied to most of the other effects he "used," too. Lots of adjusting, no input signal applied.  Or, I suppose, the input signal was so minor that my old insensitive ears and the units' VUs failed to pick it up.

At the other end of my attention was the question my friend asked me every time he and I ran past each other during the show, "how does it sound?"  My first response was "loud."  Too damn loud, was my actual opinion.  I tried to listen critically and I had to admit that most of the percussion was audible in the mix, the vocal was reasonably clear (considering the microphone being used), and everything but the bass and kick were present and identifiable in the mix.  That's about as good as amplified live sound gets, in my experience.  So, that's what I reported at the end of the show. 

He was happy.  The band was really happy.  The venue was impressed.  The union guys helped us reload the gear and we got the hell out of there before anyone became undiplomatic and violence erupted.  I've since heard great reports from my friend about the reviews of the show and the response from his potential customers.  Everyone's happy and I survived. 

The next day, I flew home.  Now I'm back in the studio and, if it's another couple of years before I see, do, or think about a live show, I've had my Jones satisfied and it won't cross my mind that I might be missing something by living in the virtual world. 

You live guys have a hard road to hoe and I respect the work you do under the conditions you do it.  When I was building pro sound gear, I thought we were doing the world of music lovers a disservice by making it possible for their hearing to be so abused.  When I get my hands dirty in the actual production of a live show, it makes me feel even more guilty.  My generation is going deaf faster than any before it and the generations following us are even more damaged.  There is nothing good to be said about volume over quality and you won't catch me even trying to defend the majority of live music situations.  This experience reminded me of why I rarely go out to see an act, even one I'm recording, and why I do any damn thing I can think of to avoid sitting at either the FOH or the monitor desk.  All of my heroes have ears and I'd like to hang on to mine as long as I can. 

PERFORMANCE REVIEW: Jeff Beck at the State Theater in Minneapolis (3/9/2001)

Many of us own guitars, last night I saw someone play a guitar. In fact, I believe on Friday night I sat among one of the larger gatherings of Minnesotans who own guitars. I also suspect a fair number of guitars were bought and sold in Minnesota on Saturday morning.  Some of us probably felt fairly discouraged at how little we know about the limits of our instrument. Some were encouraged at the previously unrealized capabilities of our instrument. 

In the "historic Minneapolis State Theater," Jeff Beck, and band (Jennifer Batten - guitar/synth, Andy Gangadeen - drums, and Randy Hope-Taylor - bass), may have put on the best show I've ever seen. What Jeff Beck does with fingers and a Strat is physically impossible. He putzed around a little with a wah petal, otherwise, every sound you hear comes from his hands. His fingers, in fact, since he doesn't use a pick. Ten quick fingers creating more sound effects and tonal variations than George Lucas gets from a team of effects engineers and a space ship full of computers.

For 30 years, I've studied Jeff Beck interviews and attended his rare Midwestern concerts, trying to glean something that could apply to my own music. He's so far above my capabilities that I rarely come away with much more than inspiration.  Again, this year I am inspired but uninformed. 

One of the great disappointments in seeing him live in the past was that he's always said that he wouldn't mess with a slide on stage. This show, he gave us two nothing-but-slide-guitar songs and he liberally used the slide on a half dozen other tunes. Compared to any other human playing guitar with a bottleneck, Jeff Beck is unbelievable. During Hip-Notica, I swear, he played with such incredible precision, within millimeters of the bridge, that he bordered on performing the impossible.

A spacey Minneapolis kid intro'd the show, doing something a guitar style and technique somewhere between Michael Hedges and Stanley Jordon. I thought Jeff might have lost his marbles, putting someone that good in front of his show. The kid described the set he was going to do as something where he would just try to "love you up, because Jeff is going to kick your ass." And he followed that by playing some wonderful two-handed tapping stuff. Still, he was right. Jeff took an audience full of guitar players and had them beating their hands till they were raw and he had us yelling for more till a few of us sounded like refuges from a TB ward.

Two encores, a 2 1/2 show that never let up. Fun lighting and visuals, lots of car stuff and 50's movies. It sort of reminded me of the kind of show that ZZ Top always tried to do but didn't have the chops to pull off.

A lot the stuff on Jeff's new records sounds sequenced, it's so complicated and polyrhythmic. It's not. The fucker is unbelievable. He is a ten fingered rock and roll orchestra. I wish he were going to be where ever you live, I'd rag on you to see him till he was back in France playing with hot rods.

Somewhere I read that Andy Gangadeen played drums for the Spice Girls. If so, he's redeemed himself.

ARTICLE: Who Would That Inconvenience?

[Clearly, in 2004 I was in a prolific self-publishing mode. I didn’t even try to find a home for some of these articles, as strongly as I felt then or now about their relevance.]

You can't read an entertainment or computer magazine without running into an article or ad about how much "damage" is being done to the nation's creativity by "intellectual rights theft."  The game manufacturers are, apparently, going broke because so many people are "stealing" game software.  The music industry execs are all living in cardboard boxes in New York and LA's back alleys because nobody is buying music, we're all stealing it through MP3s online.  And those poor, poor software designers are working nights at SuperAmerica to support their creativity because everyone is bootlegging software.  Billy Gates has sold his mansions and lives in a Seattle commune with 35 other desperate, but dedicated-to-their-art, Microsoft employees.  Congress and the international community needs to "do something" to protect these poor, starving, and productive folks before these vital industries are starved into non-existence.

Or not. 

I think not.  For instance, the music industry thinks nothing of "giving away" it's product on the radio.  Radio stations "give" us more of the music industry's product than most of us want to hear.  Sure, they're selling ads and using the music as a loss-leader to attract listeners, but they have worked overtime to own the broadcast business so that they have easy access to free distribution of their god-awful products.  How that is different than the distribution of these same products through websites is too indistinct for my sensitivities.  I'll admit that I'm not particularly sensitive to the needs of overstuffed executives and their desperate desire to own everything from natural resources to original thought.  But I do believe that if you give away your product in one venue, you've given up the right to be able to protect it in another.  The Supreme Court thought the same way when they prevented record companies from limiting the use of cassette recorders, 30 years ago. 

But my real reason for writing this article is to address the pitiful condition of software companies.  In our industry, bootlegged software is as common as guitar picks.  At the tech school where I sometimes teach, I hear about the use of "illegal" versions of every program from ProTools to Sonar to Logic to $50 plug-ins and $10 sampled loop CDs.  Everything you might ever want to own is available, for free, in a bootleg version.  Acquaintances who aren't even involved in the music business have copies of programs that I couldn't justify buying if I didn't need to eat regularly. 

Software manufacturers estimate that they've "lost" somewhere between two hundred million to a billion-zillion dollars due to software bootlegging.  According to their estimates, everyone on the planet would have purchased their products if they hadn't had access to illegal versions.  Some of us would, surely, have bought those products several times if legal channels were the only way we could obtain software.  Software companies have moved from vaporware to vapor markets.  Their hallucinations of wealth and power have infected the magazines with whom they advertise, too.

I beg to disagree.  In fact, I'd bet that the opposite of their argument is true.  I'd argue that if bootleg "test" software wasn't available, most of us wouldn't bother with computers at all.  The computer industry has made that a fact by its inability to build reliable, quality products.  The business tells us that bootleg software is unreliable and to obtain reliability we should pay for the software and pay again for the minimal "customer support" provided by the manufacturers.  Experiences tells us that the words "unreliable" and "software" are inseparable.  It doesn't matter if you pay for software or if you steal it, it's going to be unstable, undependable, buggy, and a constant irritant.  And you're going to have to pay, over and over again, for future versions that "fix" a few problems and create a lot more. 

That's a great business, if you can find it.  Build a crappy, unreliable product and follow that up by charging your customers extra for "supporting" your mistakes.  That is the next step beyond planned obsolescence, assuming that your customers don't revolt.

Basing a home studio around bootleg software is a risky business, because putting any faith in software is an act of unfounded faith.  The products are, simply, buggy as a Tijuana hotel bed.  Software crashes, we reboot, software crashes again, this time rebooting for us, and we pay more money in the foolish hope that, this time, the code will be more reliable.  The more features we get, the less reliable the software becomes.  It's an endless, unproductive closed-loop.  Many users are honestly "test driving" software by using hacked versions.  On the rare occasion that a piece of software is actually useful, many users turn from test drivers to legitimate owners.  I see it happen with students all the time. 

Occasionally, a company screws up and releases a version of software that crashes less often than its competition.  When that happens, many users end their search for improvement and get back to doing productive work.  A few months ago, I read a business computer article about a large number of businesses who are still using Microsoft's Windows for Workgroups, version 3.11, and the associated version of Office, because it did the job that needed doing.  Those of us who migrated to Windows 95, 98, 2000, ME, and XP often think that the few functioning features we've gained haven't repaid the investment in retraining, debugging, and the reliability problems we've suffered in transition. 

What would happen if every computer user decided to quit chasing the holy grail of the newest, coolest version of software and found something that worked reasonable well and stuck with it?  There are a fair number of analog studios out there who have gone back (or stayed) to 20 year old technology and spend their time tweaking reliable and functional technology and making music.  Computer recordists could do the same thing.  Quality work was done on almost every version of almost every program.  We could have all quit looking for something new at any time in the last decade or two. 

Of course, that would break the software company income chain.  If every software company in the industry went broke and vanished tomorrow, who would that inconvenience?  If we all were forced to use the products we're using today for the next fifty years, would we stop recording music?  If today's software glitches and crashes became a constant, known, and never-ending irritant, would that make a lick of difference to people currently using that software?  You know it wouldn't.

If the state of the art never moved an inch from its current status, we'd still be doing what we're doing today.  We'd be recording music, burning CDs, cutting vinyl, posting MP3s to websites, and looking for the next cool music.  The profits or losses of software companies means nothing to the flow of music.  We have the tools we need and a hell of a lot of tools that nobody needs.  Software companies have their problems and we have ours and the two have nothing to do with each other.