Links

Notes by ff123

Formal (and semi-formal) listening tests

• MPEG Site, "Results of AAC Subjective Tests," February 1998. Comparison of AAC Main 128, AAC LC 128, AAC SSR 128, AAC Main 96, AAC LC 96, MP2 192, and MP3 128. I don't know what version of FhG mp3 codec was used. AAC Main 128 was the clear winner. AAC LC 128 and AAC SSR 128 were at the next tier, with the rest lumped together at the bottom.
• "Download Showdown", Stereo Review's Sound & Vision Online. September 1999. Double-blind listening tests of MP3ENC 3.1, Lucent's EPAC 1.2, and Microsoft's WMA V2 at 128kbs. MP3 won by a slight margin, with EPAC mostly coming in second, and WMA trailing.
• "Download Showdown II", Stereo Review's Sound & Vision Online. About a year later (Summer or Fall 2000). Double-blind comparison of three codecs at 128kbs: FhG MP3ENC 3.1.1, FhG AAC Encoder 1.2, and Microsoft WMA beta version (at press time, scheduled to be released at the same time as the official release of Media Player 7). Results: AAC the winner, WMA in second and much improved from last year, and MP3 bringing up the rear.
• "Facing the Codec Challenge", Sound & Vision's 3rd round of comparisons (Summer, 2002). Double-blind comparisons of FhG's slow mp3 codec in Cool Edit 2000 at 128 kbit/s, WMA8 at 128 kbit/s and 64 kbit/s, and Real at 132 kbit/s and 64 kbit/s. The mp3 codec they chose sounds bad (see Bug in Older FhG "Alternate" Codec). Overall, WMA8 at 128 kbit/s was judged best, while the 64 kbit/s codecs did not achieve "CD quality" as claimed by their makers.
• EBU listening tests on Internet audio codecs (PDF file), Seven audio codecs tested at low bitrates (16 to 64kbit/s): Windows Media 4, MPEG-2 AAC, Opticom mp3, Q-Design Music Codec 2, Real Networks 5.0, RealNetworks G2, and Yamaha SoundVQ. At 64kbit/s, AAC was the clear winner, and was rated "excellent" for all samples. WMA4 ranked next, followed by Opticom mp3, Real G2, Twin VQ and Qdesign at the bottom.
• "Cross Interrogation", c't Magazine's mp3 Listening Tests, 6/2000, translated from German using freetranslation.com. Fourteen listeners, including twelve interested readers, compared music encoded with MusicMatch 4.4 (FhG codec) at 128kbs and 256kbs against the original CD. Listeners, on average, could not distinguish 256kbs from CD. (Original article in German)
• Summary table of c't Magazine test, 23/2001: mp+ easily wins over various codecs tested. Note: Fatboy Slim's "Kalifornia" is the source of the infamous "fatboy.wav" sample. One of the mp3 encoders used was MusicMatch 5.10.0149 (I'm almost certain the quality setting was "normal," making it the good FhG FastEnc codec). For more information on this article, see Citay's post at the r3mix.net forum.
• NSTL Test Report for RealNetworks: RealAudio 8 vs. WMA 7 at 20 kbs and 32 kbs (Nov. 2000). Click on PDF file link from this page. 100 people were randomly recruited from the Philadelphia, PA area. Test system was a PC with a Sound Blaster! Live audio card and a Cambridge Soundworks Microworks speaker system. On four clips of music, listeners selected RealAudio as sounding more like the original music about 70% of the time.
• NSTL Test Report for Microsoft: WMT 4.0, RealSystem G2, and MP3 (April 1999). Click on PDF file link from this page. 77 men and women were recruited from the local community. Test system was a PC with a Sound Blaster! Live audio card and a Cambridge Soundworks Microworks speaker system. WMT 4.0 was compared with RealSystem G2 at 20 kbs and 32 kbs. WMT 4.0 sounds more like the original music 81% of the time. Also, WMT at 64 kbs was compared with MP3 at 128 kbs (MP3 codec in MusicMatch Jukebox not specified, but is most likely from Xing at this date). Listeners chose WMT at 64 kbs over MP3 at 128 kbs 51% of the time. MP3 was chosen 29% of the time, and no difference was heard 19% of the time.
• AAC: Life Beyond MP3 (PDF file 99kB). Reprinted from the October, 2000 issue of Hi-Fi News.

Informal listening tests

• Speek's Listening tests at 128 kbs. Listening tests of various codecs including mp3, Ogg Vorbis, Psytel AAC, and MPEGplus.
• SoundExpert. Blind tests conducted over the Internet.
• Comparison by regeya - MP3 and Ogg Vorbis: A User's Perspective: on the Kuro5hin message board.
• Will Ryu -- ArS Technica: Do MP3 Encoders Sound Different? (3/2000): Audioactive, Xing 2.1, Blade 0.91, Lame 3.61 compared. In order of preference at high bitrates (192 kbs and 256 kbs): Fraunhofer (Audioactive), Lame, Blade, and Xing. At 128 kbs, Fraunhofer was least objectionable.\
• Ogg vs. Lame. Listening tests of Lame 3.88 using ABR and CBR vs. Ogg Vorbis beta 4. Various bitrates tested.

• Gian-Carlo Pascutto's ("Garf") reviews of listening tests

Other interesting sites

• About Subjective Testing Interesting Powerpoint presentation on using the Rasch model to separate out listener ability from program material
• Audio, Science in the Service of Art White paper by noted loudspeaker researcher Floyd E. Toole. Comments about subjectivity in listening tests, the effect of broadband hearing loss, and the importance of listener training, among other topics.
• A New Laboratory for Evaluating Multichannel Audio Components and Systems. Sean Olive paper on Revel's website, describing the tool used to train listeners to hear defects in loudspeakers.
• EBU Technical Documents
  • Sound Quality Assessment Material Recordings for Subjective Tests (Users' handbook for the EBU - SQAM Compact Disc
  • Listening conditions for the assessment of sound programme material: monophonic and two-channel stereophonic
  • Assessment methods for the subjective evaluation of the quality of sound programme material -- Music
• EBU Technical Review
  • Subjective assessment of audio quality -- the means and methods in the EBU
• MP3 and AAC explained, on the MPEG Tutorial Documents page. Not-too-technical overview.
• ITU-R BS.1116-1, "Methods for the Subjective Assessment of Small Impairments in Audio Systems Including Multichannel Sound Systems." This document can be ordered (takes just a few minutes) from the ITU website for CHF 22 (French Francs), which is about $14 US dollars (11/23/00).
• How MP3 Works: Inside The Codec sample chapter from Scot Hacker's book, MP3: The Definitive Guide. Good non-technical description of how mp3 works.
• PC ABX Web site, Arny Krueger's ABX program provides a way for individuals to perform the equivalent of a double-blind comparison of original and encoded files.
• Frank Baumgarte's home page, abstracts of papers about developing physiological models of auditory perception, rather than psychoacoustical models (as are typically used in audio codecs).
• Earguy's Virtual Listener page
• Project EAQUAL (Evaluation of Audio QUALity): an objective measurement tool for audio quality, designed especially for high quality audio codecs. It is based on the recommendation ITU-R BS.1387, which is an algorithmic description for quality evaluation. Project has been removed from sourceforge.
  • Windows 32 binary available at Speek's site
  • Some encoder tests with EAQUAL, on Speek's Site
• Objective Perceptual Measurement with a variety of links to perceptual measurement such as PSQM, NMR-DART, and PAQM
• PCAVTech Sound Card Technical Benchmarks, Arny Krueger's reports on various Sound Cards.
• Codec Capers, September 1, 2000. Digital Audio codec project (free registration required)
  • Continuing Codec Capers, December 1, 2000
  • Even More Codec Capers, December 1, 2000
• mp3 decoder tests, David JM Robinson's excellent comparison of popular mp3 decoders.
• MAD MPEG decoder, 24-bit MPEG decoder.
• Is 24 Bit Recording Really Better? Listening test
• Total Recorder, an application which allows one to record digital bitstreams from audio applications.
• Bob Katz's Digital Domain. Excellent essays from a master mastering engineer
  • Bob Katz on the audibility of extended bandwidth digital signals, shows the importance of good filtering.
  • The Secrets of Dither, Bob Katz's essay on digital processing.
  • Integrated Approach to Metering, Monitoring, and Leveling Practices, by Bob Katz. A proposal for uniform leveling standards.
• David Robinson's Proposed Standard for Replay Level
  • MP3Gain
• Lame Ain't an MP3 Encoder Home Page. Mirror site. Best open-source mp3 encoder.
• r3mix.net, Very popular site about mp3 ripping and encoding quality. No longer updated.
• EncSpot, an MP3 Analyzer
• MPEG Plus. Generally agreed to currently be the highest-quality lossy format starting at bitrates averaging 160 kbit/s or higher.
• Frank Klemm's MPEGplus Page. Also visit his Dither and Noise Shaping page and his Clipping page.
• Shibatch Audio Tools, Home of Naoki Shibata's accurate Winamp equalizer plugin, high quality Winamp decoder plugin (mpeg123), sample-rate converter, and mp3 encoder guesser.
• MP3Prozone
• Psytel Research. ISO AAC codec
• Audiocoding.com Home of FAAC, FAAD, and Waverate
• xiph.org: Ogg Vorbis, high-quality, full-featured, non-proprietary compressed audio format
• FLAC, Free Lossless Audio Codec: mutliplatform, open-source lossless compression
• RareWares, A collection of hard to find pieces of software
• Frequency Response of the Ear. Shows how to plot your ears' frequency response.
• HeadWize Article: Preventing Hearing Damage When Listening With Headphones
• Digital Recordings, where you can pick up a test CD and acoustical calibrator to measure your hearing ability.
• A Love Song For Napster, thought experiment on taking copyright protection to its logical conclusion (Discover Magazine, Feb '01)

Discussions

• Project Mayhem Discussion of all types of lossy formats
• r3mix.net Forum Discussion of mp3 (and other compression formats) with an emphasis on Lame.
• r3mix.net Forum Read-only archive as of May 22, 2001
• VQF.com User's Forum Discussion of audio compression, with emphasis on VQF. Now closed, but still archived.
• Headwize, Headphone forums
• Head-Fi.org, HeadWize regulars camped here while HeadWize revamped its forums.
• mp3encoder list: Emphasis on Lame development
• MAD-users list
• MP3.com old Message Board, Encoders Forum (read-only archive). The old MP3.com board was superior to the current board.

Test Sample Sounds

• My Audio Samples Page, containing various test cases described elsewhere on my site
• Sounds used to tune GPSYCHO (Lame psychoacoustic model)
• HydrogenAudio samples archive
• SQAM: Sound Quality Assessment Material
• Sound samples from Bo Lincoln's MUS420 project report
• Sounds on Joda's Space
• Jan Rafj's test cases
• Sounds in Arny Krueger's PCABX Reference Sample Download Page
• r3mix Test Signal Database
• Frank Klemm's Test Cases
• short clip of Pearl Jam's "Daughter", found on Speek's Listening Tests page
• Magnus Roth's samples used to test early beta version of Ogg Vorbis

Return to ff123's home page