Results of 64 kbit/s Listening Test

Results of 64 kbit/s Group Listening Test

Notes by ff123

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using Tukey's Honestly Significant Difference) for each codec.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the ATrain plot below, oggq0 and ogg64 are rated better than wma8 with 95% confidence. In turn, wma8 is rated better than aac with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Plot

Comment

Artist: Jazz-Preachers
Title: Take the A-Train
Album: Stax Space-Sound CD
Submitted by: ff123

Brass band

Results: The oggs win with mp3pro and wma8 tied in the second tier. aac rates well below the rest.

Title: Bach - Suite No. 1, S. 1007, G Major - Prelude
Album: Bach 6 Suites For Unaccompanied Cello
Cellist: Yo-Yo Ma
Label: CBS Masterworks
Submitted by: Amadeus93

Results: mp3pro, wma8, and ogg64 cluster at the top. oggq0 and aac are tied for worst.

Artist: Cradle Of Filth
Title: Beauty Slept In Sodom
Submitted by: Frank Klemm

Solo Harpsichord

Results: wma8 wins with mp3pro close behind. The oggs are well behind mp3pro, but well above aac.

Artist: Opeth
Title: Blackwater Park
Album: Blackwater Park

Quiet intro with acoustic and electric guitars followed by loud metal riffs.

Results: mp3pro is the winner with the oggs sharing second. aac and wma8 are far below the level of the others.

Artist: Man With No Name
Title: Floor-essence
Submitted by: Karl Heyes

Electronic mix

Results: oggq0 wins by a sizable margin, but mp3pro is better than the rest, which bunch at the bottom: ogg64, wma8, and aac.

Artist: Eric Clapton
Title: Layla (unplugged)
Album: The All Time Greatest Rock Songs
Submitted by: Jan S.

Applause at the beginning, followed by piano and acoustic guitar

Results: oggq0 ranks best followed by mp3pro. ogg64 is only slightly worse than mp3pro and slightly better than wma8. aac is last.

Artist: Desultory
Title: Life Shatters
Album: Bitterness
Track: 01
Submitted by: treech

Metal, complete with screeching vocals

Results: oggq0 is clearly best. mp3pro and ogg64 share second. aac is well below them, and wma8 is well below aac.

Artist: Minoru Nojima, pianist
Title: List B Minor Piano Sonata
Album: Nojima Plays Liszt
Label: Reference Recordings
Submitted by: ff123

Solo piano

Results: aac and mp3pro share first, and are closely followed by wma8. ogg64 is clearly below the level of the first two, but well above oggq0, which occupies last.

Artist: Mamas and Papas
Title: Midnight Voyage
Submitted by: Jan S.

drums and bass in the far left, guitar in the far right. Female vocal in the center.

Results: oggq0 and mp3pro are in a virtual dead-heat for first. ogg64 is runner up. aac is better than wma8, but both are far below the first three.

Artst: Tiamat
Album: A Deeper Kind Of Slumber
Track: 01
Submitted by: treech

rock/metal riffs

Results: mp3pro edges out the oggs, which are tied for second. aac and wma8 are far below the quality of the others.

Artist: Candy Staton
Track: The Source
Album: You've Got The Love
Submitted by: Karl Heyes

Drums and ride cymbals intro, followed by bass and female vocals.

Results: The oggs clearly occupy and share first place. mp3pro ranks in the second tier. wma8 and aac are far below this level.

Artist: Green Day
Title: Waiting
Album: Warning

Intro consists of guitar in far left with male vocal in center.

Results: all codecs rate very low, but mp3pro picks up the nod for least objectionable. oggq0 and ogg64 share second, and aac and wma8 occupy last.

Bitrates: The bitrates were calculated using the size of the file in bytes and the time in seconds with the following formula: bitrate = 0.008 * size / time. Note that some formats, for example the oggs, might contain a header, which adds to the bitrate for short samples. I don't know what the header sizes are for all the formats, but for ogg it is about 3 to 4 kB. The bitrates shown below do not correct for this header, and show roughly 1 to 2 kbit/s additional increase because of it.

sample		mp3pro	oggq0	ogg64	wma8	aac
Atrain		64	65	68	66	66
BachS1007	64	49	64	67	66
BeautySlept	64	64	68	67	66
Blackwater	64	60	68	69	66
FloorEssence	64	74	67	66	66
Layla		64	65	65	67	66
LifeShatters	64	70	65	68	66
LiszBMinor	64	45	65	69	66
MidnightVoyage	64	66	67	67	66
thear1		64	66	70	69	66
TheSource	64	65	68	67	66
Waiting		64	65	66	68	66

Overall Rankings: Any overall ranking is highly dependent on the mix of music chosen. If one compares all classical chamber music, wma8 might come out near the top. On the other hand, if one compares electronic dance music, oggq0 might win out. However, given the mix of music represented by these 12 samples, I employed the following method to try to rank the codecs overall: First, I eyeballed each graph and ranked the codecs from best to worst, giving the best codec a score of 5 and the worst a score of 1. If two codecs tied (for example for first and second), I gave both codecs the average score (4.5). Although the procedure is somewhat subjective, I think it produces reasonable results.

sample		mp3pro	oggq0	ogg64	wma8	aac
Atrain		2.5	4.5	4.5	2.5	1.0
BachS1007	4.0	1.5	4.0	4.0	1.5
BeautySlept	4.0	2.5	2.5	5.0	1.0
Blackwater	5.0	3.5	3.5	1.0	2.0
Floor-Essence	4.0	5.0	2.0	2.0	2.0
Layla		4.0	5.0	3.0	1.5	1.5
LifeShatters	3.5	5.0	3.5	1.0	2.0
LisztBMinor	4.5	1.0	2.5	2.5	4.5
MidnightVoyage	4.5	4.5	3.0	1.0	2.0
thear1		5.0	3.5	3.5	1.0	2.0
TheSource	3.0	4.5	4.5	1.5	1.5
Waiting		5.0	3.5	3.5	1.0	2.0

Ranksums:      49.0    44.0    40.0    24.0    23.0

Then I performed a Friedman non-parametric analysis. The results are graphed below. Mp3pro wins overall ranking points, but it's not significantly better than oggq0 or ogg64. On the other hand, wma8 and aac are definitely ranked at the bottom.