Read the instructions for the test
270 = xing, bitrate = 128 kbit/s 933 = wma8, bitrate = 130 kbit/s 563 = lame, bitrate = 127 kbit/s 304 = ogg, bitrate = 121 kbit/s 129 = aac, bitrate = 132 kbit/s 021 = mpc, bitrate = 138 kbit/s
Results to date:
xing wma8 lame ogg aac mpc 4.7 4.6 4.7 4.7 3.4 3.6 4.5 5.0 5.0 4.8 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 4.2 4.8 4.9 5.0 5.0 4.7 3.2 3.0 2.3 3.8 4.0 2.5 4.5 4.5 5.0 5.0 5.0 4.5 2.8 5.0 2.8 3.2 3.8 3.2 3.5 4.5 3.8 4.5 4.5 4.0 5.0 5.0 5.0 5.0 5.0 5.0 3.5 4.3 5.0 4.7 5.0 3.2 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 4.0 5.0 5.0 5.0 5.0 4.0 5.0 4.0 5.0 4.0 4.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 4.0 4.2 5.0 4.1 2.7 2.5 3.0 4.0 4.3 3.5 4.0 5.0 5.0 4.8 4.4 4.6 3.7 5.0 3.5 4.6 4.9 4.8 3.5 3.0 4.0 5.0 5.0 4.5 5.0 5.0 4.0 4.0 5.0 4.0 5.0 5.0 4.0 5.0 5.0 5.0 4.0 4.0 5.0 4.0 3.0 3.0 2.8 3.0 4.0 4.2 2.8 4.5 4.0 5.0 4.0 3.5 4.0 3.5 5.0 4.0 4.0 4.0 5.0 5.0
Resampling analysis, rawhide.wav:
26 listeners
Each listener's resampled values are chosen from his own pool of ratings
(blocked analysis).
Blocked model used to calculate unadjusted p-values.
100,000 bootstrap trials are performed to adjust p-values using free step-down.
Means:
aac ogg wma8 lame mpc xing
4.54 4.54 4.47 4.31 4.28 4.22
Unadjusted p-values
ogg wma8 lame mpc xing
aac 0.979 0.611 0.104 0.066 0.024*
ogg - 0.630 0.110 0.070 0.026*
wma8 - - 0.261 0.182 0.079
lame - - - 0.830 0.520
mpc - - - - 0.668
Adjusted p-values
ogg wma8 lame mpc xing
aac 0.984 0.978 0.505 0.413 0.206
ogg - 0.978 0.505 0.417 0.209
wma8 - - 0.781 0.665 0.435
lame - - - 0.978 0.959
mpc - - - - 0.978
Comparisons in italicized red below are true as a group with 95% confidence.
aac is better than xing
ogg is better than xing
Listener 1 (Marcel
Bierling)
Listener 2
(xercist)
Listener 3
(Mark Hetherington)
Listener 4
(Andreas Karlsson)
Listener 5
(Garf)
Listener 6
Listener 7
(neoman)
Listener 8
(ff123)
Listener 9
Listener
10
Listener
11 (Wikan)
Listener
12 (Eric Ding)
Listener
13
Listener
14
Listener
15 (Alex Johns)
Listener
16 (Greg Wooledge)
Listener
17 (Filburt)
Listener
18
Listener
19 (2BDecided)
Listener
20 (Hans Heijden)
Listener
21 (C. H. C.)
Listener
22 (mikpos)
Listener
23 (Yannick)
Listener
24 (xiphmont)
Listener
25 (jack)
Listener
26 (Sunthelazar)
Listener
1 (Marcel Bierling):
d270 4.7
b933 4.6
a563 4.7
e304 4.7
f129 3.4
c021 3.6
Listener
2 (xercist):
270: 4.5 - I could only
identify the difference by listening carefully to how the codec
changed the defect in the original, near the word 'weather'.
Guessed correctly 11/12 times.
933: 5.0
563: 5.0 - Defect in original seems less pronounced with this
codec, but I wasn't able to detect the subtle change when testing.
304: 4.8 - Again I could only tell via the 'weather' defect. 10/12.
129: 5.0
021: 5.0
Listener
3 (Mark Hetherington):
270 5.0
933 5.0
563 5.0
304 5.0
129 5.0
021 5.0
Listener
4 (Andreas Karlsson):
rawhide_b270.flac - 4.2 - Missing some bass.
rawhide_f933.flac - 4.8 - A minor distortion in the high
frequencies.
rawhide_c563.flac - 4.9 - Just slightly to much bass.
rawhide_e304.flac - 5.0
rawhide_a129.flac - 5.0
rawhide_d021.flac - 4.7 - There is some phasing in the mid
frequencies.
Listener
5 (Garf):
270: 3.2 12/16 'swollen' has an underwater sound
933: 3.0 12/16 as above
563: 2.3 12/16 very noisy overall
304: 3.8 13/16 noise masking trouble in the rythm section on the
background
129: 4.0 12/16 (forgot to write down)
021: 2.5 14/16 sounds unrealistic & noisy
Listener
6:
4.5 rawhide_c270.wav
4.5 rawhide_a933.wav
5 rawhide_f563.wav
5 rawhide_d304.wav
5 rawhide_e129.wav
4.5 rawhide_b021.wav
Listener
7 (neoman):
Rawhide - Most of the problems had to do with the "fullness"
of the sound.
270 2.8 Bass is muddier. It would override the rest of the sound
during it's louder spikes.
933 5.0 Can't hear the difference.
563 2.8 Voice was thinner and more prominent. Not really that
annoying, but noticeable. Much more annoying is the metalic buzz
introduced on the guitar.
304 3.2 The guitar work has a much reduced image. It sounds lost
in the mix.
129 3.8 Thin voice again. Most noticable when he sings "..side"
at the end of the clip
21 3.2 Bass line is much less substantial. Mids and highs seemed
ok, though.
Listener
8(ff123):
270: 3.5 16/16 ABX, more wavery than 563, especially
during "through rain and wind and weather"
933: 4.5 14/16 ABX, same as 129 and 304 -- less background noise
than original, and not a lot of wavering.
563: 3.8 14/16 ABX, slight waver in the beginning and again
during "through rain and wind and weather"
304: 4.5 14/16 ABX, same as 129 -- less background noise than
original, and not a lot of wavering.
129: 4.5 14/16 ABX, the backgound noise is *less* than it is in
the original. There's no wavering (which is a more annoying
artifact to me), but it's still different from the original.
021: 4.0 15/16 ABX, although there's not a lot of waver, there's
a background buzzing sound, present in the original, but which is
louder in the encode. This is less annoying than the slight
wavering in 563 to me, but more annoying than 129, 304, and 933,
where the background buzzing is less.
Listener
9:
rawhide_270.pac: 5.0
rawhide_933.pac: 5.0
rawhide_563.pac: 5.0
rawhide_304.pac: 5.0
rawhide_129.pac: 5.0
rawhide_021.pac: 5.0
Listener
10:
270 - 3.5 - some distortion of voice, noticeable on
"th" in weather
933 - 4.3 - sounds a little sandpapery
563 - 5.0 - nothing I could hear
304 - 4.7 - seems a little quieter than the original
129 - 5.0 - nothing I could hear
021 - 3.2 - annoying rough subsounds
Listener
11 (Wikan):
rawhide_270.pac: 5.0
rawhide_933.pac: 5.0
rawhide_563.pac: 5.0
rawhide_304.pac: 5.0
rawhide_129.pac: 5.0
rawhide_021.pac: 5.0
Listener
12 (Eric Ding):
270 - 5.0
933 - 5.0
563 - 5.0
304 - 5.0
129 - 5.0
021 - 5.0
Listener
13:
Rawhide_270 5
Rawhide_933 4 Sharper high end.
Rawhide_563 5
Rawhide_304 5
Rawhide_129 5
Rawhide_021 5
Listener
14:
270 - 4 sounds flat
933 - 5 couldn't tell the difference
563 - 4 sounds flat
304 - 5 pretty good
129 - 4 looses the dynamics
021 - 4 sounds flat
Listener
15 (Alex Johns):
270 - 5.0
933 - 5.0
563 - 5.0
304 - 5.0
129 - 5.0
021 - 5.0
Listener
16 (Greg Wooledge):
rawhide_a270.wav: rating = 5.0
rawhide_b933.wav: rating = 5.0
rawhide_e563.wav: rating = 4.0 (ABX = 13/16)
Comment: distortion/echo in the parts with vocals. Not as high-freq.
as 304's echos; more of a mid-freq. noise. Subtle, but slightly
more noticeable (to me) than the high-freq. echoes.
rawhide_c304.wav: rating = 4.2 (ABX = 9/13 but I had the first 8/8)
Comment: some high-freq. echo of the vocals on the high notes at
the end of each phrase (and esp. "hell-bound for leather").
Very subtle.
rawhide_d129.wav: rating = 5.0
rawhide_f021.wav: rating = 4.1 (ABX = 12/17)
Comment: very similar to 304: high-freq. echo/distortion, most
noticeable in the vocals (esp. "rain and wind and weather").
Still very subtle. I think I might be able to detect this one
more reliably if my ears were not so "tired" (this file
was presented 6th).
Listener
17 (Filburt):
basically what gave them most away was change in the
stereo image, and flange on the cymbals
129 seemed to have the least amount of it
304 was close, but had a slight shift in the stereo image...so i
rated it a bit lower
304 was a bit noisier than 129
270- 2.7
933- 2.5
563- 3
304- 4
129- 4.3
021- 3.5
Listener
18:
rawhide_270: 4.0 16/16 fluttery vocals
rawhide_933: 5.0
rawhide_563: 5.0
rawhide_304: 4.8 12/16 added noise on far left in "HIDE"
(time index ~7.5)
rawhide_129: 4.4 14/16 fluttery vocals
rawhide_021: 4.6 13/16 vocals slightly hoarse
Listener
19 (2BDecided):
I've tried rawhide now, and I found it even harder than
fossiles. Sometimes, there wasn't one strong artefact which I
could identify and remember, but there was a general "sense"
that the track wasn't quite right. ABXing allowed me to switch
between versions instantly to notice this, and prove/disprove the
existence of a real difference.
b=270=3.7
f=933=5.0
d=563=3.5
e=304=4.6
c=129=4.9
a=021=4.8
270 and 563 were poor. 563 suffered from a horrible lack of high
frequency content - like a variable low-pass at too low a
frequency! 270 had the typical watery codec sound, and the vocal
sounded quite closed in due to this. I didn't like these two!
304 - the background noise was slightly louder (or maybe the
track was brighter). It wasn't at all annoying, but it was quite
audible.
I can't tell you what was wrong with 021, but I could reliably
pick out the difference between it at the original. The
difference was similar to that found between two hi-fi
components, rather than two audio codecs - 021 didn't sound quite
as "real" as the original. 129 was similar, but the
difference was less obvious - only 6/8 then 7/8 in ABX.
I couldn't hear a difference between 933 and the original - ABX 4/9!!!
Hope this helps your stats! apologies if it doesn't.
Listener
20 (Hans Heijden):
rawhide: sorted on highs only
270 3.5
933 3.0
563 4.0
304 5.0
129 5.0
021 4.5
Listener
21 (C. H. C.):
270 5.0
933 5.0
563 4.0 bass not tight
304 4.0 boomy bass
129 5.0
021 4.0 slight distortion on sibilants
Listener
22 (mikpos)
270 5.0
933 5.0
563 4.0
304 5.0
129 5.0
021 5.0
Listener
23 (Yannick)
270 4.0 (doesn't sound as full)
933 4.0 (voice echoes not as crisp)
563 5.0
304 4.0 (voice is not as thick)
129 3.0 (voice is not as thick, bass seems a bit distorted)
021 3.0 (voice is not as thick and loses a lot of definition)
Listener
24 (xiphmont)
rawhide_270: ABX score 16/16 rating: 2.8
lots of swrling in noise, the voice consonants especially swirl.
rawhide_933: ABX score 16/16 rating: 3.0
Unvoiced consonants swirl slightly, but even more take on an odd
quality different from the original, almost as if the perceived
pitch of the noise is shifting downward.
rawhide_563: ABX score 16/16 rating: 4.0
very slight differences on a few fronts. Ever so slightly not as
sharp, some loss of definition in unvoiced consonants and symbol,
a tiny bit of swirl. perceptable, but not annoying.
rawhide_304: ABX score 16/16 rating: 4.2
Honestly, this one was only very very slightly easier to pick out
than 021, but for the same reason; extra slight fuzz, most
noticable in the sides of the image.
rawhide_129: ABX score 16/16 rating: 2.8
Considering how good the others sounded on this sample, this one
was embarrassing. Tons of swirl, random loss of brightness,
across the board loss of definition. The others took alot of
concentration to pick out; this one I just kept hitting keys.
rawhide_021: ABX score 14/16 rating: 4.5
I barely noticed why it Didn't Sound Right: extra noise,
especially in the sides of the stereo image. A fuzzy, gravelly
sound permeates the image/spectrum.
Listener
25 (jack)
270 4 - 12/16 - on 'weather' i can hear a slight
distortion
933 5
563 4 - 12/16 - can hear a different on 'rain and wind'
304 3.5 - 13/16 - on 'wea' it sound like it drops out
129 4 - 13/16 - sibilants ('s'tream's' are 's'wollen) are messed
up
021 3.5 - 23/32 - slight noise everywhere
Listener
26 (Sunthelazar)
270 5
933 4
563 4
304 4
129 5
021 5