Read the instructions for the test
200 = xing, bitrate = 128 kbit/s 234 = wma8, bitrate = 127 kbit/s 321 = lame, bitrate = 125 kbit/s 413 = ogg, bitrate = 127 kbit/s 299 = aac, bitrate = 132 kbit/s 854 = mpc, bitrate = 136 kbit/s
Results to date:
xing wma8 lame ogg aac mpc 3.6 3.3 2.8 3.4 3.9 3.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 4.2 4.5 3.8 4.9 3.1 4.1 5.0 5.0 3.5 2.5 5.0 4.0 4.0 4.5 4.9 4.0 5.0 4.2 3.5 4.9 4.9 4.9 3.9 2.7 5.0 5.0 4.0 3.9 5.0 5.0 2.5 3.5 5.0 3.0 3.8 3.9 4.7 5.0 4.7 4.7 5.0 4.5 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 4.0 4.0 5.0 5.0 3.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 4.0 4.0 3.0 4.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 2.5 5.0 3.5 4.0 4.5 4.0 4.0 5.0 4.9 4.0 4.0 4.9 3.0 5.0 5.0 5.0 5.0 5.0 3.0 4.0 4.5 5.0 4.0 4.5 5.0 4.0 5.0 4.0 5.0 5.0 3.9 4.4 4.8 3.7 4.0 5.0 5.0 5.0 5.0 4.0 4.5 5.0 3.5 3.5 4.5 5.0 3.0 4.8 5.0 5.0 5.0 5.0 5.0 5.0
Resampling analysis, fossiles.wav:
25 listeners
Each listener's resampled values are chosen from his own pool of ratings
(blocked analysis).
Blocked model used to calculate unadjusted p-values.
100,000 bootstrap trials are performed to adjust p-values using free step-down.
Means:
wma8 lame aac mpc ogg xing
4.58 4.55 4.51 4.50 4.36 4.30
Unadjusted p-values
lame aac mpc ogg xing
wma8 0.846 0.646 0.628 0.177 0.083
lame - 0.790 0.771 0.247 0.123
aac - - 0.981 0.371 0.201
mpc - - - 0.384 0.210
ogg - - - - 0.699
Adjusted p-values
lame aac mpc ogg xing
wma8 0.991 0.987 0.987 0.738 0.507
lame - 0.991 0.991 0.805 0.626
aac - - 0.991 0.917 0.770
mpc - - - 0.915 0.771
ogg - - - - 0.991
Comparisons in italicized red below are true as a group with 95% confidence.
Listener 1 (Marcel
Bierling)
Listener 2
(xercist)
Listener 3
(Mark Hetherington)
Listener 4
(Andreas Karlsson)
Listener 5
(Garf)
Listener 6
Listener 7
Listener 8
Greg Wooledge)
Listener 9
(neoman)
Listener
10 (Shaun)
Listener
11
Listener
12 (Wikan)
Listener
13
Listener
14 (Eric Ding)
Listener
15
Listener
16 (Alex Johns)
Listener
17 (ff123)
Listener
18 (2BDecided)
Listener
19
Listener
20 (Hans Heijden)
Listener
21 (C. H. C.)
Listener
22 (mikpos)
Listener
23 (Yannick)
Listener
24 (xiphmont)
Listener
25 (Sunthelazar)
Listener
1 (Marcel Bierling):
f200 3.6
e234 3.3
b321 2.8
d413 3.4
c299 3.9
a854 3.0
Listener
2 (xercist):
I tried my hardest not
to give 5's, but you can see I gave many. I suppose
my ears arn't what they used to be *sigh*.
200 5.0
234 5.0
321 5.0
413 5.0
299 5.0
854 5.0
Listener
3 (Mark Hetherington):
I downloaded all the files in FLAC format, found a FLAC
decompresser, and had a good listen to the tracks (and linABX'd
some of them). All it did was show me how bad my hearing must be!!!!!
I can usually tell a 128k mp3 when I hear one, and so I have been
using Lame at 192k (couldn't get VBR to work too well- the
encoder was crashing?? Probably my effort at compilation!!). Now
I've switched to OGG Vorbis at 160k (nominal), and I'm much
happier with the sound in general, it sounds a lot more lifelike
than the mp3s.
I'm listening to all this on either my handbuilt Playmaster
Amplifier, and handbuilt speakers. I'm quite proud of them having
built them myself, and at least to me they sound very comparable
to the best I've heard in hi-fi stores. They aren't the best of
the best, but the are in the hi-fidelity range. Sound source is
of course the computer, through a SB Live. If your interested and
it helps your survey specs of the amp are: 185W RMS, 0.005% THD
at 100W RMS into 8 ohms, and 0.008% at 180W RMS. IMD 0.005% at
180W RMS (50hz/7Khz 4:1 ratio). All of that is into 8 ohms. >100dB
signal to noise (unweighted) with respect to 100W.
I'm listening further, but all it's doing is making me more
dissapointed in my ears..... (I'm pretty sure all my listening
equipment is up to scratch).
So I'm basically going to have to give a 5.0 for all of them. All
my linabx tests have given fairly inconclusive results, not many
results were too close to 12, let alone above it.
I hope that helps in some way.
Listener
4 (Andreas Karlsson):
The diffrences was much easier to find in the fossiles_*.* :)
200.flac - 4.2 - No distortions but the treble/bas balance is odd.
More .
234.flac - 4.5 - Very good. Just some glitch in the high
frequesncies, .
321.flac - 3.8 - Slight phaseshifting in the high frequencies and
a less .
413.flac - 4.9 - Almost perfect. Sounds just a bit "cold".
299.flac - 3.1 - Missing a lotīs of frequencies. I must
compensate to enjoy imagine sound).
854.flac - 4.1 - Sound noisy in the mid frequencies. Much like
encoded a 44.4kHz sample.
Listener
5 (Garf):
200: 5.0
234: 5.0
321: 3.5 8/10 as [854] but additional smearing
413: 2.5 6/6 same as [321] but worse
299: 5.0
854: 4.0 14/16 flutter in the lively parts
Listener
6:
4.0 fossiles_c200.wav sounds a little flatter
4.5 fossiles_a234.wav valve sounds not as noticeable
4.9 fossiles_f321.wav
4.0 fossiles_b413.wav High-pitched squeaking noticeable
5.0 fossiles_e299.wav I couldn't tell the difference
4.2 fossiles_d854.wav tinny
Listener
7:
fossiles_200:
-------------
noticeable reduction in clarity of high frequency content, mid-range
hissing. Rating: 3.5
fossiles_234:
-------------
slightly flattens sound; otherwise almost perfect. Rating: 4.9
fossiles_321:
-------------
very, very slight loss of high frequency clarity. Rating: 4.9
fossiles_413:
-------------
slightly flattens sound; otherwise almost perfect. Rating: 4.9
fossiles_299:
-------------
flattens tone, emphasizes high range, doesn't sound *right*.
Rating: 3.9
fossiles_854:
-------------
less spacious, noticeable smearing of high frequencies, audible
high
frequency buzz, mid-range hissing. Rating: 2.7
Listener
8 (Greg Wooledge):
fossiles_d200.wav: rating = 5.0
fossiles_a234.wav: rating = 5.0
fossiles_b321.wav: rating = 4.0 (ABX = 12/16)
Comment: during the final seconds, when the instruments all play
together, there's some distortion/noise/echo in the background.
It's subtle, and hard to hear. Also, the violinist's "string
noise" (at about 7.0s) seems to be a bit "harsher"
in the encoded song, although I can't consistently judge from
that alone.
fossiles_e413.wav: rating = 3.9 (ABX = 13/16)
Comment: same as for 321, just *slightly* easier to detect.
fossiles_f299.wav: rating = 5.0
fossiles_c854.wav: rating = 5.0
Listener
9 (neoman):
Fossiles - There is a sort of high pitched tone/harmonic over
some of the
notes in the origional. Several of the samples magnified this
sound.
200 2.5 Much the same as 234, except the ring is even worse.
234 3.5 Strange ring/high pitched distortion at the end of some
of the notes. It sounds like it might be exagerating a noise that
exists in the origional.
321 5.0 As far as I can tell, they are similar.
413 3.0 Instruments in the background seem to "merge"
into one towards the end of the clip.
299 3.8 Same ring, but the rest of the clip sounds fuller than
many of the others.
854 3.9 Looses some fullness. It sounds accurate, but like it's
though a cheap(er) pair of headphones.
Listener
10 (Shaun):
I think I have a terrible ear - all the files sounded great to me
:-)
fossiles_b200.pac 4.7
fossiles_a234.pac 5
fossiles_d321.pac 4.7
fossiles_f413.pac 4.7
fossiles_c299.pac 5
fossiles_e854.pac 4.5
Listener
11:
fossiles_200.pac: 5.0
fossiles_234.pac: 5.0
fossiles_321.pac: 5.0
fossiles_413.pac: 5.0
fossiles_299.pac: 5.0
fossiles_854.pac: 5.0
Listener
12:
fossiles_200.pac: 5.0
fossiles_234.pac: 5.0
fossiles_321.pac: 5.0
fossiles_413.pac: 5.0
fossiles_299.pac: 5.0
fossiles_854.pac: 5.0
Listener
13:
I had a hard time distinguishing - they all sounded really good.
<70% accurate on ~12 trials resulted in a '5' rating. Tested
using ABX, headphones, on W2K PC w/SB Live Sound Card:
Fossiles_200 5
Fossiles_234 4 brittle High End, or too much treble.
Fossiles_321 4 I just let my ears direct me to which sounded like
orig and had good results ('guessing'). 321 sounded good, just
not quite the same.
Fossiles_413 5
Fossiles_299 5
Fossiles_854 3 Sorry, I don't remember the problem here :( I
didn't write it down. But this was the only one that I had little
difficulty discriminating. My recollection was that the sound was
a 'grainy'.
Listener
14 (Eric Ding):
I suppose it just means I have wooden ears, but I couldn't hear
perceptible differences in any of the samples. It's only a useful
data point, I suppose, in showing that for some folks, it doesn't
matter which format is used, even at 128 kbps.
fossiles_200: 5.0
fossiles_234: 5.0
fossiles_321: 5.0
fossiles_413: 5.0
fossiles_299: 5.0
fossiles_854: 5.0
Listener
15:
200 - 5 unable to tell the difference
234 - 4 Lacks a lot of presence
321 - 4 Missing v high frequencies / slightly muffled
413 - 3 sounds muffled and high frequencies are muted / slightly
electronic sounding
299 - 4 lacks presence (I think some of the low frequencies are
missing)
854 - 5 Pretty good
Listener
16 (Alex Johns):
First, the caveats: I'm listening on my work machine - I'm not
sure what kind of sound card it has. I have a cheapo lineout
cable from the sound card out to split inputs on an old boombox,
sitting on my desk. From there, I have some Kenwood KPM410
headphones plugged into the headphone jack of the boombox. I set
the equalizer on the boombox to neutral and I turned it off in
Winamp. So I don't have great equipment, but it's a test of some
kind.
So... Well, I can't tell the difference. They all sound the same.
I've listened to all 3 snippets in all 7 ways and they all sound
clear and perfectly listenable. I do have some ability to discern
a bad RIP - I 've been busy ripping at home and most seem to come
out fine, but there's a Roger Miller's Greatest Hits that for
some reason came out bad recently. My wife said it sounded like
it was coming from a well.
So, either there's no qualitative difference between the
compression methods, or I have really bad ears, or (most likely)
if you listen on bad enough equipment, it really doesn't matter.
I would run this test at home where I have much better equipment,
but I still only have modem access there and I really can't see
spending the time to download all these files.
(Sorry.)
I have no idea what you want to do with this. I almost didn't
send it to you since it seems worthless, but I thought that
perhaps it might help for statistical purposes. Hope it helps in
some fashion.
Listener
17 (ff123):
200: 2.5 16/16 ABX: heavy transient smearing, moderate fluttering
234: 5.0
321: 3.5 16/16 ABX: moderate transient smearing
413: 4.0 16/16 ABX: slight transient smearing, also slight
fluttering
299: 4.5 15/16 ABX: very slight wavering in beginning notes, but
not a lot of transient smearing
854: 4.0 16/16 ABX: noisier than the original, as in not as
clear, however not a lot of transient smearing
Listener
18 (2BDecided):
I've taken the time this afternoon to sit down and listen
carefully to
Fossiles. Usual equipment: CardD+, external DAC, Sony MD acting
as
amplifier and headphones. There is a fair amount of background
noise
around here today, so if there's any very quiet background sound
problems, I'll have missed them. However, I could hear problems
in all
but one excerpt.
randomized:
a=854
b=234
c=321
d=200
e=299
f=413
Best = b = 5.0 - can't hear any problems
Next best = a = 4.9. ABX = 6/8 (twice) so that's just maybe I can
hear a
difference. I had to turn the volume up (all the rest were
auditioned at
a sensible level, where the loudest part wasn't too loud), and
concentrate on the first xylophone note. You can just hear that
it has
slightly more of the brightness/rasping from the strings behind
it than
in the original. This is so hard to hear that it's on the verge
of
imagining it - the ABX seems to confirm that I'm [i]just[/i]
hearing
something.
c = 4.9. ABX 8/8. Slight smearing of the xylophone notes, but
it's not
too bad. It wouldn't annoy me.
d, e, and f I've given 4.0. I can hear different problems with
them, but
can't decide which I find more annoying...
d is just the smearing of the xylophone notes. Wouldn't really
want to
listen to this one.
e is weird - again there's some smearing. More significantly:
right at
the start, and also somewhere mid-way through the extract, it
sounds as
though it's being starved of bits - the high frequencies (the
harmonics)
are cut out.
f is nasty. I think it's the slight underwater bubbly kind of
sound
behind the whole track, but it makes it sound slightly out of
tune to my
ears.
I've tried, but I can't decide which of d, e, and f I dislike
more -
they're so different, but the annoyance is the same magnitude
with each.
Listener
19:
fossiles_200: 3.0 16/16 pre-echo in xylophone, generally unclean
sound
fossiles_234: 5.0
fossiles_321: 5.0
fossiles_413: 5.0
fossiles_299: 5.0
fossiles_854: 5.0
Listener
20 (Hans Heijden):
200 3.0 fluttery highs
234 4.0 quiet parts have less highs (3rd second)
321 4.5 rawness
413 5.0
299 4.0 overall less highs
854 4.5 rawness
Listener
21 (C. H. C.):
200 5.0
234 4.0 slight distortion L channel percussion
321 5.0
413 4.0 slight coloration
299 5.0
854 5.0
Listener
22 (mikpos):
200 3.9
234 4.4
321 4.8
413 3.7
299 4.0
854 5.0
Listener
23 (Yannick):
200 5.0
234 5.0
321 5.0
413 4.0 (xylophone doesn't sound as crisp)
299 4.5 (just doesn't sound as good)
854 5.0
I was using XMMS, this is
all IMO of course, I'm not a music expert.
If you require any additionnal info don't hesitate to ask.
ps: Just a thought for the next set of blind tests, maybe you
should try with more intricate music that stimulates more of the
spectrum so it would expose the flaws more, but then again at 128
they probably all would have some deterioration, just a matter of
finding out who does the better job.
Listener
24 (xiphmont):
fossiles_200: ABX score 16/16, rating: 3.5
The original has some harsh, strong, peaky brightness in the high
treble that this codec swirls. The effect is instantly noticable
(I blew through the 16 comparison ABX test in about 30 seconds),
but not in the 'unlistenable' category.
fossiles_234: ABX score 16/16 rating: 3.5
A weird lowpass keeps creeping down over the treble, sapping
brightness out... then disappearing again. It's fairly subtle,
but it's there and easy to hear.
fossiles_321: ABX score 15/16 rating: 4.5
There's no single noticable artifact in this sample, but there is
a general perception of slight loss of refinement. In the first
few comparisons, I had to go back to the original A/B to check,
but after the first four or so (in which I chose wrong once) I
was able to zip through the end. This sample was ever so slightly
brighter than the original.
fossiles_413: ABX score 11/16 rating: 5.0
fossiles_299: ABX score 16/16 rating: 3.0
more flanging than any of the other codecs. Background noise
swirls. This one was easy.
fossiles_854: ABX score 14/15 rating: 4.8
This is probably as close as I could come to being able to
somehow hear some difference with much straining as opposed to
being baffled. Some of the noisiness in the strings is ever so
slightly different from the original. Although I'm not in an
ideal listening environment, I'm being otherwise careful, and
this one was barely perceptibly different.
Listener
25 (Sunthelazar)
200 5.0
234 5.0
321 5.0
413 5.0
299 5.0
854 5.0