Read the instructions for the test
290 = xing, bitrate = 128 kbit/s 164 = wma8, bitrate = 130 kbit/s 549 = lame, bitrate = 125 kbit/s 429 = ogg, bitrate = 123 kbit/s 310 = aac, bitrate = 132 kbit/s 772 = mpc, bitrate = 140 kbit/s
Results to date:
xing wma8 lame ogg aac mpc 2.8 3.5 4.2 5.0 3.8 5.0 2.0 2.5 3.5 5.0 3.8 5.0 3.9 2.9 2.7 3.7 4.7 3.9 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 1.0 1.0 3.5 3.0 2.0 4.0 4.5 5.0 5.0 4.4 4.8 4.0 5.0 4.9 5.0 5.0 5.0 5.0 1.8 1.4 4.8 2.8 3.9 3.5 2.0 2.0 4.0 4.0 3.0 5.0 2.3 3.5 2.7 5.0 3.0 4.2 5.0 5.0 5.0 5.0 5.0 5.0 4.4 4.5 4.7 5.0 4.1 4.7 4.0 4.0 4.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 3.0 4.4 4.8 4.4 4.0 4.2 5.0 5.0 4.0 4.3 5.0 3.5 2.5 2.5 4.5 4.0 3.5 5.0 1.1 1.0 3.5 3.8 3.3 4.5 5.0 5.0 5.0 5.0 5.0 4.0 4.8 3.5 3.4 5.0 4.7 5.0 5.0 4.5 5.0 5.0 4.0 3.0 1.0 1.0 2.8 3.3 1.8 4.0 4.0 4.0 3.0 4.0 5.0 3.0 5.0 2.5 5.0 4.0 5.0 5.0 4.0 5.0 4.0 5.0 5.0 5.0 2.0 1.5 2.5 3.7 4.0 4.2 4.5 3.5 1.0 2.0 2.7 1.5
Resampling analysis, wayitis.wav:
28 listeners
Each listener's resampled values are chosen from his own pool of ratings
(blocked analysis).
Blocked model used to calculate unadjusted p-values.
100,000 bootstrap trials are performed to adjust p-values using free step-down.
Means:
ogg mpc aac lame xing wma8
4.34 4.33 4.15 4.02 3.59 3.52
Unadjusted p-values
mpc aac lame xing wma8
ogg 0.973 0.379 0.145 0.001* 0.000*
mpc - 0.397 0.154 0.001* 0.000*
aac - - 0.561 0.011* 0.004*
lame - - - 0.048* 0.021*
xing - - - - 0.740
Adjusted p-values
mpc aac lame xing wma8
ogg 0.974 0.848 0.539 0.007* 0.002*
mpc - 0.840 0.534 0.008* 0.003*
aac - - 0.915 0.081 0.036*
lame - - - 0.251 0.137
xing - - - - 0.932
Comparisons in italicized red below are true as a group with 95% confidence.
ogg is better than wma8
mpc is better than wma8
ogg is better than xing
mpc is better than xing
aac is better than wma8
aac is better than xing
lame is better than wma8
lame is better than xing
Listener 1 (termite)
Listener 2
(ff123)
Listener 3
(Marcel Bierling)
Listener 4
(xercist)
Listener 5
(Mark Hetherington)
Listener 6
(Filburt)
Listener 7
(Andreas Karlsson)
Listener 8
(Wikan)
Listener 9
(neoman)
Listener
10
Listener
11 (Garf)
Listener
12 (Eric Ding)
Listener
13 (roland)
Listener
14
Listener
15 (Alex Johns)
Listener
16
Listener
17 (Greg Wooledge)
Listener
18 (Hans Heijden)
Listener
19 (Volcano)
Listener
20 (C. H. C.)
Listener
21 (mikpos)
Listener
22 (Yannick)
Listener
23 (xiphmont)
Listener
24 (jack)
Listener
25
Listener
26 (Sunthelazar)
Listener
27
Listener
28 (Joerg Dietrich)
Listener
1 (termite):
Here's my results. My
first listening test so sorry that I don't know right terms for
different artifacts yet. Looking forward to see the results.
I took the piano sample because in first two ones the original
sample was already too annoying for me.
f290 (2.8) - Sound is quite
muffled whole time and artifacts especially in first 5 secs. 16/16
in ABX.
c164 (3.5) - 16/16 in ABX test. Some kind of metallic sound
compared to original.
b549 (4.2) - I heard some minor differences. Not too annoying
anyway. 14/16 in ABX test and 15/16 second time just to make sure.
d429 (5.0) - Couldn't differ from original.
a310 (3.8) - 16/16 in ABX. Same kind of metallic sound as in 164
but not as strong.
e772 (5.0) - Couldn't differ from original.
Listener
2 (ff123):
a290 (2.0) 16/16 ABX: obvious transient smearing of keyboard
attacks; wavery in the beginning notes
c164 (2.5) 16/16 ABX: one loud puff of noise near beginning; also
transient smearing or noise pumping during the keyboard attacks
d549 (3.5) 15/16 ABX: some noise pumping, some transient
smearing, a few wavery notes, but not nearly as bad as 290.
b429 (5.0)
e310 (3.8) 15/16 ABX: some transient smearing; but nowhere near
the level of 290; slightly better than 549 in amount of wavery
notes.
f772 (5.0)
Listener
3 (Marcel Bierling):
b290 3.9
e164 2.9
a549 2.7
c429 3.7
f310 4.7
d772 3.9
Listener
4 (xercist):
290 5.0
164 5.0
549 5.0
429 5.0
310 5.0
772 5.0
Listener
5 (Mark Hetherington):
290 5.0
164 5.0
549 5.0
429 5.0
310 5.0
772 5.0
Listener
6 (Filburt):
290 1.0 - watery, lots of pre-echo. Better than 164, but
still..it's annoying.
164 1.0 - absolutely awful pre-echo. Reedy sounding, shrill.
completely unacceptable
549 3.5 - similar to 429 except just to a bit lesser degree.
That's about it on that one. It's okay...but not great.
429 3.0 - watery, some sharp high frequency spikes. It's okay...but
I can and would choose to do better than this through some other
encoder.
310 2.0 - VERY noisy. Stereo is a bit off. Pre-echo makes it a
bit mushy sounding.
772 4.0 - Seems to be the least annoying. I'm not sure if it's
the best in every way...but it seems to have the least blatantly
obvious problems.
Listener
7 (Andreas Karlsson):
wayitis_a290.flac - 4.5 - Not so clear in the high frequencies.
wayitis_b164.flac - 5.0
wayitis_e549.flac - 5.0
wayitis_d429.flac - 4.4 - Missing bass.
wayitis_c310.flac - 4.8 - A slight remble in the high
frequencies, but not disturbing.
wayitis_f772.flac - 4.0 - To much high frequencies and loss of
bass.
Listener
8 (Wikan):
I rate waytis_164 4.9. And all other files 5.0. waytis_164 is
somehow different, but I can't describe how.
Listener
9 (neoman):
way it is - I didn't expect it, but this one tripped things up
the worst that
I heard.
290 1.8 VERY annoying warble on piano notes.
164 1.4 HORRIBLE buzzing/metalic/warble on piano notes. Could
barely stand to listen to this one. Worse even than 290.
549 4.8 Highs are slightly more prominent. Not particularly
annoying, though.
429 2.8 Strange. Sounds like most of the "tone" portion
of the long background notes have been sucked out, leaving only
white noise. Or, possibly, it's just a louder hiss drowning the
background notes.
310 3.9 Stereo image not very transparent. Strange kind of effect
towards the end of the clip. Almost like speakers out of phase,
or noise being cancled.
772 3.5 Much less full sound on the piano.
Listener
10:
wayitis_290.pac: 2.0 lots of artifacts in the piano
wayitis_164.pac: 2.0 electronic sloshiness in piano attack
wayitis_549.pac: 4.0 very slight artifacts in piano attack
wayitis_429.pac: 4.0 very slight artifacts in piano attack
wayitis_310.pac: 3.0 electronic sloshiness in piano
wayitis_772.pac: 5.0
Listener
11 (Garf):
290: 2.3 9/10 very audible smearing (short blocks anyone?)
164: 3.5 8/8 harsh/overbright, smearing in the HF
549: 2.7 6/6 sounds as if the piano player is hitting two notes
at the same time (maybe smearing)
429: 5.0
310: 3.0 9/10 like 549, but less. almost sounds like vibrato
772: 4.2 28/40 slight distortion on one of the notes. easy to
pick up, much harder to do so reliably
Listener
12 (Eric Ding):
290 5.0
164 5.0
549 5.0
429 5.0
310 5.0
772 5.0
Listener
13 (roland):
I did not have the opportunity to try ABX testing. I'm sorry. I
hope this helps somewhat. I used good equipment, and consider
myself quite a discerning listener, but I had much difficulty
with these files.
wayitis290 4.4 - noticeably thinner. SMALL artifacts in the piano
strikes. High frequencies seems muddier.
wayitis164 4.5 - sounds much like 290. better high
wayitis549 4.7 - very good. Hard to tell. Seems to have lost a
bit of bass, fullness.
wayitis429 5.0 - This was hard.
wayitis310 4.1 - much less high-end? And bass.
wayitis772 4.7 - I must be going crazy. I can't tell why i feel
this is a 4.7
Listener
14:
290 - 4 lacks presence
164 - 4 slgihtly muted high frequencies
549 - 4 lack presence (as above)
429 - 5 Pretty good
310 - 5 Pretty good
772 - 5 I could tell the difference but only just and there were
no obvious artifacts
Listener
15 (Alex Johns):
290 5.0
164 5.0
549 5.0
429 5.0
310 5.0
772 5.0
Listener
16:
wayitis_290: 3.0 16/16 pre-echo, detuned sound
wayitis_164: 4.4 14/16 slightly flanged
wayitis_549: 4.8 12/16 detuned sounding chords, like a chorus or
flanger
effect
wayitis_429: 4.4 14/16 ditto
wayitis_310: 4.0 16/16 pre-echo on louder chord strikes
wayitit_772: 4.2 15/16 sound is less brilliant, slightly flanged
Listener
17 (Greg Wooledge):
wayitis_e290.wav: rating = 5.0
wayitis_d164.wav: rating = 5.0
wayitis_f549.wav: rating = 4.0 (ABX 9/10)
Comment: there's some "smearing" of the piano, most
noticeable on the attacks in the opening few seconds.
wayitis_c429.wav: rating = 4.3 (ABX = 10/16)
Comment: errr.. frustrating. I can hear some distortion on the
attacks, as in 549, but I can't pin it down consistently.
wayitis_a310.wav: rating = 5.0
wayitis_b772.wav: rating = 3.5 (ABX = 13/16)
Comment: There's a continuous "low hiss" (mid-freq.
noise) in the right channel, which is most easily heard when the
instrument volume goes down (offset 11-16 seconds). Reminds me of
tape hiss, but more subtle.
Listener
18 (Hans Heijden):
290 2.5 fluttery, ringing
164 2.5 metallic, heavy noise pumping & lowpass
310 3.5 piano notes a little rough
429 4.0 light noise pumping
549 4.5
772 5.0
Listener
19 (Volcano):
Results for Volcano (Dominic Jefferies); Nov 8, 2001
290 1.1 - : I guess I don't have to explain what's wrong with
this one...
164 1.0 - : Sounds even worse than 290...
310 3.3 16/16 : Also smearing of the piano strikes [see 549 and
429].
429 3.8 16/16 : Slight smearing of the piano strikes; distorted
background hiss.
549 3.5 16/16 : Piano strikes are smeared; distortions at the 17-18
sec mark.
772 4.5 16/16* : With the treble control turned right up, I could
easily hear an inconsistent background noise that sometimes
becomes distorted (that's why I scored 16/16 runs in ABX). Under
normal conditions, I can hear it slightly, but it doesn't annoy
me. The piano strikes are reproduced well.
* with treble boost. The rating is based on normal conditions.
Listener
20 (C. H. C.):
290 5.0
164 5.0
310 5.0
429 5.0
549 5.0
772 5.0 not "full" sounding
Listener
21 (mikpos):
290 4.8
164 3.5
310 4.7
429 5.0
549 3.4
772 5.0
Listener
22 (Yannick):
290 5.0
164 4.5 (doesn't sound as full)
310 4.0 (pads in the background are a little bit mashed together)
429 5.0
549 5.0
772 3.0 (loss of echoes in the higher piano notes)
Listener
23 (xiphmont):
wayitis_290: ABX score 16/16 rating: 1.0
Good god. That's all there is to say. The stereo image/rotor
effect spends the whole sample underwater. Preecho is bad enough
to spook a moose.
wayitis_164: ABX score 16/16 rating: 1.0
...and again. Embarrassing. If you're going to lowpass, *put it
somewhere and leave it there*. Considering the dynamic lowpass on
this one sounded like it was dipping as low as 10kHz, you think
it at least wouldn't swirl [ha] or the preecho would be at least
a little better than in 290 [nope]. This sample would get a
negative score if it could; it's actually worse than 290.
wayitis_310: ABX score 16/16 rating: 1.8
random lowpassing/loss of treble that appears at every synth hit
then disappears again, artificial/mechanical high end. Background
noise keeps disappearing, reappearing, swirling from side to side.
"Next"
wayitis_429: ABX score 16/16 rating: 3.3
Uniformly bright, background noise mostly stays put... most of
the time. Noticable preecho/noise (hard to tell which) around
synth hits; made it trivial to zip through the test.
wayitis_549: ABX score 16/16 rating: 2.8
some noise/brightness swirling, not nearly as bad as in 310, but
still easy to notice. The first synth-piano chord was all it took
to pick it.
wayitis_772: ABX score 16/16 rating: 4.0
Good transients, rock solid background noise, brightness/high end
is utterly stable. Zero swirl. However, the synth hits were still
ever so slightly less crisp than the original. Easy to hear, not
annoying.
Other comments:
BTW, when I said 'uniformly bright' [referring to 429] I meant
'didn't sound like it was lowpassing to save its skin on every
synth hit like the others'. I said the same about MPC :-)
Ah! This was before all the preecho fixes!! That's how I picked
out 429. (I rewrote short block handling and added impulse blocks
for rc3; your pre-rc3 you're using doesn't have any of that. I
just encoded a new one, and it sounds much crisper (although I
can still ABX it).
Oh, that reminds me.... there are also treble masking fixes for
things Dibrom found in the actual rc3; the 8, 12 and 16kHz tone/noise
masking is different. So, come to think of it, some things *have*
changed markedly since the pre-rc3 you used.
When I randomize an official rc3 encoding into the ABX mix, it
beats the other codecs and has no wandering background noise.
Listener
24 (jack):
290 4 - 14/16. the first arpeggio was messed up. kinda muddy i'd
say.
164 4 - 13/16. same thing [as 429].
310 5 - 9/16. i couldnt' hear a damned thing.
429 4 - 10/16. i coudl only hear slight distortion on the first
arpegio.
549 3 - the whole thing was slightly fuzzy/tinny. it was the most
noticable of all of them.
772 3 - the first arpeggio was echo-y i thought.
Listener
25:
290 5
164 2.5 After any "hard" note, such as the piano hits,
there was a sound like a sand shaker, was a bit annoying and
obvious immediately.
310 5
429 4 The bass was more emphasised in 429 compared with orig,
resulting in a different note, esp noticed in the first bit of
bass in the sample.
549 5
772 5
I tried the ABX tests, some of the ones I thought I could hear a
difference originally, it turned out impossible to tell them
apart.
Listener
26 (Sunthelazar):
290 4
164 5
310 5
429 5
549 4
772 5
Listener
27:
Hi, I just did a
quick test with wayitis for your second round of 128-bitrate
tests. I'm really not very good at listening tests (I've only
been reading hydrogenaudio for a month), but I was able to ABX
all of the samples, probably based on pre-echo or transients,
which is the only kind of artifact I feel I can reliably hear.
Here are my ratings for the files, based purely on the first two
seconds of the sample:
wayitis_290: shaker sound on first piano chord is more like a
chirp, and the opening piano notes are flanged. ABX 10/10, rating:
(2.0)
wayitis_164: shaker sound on first chord is more like a swoosh.
ABX 8/8, rating: (1.5)
wayitis_310: shaker sound on second chord is more pronounced,
less so than 429 but moreso than 310. ABX 8/8, rating: (4.0)
wayitis_429: shaker sound is more apparent on second chord. ABX
12/12, rating: (3.7)
wayitis_549: sample sound on first chord is like a swoosh, but
less so than 164. ABX 8/8, rating: (2.5)
wayitis_772: shaker sound on second chord is more pronounced,
less so than 429 or 310 though. ABX 11/12, rating: (4.2)
Listener
28 (Joerg Dietrich):
I decided to
participate in your 128 kbit/s Blind Listening Test and rated the
wayitis files. I never did any blind listening tests of
compressed audio data until 4 days ago when I started tests
looking which format to use for archiving my LPs. So I'm
obviously not the most experienced listener, but I surprised
myself quite a bit when I first heard all the differences to the
uncompressed audio. Anyway, here are the results in my listening
order:
772 has a metallic sound and very weak basses, ABX: 14/16. I rate
this one 1.5.
429 has a wavering sound and a deep roaring hiss, ABX: 15/16.
This one gets a 2.0.
290 this sounds best to me. I had a lot of trouble hearing a
difference at first (well, I said my ears not trained to hear
encoder artefacts) and the ABX test looked more like random in
the beginning (until 5/9). Then I started to hear some light
flanging in the piano attacks, and ended with 11/16. Because then
I was convinved that I could hear a difference I continued until
I had 15/20. Don't know what the significance of this result is,
but I vote 4.5 for this one.
549 is worst of all. Extreme flanging, gurgling sounds and the
piano sounds horribly out of tune. Most have done something
horrible to the overtones. ABX: 15/16 could have been 16/16 if I
had concentrated more. This one gets not more than 1.0.
164 was better (second best actually). I distinguished this one
by the pre-echo of the piano. I vote 3.5 on this.
310 would have been nice if it wasn't for the hissing I hear all
over. ABX: 12/16 was hampered by a lot of processor noise. I give
this one a 2.7.
Some words about the setup: SB PCI64 under Linux with Alsa 0.5.12
drivers, into a rather poor quality Kenwood KR-A4020 receiver. I
listened to it over my Sennheiser HD475 headphones.
The PC is next to me and with its power supply fan, CPU fan and
two noisy hard disks anything but quiet. Plus it's working all
the time, so I had a lot processor and harddisk noise.