The 64 kbit/s Listening
Test plan and method are shown here.
The 64 kbit/s
Listening Test results are here.
Raw Listener comments and ratings. Ratings which were discarded or which were received after the close of the test are not included below. Ratings were discarded if:
1. The original associated with any codec was
rated less than perfect
2. The listener pulled a slider down just to enter a comment
about how he couldn't tell the difference
3. ABX was performed, at least one trial was incorrect, and the
pvalue (including the choice of slider as a trial) was > 0.05.
ATrain
BachS1007
BeautySlept
Blackwater
FloorEssence
Layla
LifeShatters
LisztBMinor
MidnightVoyage
thear1
TheSource
Waiting
The following are the comments of Chris Montgomery ("Monty"), the primary developer of Ogg Vorbis's psychoacoustic algorithms. He offered some predictions about the outcome of the test.
I'm excusing myself from this one :-)
However, I wanted to make some 'pre-test certified guesses' as to
results. Just for fun:
This test, frankly, is between Ogg and Mp3 Pro. Overall AAC and
WMA were abysmal, although WMA shone on one or two easy samples
(the classical) where the metallic lowpass didn't come crashing
down.
By sample:
ATrain.wav: Ogg wins by a small but reliable margin. MP3 Pro is
in a respectable second place. The Ogg sample picks up some
noise, the mp3pro sample flanges and muffles a bit. The WMA and
AAC aren't worth mentioning.
BachS1007.wav: An easy sample, relatively. This will have all of
MP3Pro, Ogg and WMA very close together. The winner is likely Ogg
-b64 --managed or WMA. Mp3 Pro very close behind. AAC loses due
to lowpass.
BeautySlept.wav: Ogg dies miserably on the sharp attacks in the
left channel because it has no stereo-aware masking yet. The
noise bleeding into center is instantly apparent and impossible
to ignore. This one will be kept for future artifact removal
tuning. Winner: mp3pro, very close behind is WMA (there's some
metallic lowpass in the middle) and AAC in third due to lowpass.
Ogg dies miserably here and will finish last by a large margin.
Annoying especially as this one should have been easy.
Harpsichord is spectrally simple.
Blackwater.wav: Ogg and mp3pro are very close; I think Ogg has a
slight edge at the beginning. Indistinguishable in the middle.
AAC and WMA are not worth mentioning.
FloorEssence.wav: Ogg by a sizable margin, mp3 pro second.
AAC/WMA not worth mentioning
Layla.wav: All the encoders show poorly here. mp3pro is very
swishy and murders the applause at the beginning, Ogg makes the
ride cymbal very harsh. Which wins depends on how many people
side with each as more tolerable. No one will be happy with
either. AAC and WMA, once again, might not have bothered showing
up.
LifeShatters: All ogg on this one. mp3pro second. WMA and AAC are
no-shows.
LisztBMinor.wav: Very oddly, all encoders have noticable
artifacts. Ogg -b 64 --managed has best fidelity, but noticable
center hiss concentration. WMA goes metallic in the middle. Ogg
-q0 picks up a wheedle behind some of the piano hits. mp3pro
manages to flange all through the middle and even throws in
massive preecho. AAC has the same flanging in the center. I'll
bet it will be a near-tossup with Ogg penalized for hiss.
MidnightVoyage.wav: Again, no one is perfect. Ogg picks up a
midrange rasp. MP3 Pro flanges throughout and exhibits noise
pumping when the tape hiss is audible. Ogg will win this one by a
small but definite margin with WMA and AAC in the distant rear.
TheSource: The Ogg's vocal picks up a HF rasp/wheedle. The mp3pro
muffles the whole thing and flanges throughout. Hard to say which
will be chosen as worse. Close result, either Ogg or MP3 Pro win.
WMA and AAC.... you know the pattern by now.
Waiting.wav: Wow! Unexpectedly hard sample! I'll bet this one
causes Ogg's tonality compander to classify the voice midrange as
'noise' because of the guitar in the same spectral range but left
channel. Ogg picks up tons of low midrange noise behind the
vocal. The mp3pro sounds like crap too. Flanging, weak/glassy
vocal. This is the classic mp3 problem, and I doubt it will be
judged as harsly as Ogg (probably rightly so). The WMA is
painful. The AAC is bad in the same ways as the mp3pro, but
worse, lowpassed, and with gravelly, unstable bass. Winner will
be mp3pro followed by Ogg, probably. The Ogg artifacts are nasty,
but not painful like the WMA. The AAC fails noticably on too many
fronts to place second.
thear1.wav: Ogg will edge out mp3pro; both start out strong,
mp3pro gets glassy through the sample. WMA and AAC are
embarrassing.
That's my bets. I wonder just how far off I'll be.... Ogg wins
the most samples, but the two it stumbles on, it will stumble
relatively badly (both due to missing stereo analysis machinery
and a probable error in midrange tuning). I wonder how far down
that brings the final scoring for Ogg.
Monty
Return to ff123's Home Page