Results of 128kbps Extension Public Listening Test

Notes by Roberto Amorim

with lots of help from ff123

These are the summary results of the 128kbit/s Extension Public Listening test.

User comments are available here.

Let me try to be clear: there was no winner in this listening test. Musepack didn't win. All modern codecs are tied at first place, simple as that.

How to interpret the plots: Each plot is drawn with the five codecs on the x axis and the ratings given (1.0 through 5.0) on the y axis. N is the number of listeners used to compute the means (average ratings) and 95% confidence intervals. The mean rating given to each codec is indicted by the middle point of each vertical line segment, and the value is printed next to it. Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.
This analysis is different than the one used on ff123's 64kbps test. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the 41_30sec plot below, WMA is rated better than Lame with 95% confidence. And AAC is rated better than Lame with greater than 95% confidence.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences will vary somewhat. The best codec for a person is dependent on his own preferences and the type of music he prefers.

Another important note: It's worth mentioning that the WMA codec used in this test is WMA PRO (introduced in Windows Media 9). It has better quality, but is not backwards compatible with WMA standard, that is the codec that is actually supported by portable players, DVD players, etc. It's expected that devices supporting WMA Pro start being manufactured soon.

Plot	Comment
	Artist: Dave Matthews Band Title: #41 Album: Crash Submitted by: Filburt "Rock" Results: AAC, MPC and Vorbis are tied, WMA is a little lower than the others. Lame gets third place. Blade is the worst by far.
	Artist: Jazz-Preachers Title: Take the A-Train Album: Stax Space-Sound CD Submitted by: ff123 Brass band Results: AAC, MPC and WMA are tied, with WMA outstanding a little. Vorbis is a little behind. MP3 comes at third place. Blade is worst again.
	Composer: Johann Sebastian Bach English Suites Nr.5 in E minor - Part.2: Allemande. Performer: Blandine Rannou Submitted by: Guruboolez Solo Harpsichord Results: WMA and MPC tied at first place, with WMA outstanding well. AAC and Vorbis nearly perfectly tied at second. Lame at third place, well behind, and Blade at fourth, close to Lame.
	Artist: Opeth Title: Blackwater Park Album: Blackwater Park Quiet intro with acoustic and electric guitars followed by loud metal riffs. Results: MPC, AAC and WMA tied at first place. Vorbis at second place, Lame at third just behind. Blade is faaaar....
	Artist: Two Lone Swordsmen Title: Death To All Culture Snitches Album: Tiny Reminders Submitted by: Dibrom IDM Results: MPC wins. AAC and Vorbis tied at second. Lame is at third place. Interestingly, WMA fails badly on this sample (and only on this sample), only winning compared to Blade (5th)
	Artist: Man With No Name Title: Floor-essence Submitted by: Karl Heyes Electronic mix Results: Vorbis is tied to MPC at first place. WMA is at second and AAC at third. Lame at fourth, Blade last.
	Artist: Eric Clapton Title: Layla (unplugged) Album: The All Time Greatest Rock Songs Submitted by: Jan S. Applause at the beginning, followed by piano and acoustic guitar Results: WMA, MPC, and Vorbis are tied at first, with Vorbis a little worse than the others. AAC at second, Lame at third, Blade in last position.
	Title: Danse Macabre" Album: Submitted by: Wombat Orchestral Results: WMA wins. AAC, Vorbis and MPC are tied at second. Lame at third, Blade at fourth.
	Artist: Mamas and Papas Title: Midnight Voyage Submitted by: Jan S. drums and bass in the far left, guitar in the far right. Female vocal in the center. Results: AAC, WMA, MPC and Vorbis tied at first place, with Vorbis a little worse than the others. Lame at second place, Blade at third.
	Artst: Tiamat Album: A Deeper Kind Of Slumber Track: 01 Submitted by: treech rock/metal riffs Results: AAC wins, MPC and WMA tied at second. Lame and Vorbis tied at third, Blade is last.
	Artist: The Source feat. Candy Staton Track: You've Got the Love Album: ? Submitted by: Karl Heyes Drums and ride cymbals intro, followed by bass and female vocals. Results: WMA at first place AAC and MPC almost perfectly tied at second Lame and Vorbis at third, Blade way down there.
	Artist: Green Day Title: Waiting Album: Warning Intro consists of guitar in far left with male vocal in center. Results: MPC wins, AAC comes close at second place WMA and vorbis are tied at third place. Lame at fourth, Blade at fifth place.

This is the bitrate distribution table. AAC and Ogg bitrates were gathered using Winamp; Lame, Blade and MPC bitrates using EncSpot and WMApro using Windows Media Encoder.

 AAC Lame MPC Ogg WMApro Blade 41_30sec 129 125 166 143 128 128 ATrain 129 121 144 139 128 128 Bachpsichord 129 125 197 173 128 128 Blackwater 129 124 130 130 128 128 death2 129 133 115 114 128 128 FloorEssence 129 129 145 147 128 128 Layla 129 124 150 155 128 128 macabre 129 122 156 149 128 128 MidnightVoyage 129 121 142 141 128 128 thear1 129 123 132 128 128 128 TheSource 129 127 130 126 128 128 Waiting 129 121 147 137 128 128 ---------------------------------------------- Average 129 124.5 146.1 140.1 128 128

Overall Ratings: The results for each sample were grouped together, without modifications.

Then I performed an ANOVA analysis. The results are graphed below. There's a big tie at first place, with MPC only a little above AAC, WMA and Vorbis.
Lame is at second place and Blade is far behind at third.

Here's a zoomed version showing only the most significative part of the graph, leaving out the anchor:

Back to Roberto's Listening Tests page