English 日本語

Results of the public multiformat listening test (July 2014)


These are the summary results of the multiformat listening test. The purpose was to test Opus, AAC and Ogg Vorbis at 96 kbps against a classic MP3 128 kbps, and find out which codec produces the best sound quality.

Summary

♛1 Opus 1.1 with opus-tools-0.1.9-win32
opusenc --bitrate 96 in.wav out.opus
♛2 AAC iTunes 11.2.2 via qaac 2.41
qaac --cvbr 96 -o out.mp4 in.wav
♛3 Ogg Vorbis aoTuV Beta6.03
venc603 -q2.2 in.wav out.ogg
♛3 MP3 LAME 3.99.5
lame -V5 in.wav out.mp3 *used 29% more bitrate.

The sound quality of Opus, AAC, Ogg Vorbis and MP3 was evaluated by the public blind listening test. As a standard procedure of the ABC/HR methodology, the lossy audio files were paired with lossless original before the test. Thirty eight donators from public rated their perceived fidelity of those lossy files. Thirty five diverse music tracks and five speech tracks were used as test tracks.

The Opus encoder 1.1, the AAC encoder of iTunes, and the aoTuV at 96 kbps and the LAME at 128 kbps were tested. The most common response was on or between "Imperceptible" and "Perceptible, but not annoying".

The Opus codec was the clear winner, and the AAC encoded by Apple iTunes won the second place. The Ogg Vorbis was the third. The MP3, which was allowed to use 30kbps more bitrate than those contenders, was tied to Ogg Vorbis in joint third place.

FAAC (libfaac), which was known to be inferior before the test, was used as low quality anchors.

Contents

How to interpret the charts

Each chart is drawn with 6 codecs on the X axis and the rating given (1.0 to 5.0) on the Y axis. The mean rating given to each codec is indicated by the middle point of each vertical I-shaped line segment. Each I-shaped segment represents the 95% confidence interval of the mean rating (using bootstrap analysis) for each codec.

Ratings of all tracks (40 tracks)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible 95% Confidence Interval 0.1.9-win32 iTunes 11, qaac 2.41 aoTuV b6.03 LAME 3.99.5 faac-1.28-mod from rarewares --bitrate 96 --cvbr 96 -q2.2 -V5 -b 96 -q 30 107kbps 104kbps 106kbps 136kbps 98kbps 52kbps Opus AAC Ogg MP3 FAAC 96k FAAC q30
Ratings 3.2 3.4 3.6 3.8 4.0 4.2 4.4 4.6 4.8 5.0 Opus AAC Ogg MP3

All sets of tracks (5 sets, each 8 tracks)

trackgenreOpus
1.1
AAC
Apple
Ogg
Vorbis
MP3
LAME
FAAC
96k
FAAC
q30
Average4.654.404.244.242.651.19
trackgenreOpus
1.1
AAC
Apple
Ogg
Vorbis
MP3
LAME
FAAC
96k
FAAC
q30
Chris's set of tracks
SinceAlwaysGuitar4.964.864.664.422.631.33
WaitingPops4.584.494.034.292.611.07
velvetPerc.4.354.384.374.372.461.13
trumpetWind4.844.274.243.912.441.04
girlTechno4.493.243.954.082.581.24
Can't Wait Until Tonight (Dry Wurlitzer Mix)Pops4.433.773.384.582.571.06
35_SQAM_glockenspiel_cutInst.4.704.564.434.493.071.24
Robots_oldTechno4.914.534.673.612.631.06
Tracks randomly picked from album libraries
Asleep__4.11-4.31_Strings4.894.914.794.892.951.06
Greatest_Love_of_All_2min57Pops4.674.374.324.272.971.10
Hey TonightRock4.404.574.134.062.811.20
Severance__1.31-1.51_Pops4.653.954.004.282.721.37
Shinsho_pool_3min45_4min4Pops4.814.544.293.882.921.40
SlavesOfFearMetal4.914.314.854.742.851.23
The Chastising of RenegadeRock4.764.434.714.272.881.18
TrosYGaregPops4.494.254.153.962.491.24
Mpeg Sequences
4-Sound-English-male.441Voice4.974.823.774.473.011.22
9-Have-big-expensive-car.441Pops4.714.044.284.172.591.12
12-German-male-speech.441Voice4.824.803.594.092.711.19
15-Good-evening.441Voice4.314.514.244.192.491.19
21-classic.441Classic4.694.573.963.942.891.43
24-Greensleeves-Korean-male-speech.441Voice5.004.324.234.402.921.00
25-This-is-the-end.441Voice4.894.414.204.562.701.23
27-last-song-drums-and-trampets.441Perc.4.574.343.974.062.771.00
Tracks picked from previous 2011 AAC test
bonhemian_rhapsodyRock4.634.534.043.712.491.26
clapton_44kGuitar4.294.214.464.302.411.04
CoralPops4.944.504.534.622.571.29
ExitMusicVocal4.934.434.133.802.331.00
liberateMetal4.244.434.614.502.491.49
NewYorkCityPops4.804.074.134.482.381.00
sandmanRock4.934.334.404.732.661.36
take_your_finger_from_my_hairRock4.044.663.704.132.531.21
Tracks picked from new HA thread 103989
ChangesRock4.364.214.434.252.541.19
Girl_In_The_Fire__Sample_Rock4.364.504.134.032.941.10
Hotel CaliforniaRock4.814.594.134.462.731.23
Jupiter, the Bringer of JollityClassic4.484.423.804.102.731.30
Last_Of_The_Mohicanz__Sample_Electro4.684.214.674.322.271.18
Only TimeNewage4.514.694.204.192.531.11
Through The Fire And FlamesMetal4.654.484.683.912.381.51
With Love (Outro)Techno4.654.504.183.992.491.18

Ratings of sets of tracks

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible 0.1.9-win32 iTunes 11, qaac 2.41 aoTuV b6.03 LAME 3.99.5 faac-1.28-mod from rarewares --bitrate 96 --cvbr 96 -q2.2 -V5 -b 96 -q 30 107kbps 104kbps 106kbps 136kbps 98kbps 52kbps C C C C C C T T T T T T M M M M M M R R R R R R A A A A A A Opus AAC Ogg MP3 FAAC 96k FAAC q30
Set nameOpus
1.1
AAC
Apple
Ogg
Vorbis
MP3
LAME
FAAC
96k
FAAC
q30
Set nameOpus
1.1
AAC
Apple
Ogg
Vorbis
MP3
LAME
FAAC
96k
FAAC
q30
Chris's set of tracks4.664.264.224.222.621.15
Tracks randomly picked from album libraries4.704.424.414.292.821.22
Mpeg Sequences4.754.484.034.242.761.17
Tracks picked from previous 2011 AAC test4.604.394.254.282.481.21
Tracks picked from new HA thread 1039894.564.454.284.162.581.23

Ratings of each track

1 1 2 2 3 3 4 4 5 5 Chris's set of tracks SinceAlways Guitar Waiting Pops velvet Perc. trumpet Wind girl Techno Can't Wait Until Tonight (Dry Wurlitzer Mix) Pops 35_SQAM_glockenspiel_cut Inst. Robots_old Techno Tracks randomly picked from album libraries Asleep__4.11-4.31_ Strings Greatest_Love_of_All_2min57 Pops Hey Tonight Rock Severance__1.31-1.51_ Pops Shinsho_pool_3min45_4min4 Pops SlavesOfFear Metal The Chastising of Renegade Rock TrosYGareg Pops Mpeg Sequences 4-Sound-English-male.441 Voice 9-Have-big-expensive-car.441 Pops 12-German-male-speech.441 Voice 15-Good-evening.441 Voice 21-classic.441 Classic 24-Greensleeves-Korean-male-speech.441 Voice 25-This-is-the-end.441 Voice 27-last-song-drums-and-trampets.441 Perc. Tracks picked from previous 2011 AAC test bonhemian_rhapsody Rock clapton_44k Guitar Coral Pops ExitMusic Vocal liberate Metal NewYorkCity Pops sandman Rock take_your_finger_from_my_hair Rock Tracks picked from new HA thread 103989 Changes Rock Girl_In_The_Fire__Sample_ Rock Hotel California Rock Jupiter, the Bringer of Jollity Classic Last_Of_The_Mohicanz__Sample_ Electro Only Time Newage Through The Fire And Flames Metal With Love (Outro) Techno Opus AAC Ogg MP3 FAAC 96k FAAC q30

Chris's set of tracks

SinceAlways

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Waiting

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

velvet

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

trumpet

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

girl

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Can't Wait Until Tonight (Dry Wurlitzer Mix)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

35_SQAM_glockenspiel_cut

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Robots_old

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Tracks randomly picked from album libraries

Asleep__4.11-4.31_

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Greatest_Love_of_All_2min57

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Hey Tonight

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Severance__1.31-1.51_

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Shinsho_pool_3min45_4min4

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

SlavesOfFear

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

The Chastising of Renegade

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

TrosYGareg

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Mpeg Sequences

4-Sound-English-male.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

9-Have-big-expensive-car.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

12-German-male-speech.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

15-Good-evening.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

21-classic.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

24-Greensleeves-Korean-male-speech.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

25-This-is-the-end.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

27-last-song-drums-and-trampets.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Tracks picked from previous 2011 AAC test

bonhemian_rhapsody

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

clapton_44k

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Coral

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

ExitMusic

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

liberate

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

NewYorkCity

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

sandman

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

take_your_finger_from_my_hair

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Tracks picked from new HA thread 103989

Changes

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Girl_In_The_Fire__Sample_

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Hotel California

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Jupiter, the Bringer of Jollity

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Last_Of_The_Mohicanz__Sample_

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Only Time

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Through The Fire And Flames

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

With Love (Outro)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Donators, tracks and contributions (363 results, 339 valid results)

kiavan92 is the top contributor.

Ratings of each donator

1 1 2 2 3 3 4 4 5 5 kiavan92 (38) Alexxander (35) anonymous2 (35) win32vb (32) Kamedo2 (25) Steve Forte Rio (24) IgorC (21) Caspian (16) Gainless (13) Joungmin Lee (11) 2Bdecided (10) singlethread (10) Hex144 (8) anonymous9 (7) timuer (7) anonymous3 (5) anonymous5 (4) HS (4) naylor83 (4) Panther025 (4) anonymous1 (3) anonymous10 (3) psychoticmeow (3) TL (3) yourlord (3) anonymous7 (2) johnb (2) NeoteriX (2) anonymous11 (1) anonymous12 (1) anonymous4 (1) anonymous8 (1) DH (1) Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings of the top 12 donators

Ratings submitted by kiavan92 (38 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Alexxander (35 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by anonymous2 (35 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by win32vb (32 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Kamedo2 (25 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Steve Forte Rio (24 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by IgorC (21 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Caspian (16 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Gainless (13 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Joungmin Lee (11 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by 2Bdecided (10 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by singlethread (10 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Bitrates of tracks

In each codec, settings were calibrated to provide ~96kbps on a large variety of music on average, except for the MP3 and anchors.

Track Bitrate [kbps] 0 20 40 60 80 100 120 140 160 0.1.9-win32 iTunes 11, qaac 2.41 aoTuV b6.03 LAME 3.99.5 faac-1.28-mod from rarewares --bitrate 96 --cvbr 96 -q2.2 -V5 -b 96 -q 30 107kbps 104kbps 106kbps 136kbps 98kbps 52kbps Opus AAC Ogg MP3 FAAC 96k FAAC q30

These are the file size based bitrates.

Sample	Length	Opus	AAC	Ogg	MP3	FAAC 96	FAAC q30
--------------------------------------------------------------------------
1	18.191	82	103	97	144	98	58	SinceAlways
2	20.155	111	106	107	137	98	54	Waiting
3	11.879	112	114	124	169	98	55	velvet
4	10.031	133	101	101	156	98	60	trumpet
5	15.957	122	101	134	155	98	51	girl
6	29.933	116	99	111	144	97	54	Can't Wait Until Tonight (Dry Wurlitzer Mix)
7	23.45	131	85	79	77	83	41	35_SQAM_glockenspiel_cut
8	30	100	108	105	128	98	44	Robots_old
								
9	20	103	109	116	148	98	48	Asleep__4.11-4.31_
10	20.178	103	100	89	120	98	54	Greatest_Love_of_All_2min57
11	20	92	102	96	131	98	55	Hey Tonight
12	20	120	100	94	131	98	54	Severance__1.31-1.51_
13	18.924	104	109	88	108	98	48	Shinsho_pool_3min45_4min4
14	22	90	100	107	131	98	56	SlavesOfFear
15	20	104	109	122	146	98	49	The Chastising of Renegade
16	22.304	124	99	96	127	98	55	TrosYGareg
								
17	9.028	96	95	99	121	98	46	4-Sound-English-male.441
18	9.707	119	104	113	154	99	48	9-Have-big-expensive-car.441
19	7.84	100	95	104	116	98	47	12-German-male-speech.441
20	8.566	94	99	100	129	98	52	15-Good-evening.441
21	9.656	112	105	89	133	98	52	21-classic.441
22	8.72	133	101	121	176	99	45	24-Greensleeves-Korean-male-speech.441
23	9.184	98	99	95	125	98	53	25-This-is-the-end.441
24	9.966	104	105	112	128	99	48	27-last-song-drums-and-trampets.441
								
25	28.505	101	106	94	120	98	53	bonhemian_rhapsody
26	14.986	104	107	122	145	98	53	clapton_44k
27	12.701	104	107	102	135	98	48	Coral
28	20.106	111	101	85	141	98	47	ExitMusic
29	5.007	97	116	143	160	98	54	liberate
30	29.849	93	103	102	131	98	55	NewYorkCity
31	30.002	97	99	106	134	98	49	sandman
32	22.092	114	109	133	138	98	60	take_your_finger_frin_my_head
								
33	29.932	113	106	119	137	98	51	Changes
34	13.742	121	111	108	126	98	53	Girl_In_The_Fire__Sample_
35	30.762	106	110	112	143	98	52	Hotel California
36	29.234	105	103	89	123	98	56	Jupiter, the Bringer of Jollity
37	25.481	104	110	114	147	98	48	Last_Of_The_Mohicanz__Sample_
38	24.671	96	110	100	130	98	48	Only Time
39	29.789	97	99	117	135	98	59	Through The Fire And Flames
40	16.403	103	103	99	144	98	56	With Love (Outro)
--------------------------------------------------------------------------
Mean	18.973 	106.7	103.7 	106.1 	135.5 	97.6 	51.7
Unit	second	kbps	kbps	kbps	kbps	kbps	kbps
Sample	Length	Opus	AAC	Ogg	MP3	FAAC 96	FAAC q30

  

Bitrate vs ratings 2D scatter plot

Opus3.23.43.63.84.04.24.44.64.85.0648096112128144160176192bitrate [kbps]RatingsAAC3.23.43.63.84.04.24.44.64.85.0648096112128144160176192bitrate [kbps]Ogg3.23.43.63.84.04.24.44.64.85.0648096112128144160176192bitrate [kbps]RatingsMP33.23.43.63.84.04.24.44.64.85.0648096112128144160176192bitrate [kbps]

Listen and download tracks

The Links on the left side are the lossless original tracks, and the right side are the lossy tracks.

trackgenreOpus
1.1
AAC
Apple
Ogg
Vorbis
MP3
LAME
FAAC
96k
FAAC
q30
Average4.654.404.244.242.651.19
trackgenreOpus
1.1
AAC
Apple
Ogg
Vorbis
MP3
LAME
FAAC
96k
FAAC
q30
Chris's set of tracks
SinceAlwaysGuitar4.964.864.664.422.631.33
WaitingPops4.584.494.034.292.611.07
velvetPerc.4.354.384.374.372.461.13
trumpetWind4.844.274.243.912.441.04
girlTechno4.493.243.954.082.581.24
Can't Wait Until Tonight (Dry Wurlitzer Mix)Pops4.433.773.384.582.571.06
35_SQAM_glockenspiel_cutInst.4.704.564.434.493.071.24
Robots_oldTechno4.914.534.673.612.631.06
Tracks randomly picked from album libraries
Asleep__4.11-4.31_Strings4.894.914.794.892.951.06
Greatest_Love_of_All_2min57Pops4.674.374.324.272.971.10
Hey TonightRock4.404.574.134.062.811.20
Severance__1.31-1.51_Pops4.653.954.004.282.721.37
Shinsho_pool_3min45_4min4Pops4.814.544.293.882.921.40
SlavesOfFearMetal4.914.314.854.742.851.23
The Chastising of RenegadeRock4.764.434.714.272.881.18
TrosYGaregPops4.494.254.153.962.491.24
Mpeg Sequences
4-Sound-English-male.441Voice4.974.823.774.473.011.22
9-Have-big-expensive-car.441Pops4.714.044.284.172.591.12
12-German-male-speech.441Voice4.824.803.594.092.711.19
15-Good-evening.441Voice4.314.514.244.192.491.19
21-classic.441Classic4.694.573.963.942.891.43
24-Greensleeves-Korean-male-speech.441Voice5.004.324.234.402.921.00
25-This-is-the-end.441Voice4.894.414.204.562.701.23
27-last-song-drums-and-trampets.441Perc.4.574.343.974.062.771.00
Tracks picked from previous 2011 AAC test
bonhemian_rhapsodyRock4.634.534.043.712.491.26
clapton_44kGuitar4.294.214.464.302.411.04
CoralPops4.944.504.534.622.571.29
ExitMusicVocal4.934.434.133.802.331.00
liberateMetal4.244.434.614.502.491.49
NewYorkCityPops4.804.074.134.482.381.00
sandmanRock4.934.334.404.732.661.36
take_your_finger_from_my_hairRock4.044.663.704.132.531.21
Tracks picked from new HA thread 103989
ChangesRock4.364.214.434.252.541.19
Girl_In_The_Fire__Sample_Rock4.364.504.134.032.941.10
Hotel CaliforniaRock4.814.594.134.462.731.23
Jupiter, the Bringer of JollityClassic4.484.423.804.102.731.30
Last_Of_The_Mohicanz__Sample_Electro4.684.214.674.322.271.18
Only TimeNewage4.514.694.204.192.531.11
Through The Fire And FlamesMetal4.654.484.683.912.381.51
With Love (Outro)Techno4.654.504.183.992.491.18

Post screening

Invalid results were discarded according to the following criteria, which had been made public prior to the test:

The discarded results were less than 7% of all results.

Bootstrap analysis

bootstrap.py --compare-all -p 1000000 -s 1000000 aq23.txt
bootstrap.py v1.0 2011-02-03
Copyright (C) 2011 Gian-Carlo Pascutto
License Affero GPL version 3 or later <http://www.gnu.org/licenses/agpl.html>

Reading from: C:\Users\PCC\Downloads\friedman124\aq23.txt
Read 6 treatments, 40 samples => 15 comparisons
Means:
AppleAAC      Opus       Ogg     MP3V5   FAAC96k   FAACq30
   4.400     4.653     4.235     4.237     2.652     1.194

Unadjusted p-values:
          Opus      Ogg       MP3V5     FAAC96k   FAACq30
AppleAAC  0.000*    0.023*    0.016*    0.000*    0.000*
Opus      -         0.000*    0.000*    0.000*    0.000*
Ogg       -         -         0.981     0.000*    0.000*
MP3V5     -         -         -         0.000*    0.000*
FAAC96k   -         -         -         -         0.000*

Opus is better than AppleAAC (p=0.000)
Ogg is worse than AppleAAC (p=0.023)
Ogg is worse than Opus (p=0.000)
MP3V5 is worse than AppleAAC (p=0.016)
MP3V5 is worse than Opus (p=0.000)
FAAC96k is worse than AppleAAC (p=0.000)
FAAC96k is worse than Opus (p=0.000)
FAAC96k is worse than Ogg (p=0.000)
FAAC96k is worse than MP3V5 (p=0.000)
FAACq30 is worse than AppleAAC (p=0.000)
FAACq30 is worse than Opus (p=0.000)
FAACq30 is worse than Ogg (p=0.000)
FAACq30 is worse than MP3V5 (p=0.000)
FAACq30 is worse than FAAC96k (p=0.000)

p-values adjusted for multiple comparison:
          Opus      Ogg       MP3V5     FAAC96k   FAACq30
AppleAAC  0.000*    0.044*    0.043*    0.000*    0.000*
Opus      -         0.000*    0.000*    0.000*    0.000*
Ogg       -         -         0.981     0.000*    0.000*
MP3V5     -         -         -         0.000*    0.000*
FAAC96k   -         -         -         -         0.000*

Opus is better than AppleAAC (p=0.000)
Ogg is worse than AppleAAC (p=0.044)
Ogg is worse than Opus (p=0.000)
MP3V5 is worse than AppleAAC (p=0.043)
MP3V5 is worse than Opus (p=0.000)
FAAC96k is worse than AppleAAC (p=0.000)
FAAC96k is worse than Opus (p=0.000)
FAAC96k is worse than Ogg (p=0.000)
FAAC96k is worse than MP3V5 (p=0.000)
FAACq30 is worse than AppleAAC (p=0.000)
FAACq30 is worse than Opus (p=0.000)
FAACq30 is worse than Ogg (p=0.000)
FAACq30 is worse than MP3V5 (p=0.000)
FAACq30 is worse than FAAC96k (p=0.000)

  

ANOVA analysis

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 40
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              239         385.20
Testers (blocks)    39           4.52
Codecs eval'd        5         369.44   73.89   1282.39  0.00E+000
Error              195          11.24    0.06
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.106

Means:

Opus     AppleAAC MP3V5    Ogg      FAAC96k  FAACq30
  4.65     4.40     4.24     4.24     2.65     1.19

---------------------------- p-value Matrix ---------------------------

         AppleAAC MP3V5    Ogg      FAAC96k  FAACq30
Opus     0.000*   0.000*   0.000*   0.000*   0.000*
AppleAAC          0.003*   0.002*   0.000*   0.000*
MP3V5                      0.976    0.000*   0.000*
Ogg                                 0.000*   0.000*
FAAC96k                                      0.000*
-----------------------------------------------------------------------

Opus is better than AppleAAC, MP3V5, Ogg, FAAC96k, FAACq30
AppleAAC is better than MP3V5, Ogg, FAAC96k, FAACq30
MP3V5 is better than FAAC96k, FAACq30
Ogg is better than FAAC96k, FAACq30
FAAC96k is better than FAACq30

  

Formatted ratings

The ratings parsed for ANOVA analysis, Bootstrap analysis and graphmaker 5.

Opus	AAC	Ogg	MP3	FAAC96k	FAACq30
%feature	10	0.1.9-win32	iTunes 11, qaac 2.41	aoTuV b6.03	LAME 3.99.5	faac-1.28-mod from rarewares	faac-1.28-mod from rarewares
%feature	11	--bitrate 96	--cvbr 96	-q2.2	-V5	-b 96	-q 30
%feature	12	107kbps	104kbps	106kbps	136kbps	98kbps	52kbps
%genre Chris's set of tracks
4.960 	4.860 	4.660 	4.420 	2.630 	1.330 
4.580 	4.490 	4.030 	4.290 	2.610 	1.070 
4.345 	4.382 	4.373 	4.373 	2.464 	1.127 
4.840 	4.270 	4.240 	3.910 	2.440 	1.040 
4.492 	3.238 	3.954 	4.077 	2.577 	1.238 
4.433 	3.767 	3.378 	4.578 	2.567 	1.056 
4.700 	4.558 	4.425 	4.492 	3.067 	1.242 
4.911 	4.533 	4.667 	3.611 	2.633 	1.056 
%genre Tracks randomly picked from album libraries
4.885 	4.908 	4.785 	4.885 	2.946 	1.062 
4.667 	4.367 	4.322 	4.267 	2.967 	1.100 
4.400 	4.567 	4.133 	4.056 	2.811 	1.200 
4.650 	3.950 	4.000 	4.283 	2.717 	1.367 
4.811 	4.544 	4.289 	3.878 	2.922 	1.400 
4.913 	4.313 	4.850 	4.738 	2.850 	1.225 
4.760 	4.430 	4.710 	4.270 	2.880 	1.180 
4.488 	4.250 	4.150 	3.963 	2.488 	1.238 
%genre Mpeg Sequences
4.970 	4.820 	3.770 	4.470 	3.010 	1.220 
4.711 	4.044 	4.278 	4.167 	2.589 	1.122 
4.820 	4.800 	3.590 	4.090 	2.710 	1.190 
4.314 	4.514 	4.243 	4.186 	2.486 	1.186 
4.686 	4.571 	3.957 	3.943 	2.886 	1.429 
5.000 	4.317 	4.233 	4.400 	2.917 	1.000 
4.888 	4.413 	4.200 	4.563 	2.700 	1.225 
4.571 	4.343 	3.971 	4.057 	2.771 	1.000 
%genre Tracks picked from previous 2011 AAC test
4.629 	4.529 	4.043 	3.714 	2.486 	1.257 
4.286 	4.214 	4.457 	4.300 	2.414 	1.043 
4.940 	4.500 	4.530 	4.620 	2.570 	1.290 
4.933 	4.433 	4.133 	3.800 	2.333 	1.000 
4.243 	4.429 	4.614 	4.500 	2.486 	1.486 
4.800 	4.067 	4.133 	4.483 	2.383 	1.000 
4.925 	4.325 	4.400 	4.725 	2.663 	1.363 
4.043 	4.657 	3.700 	4.129 	2.529 	1.214 
%genre Tracks picked from new HA thread 103989
4.363 	4.213 	4.425 	4.250 	2.538 	1.188 
4.363 	4.500 	4.125 	4.025 	2.938 	1.100 
4.814 	4.586 	4.129 	4.457 	2.729 	1.229 
4.483 	4.417 	3.800 	4.100 	2.733 	1.300 
4.678 	4.211 	4.667 	4.322 	2.267 	1.178 
4.514 	4.686 	4.200 	4.186 	2.529 	1.114 
4.650 	4.475 	4.675 	3.913 	2.375 	1.513 
4.650 	4.500 	4.175 	3.988 	2.488 	1.175 
%samples	SinceAlways	Guitar
%samples	Waiting	Pops
%samples	velvet	Perc.
%samples	trumpet	Wind
%samples	girl	Techno
%samples	Can't Wait Until Tonight (Dry Wurlitzer Mix)	Pops
%samples	35_SQAM_glockenspiel_cut	Inst.
%samples	Robots_old	Techno

%samples	Asleep__4.11-4.31_	Strings
%samples	Greatest_Love_of_All_2min57	Pops
%samples	Hey Tonight	Rock
%samples	Severance__1.31-1.51_	Pops
%samples	Shinsho_pool_3min45_4min4	Pops
%samples	SlavesOfFear	Metal
%samples	The Chastising of Renegade	Rock
%samples	TrosYGareg	Pops

%samples	4-Sound-English-male.441	Voice
%samples	9-Have-big-expensive-car.441	Pops
%samples	12-German-male-speech.441	Voice
%samples	15-Good-evening.441	Voice
%samples	21-classic.441	Classic
%samples	24-Greensleeves-Korean-male-speech.441	Voice
%samples	25-This-is-the-end.441	Voice
%samples	27-last-song-drums-and-trampets.441	Perc.

%samples	bonhemian_rhapsody	Rock
%samples	clapton_44k	Guitar
%samples	Coral	Pops
%samples	ExitMusic	Vocal
%samples	liberate	Metal
%samples	NewYorkCity	Pops
%samples	sandman	Rock
%samples	take_your_finger_from_my_hair	Rock

%samples	Changes	Rock
%samples	Girl_In_The_Fire__Sample_	Rock
%samples	Hotel California	Rock
%samples	Jupiter, the Bringer of Jollity	Classic
%samples	Last_Of_The_Mohicanz__Sample_	Electro
%samples	Only Time	Newage
%samples	Through The Fire And Flames	Metal
%samples	With Love (Outro)	Techno
%link_path tracks/ .wav .opus1.1.b96k.opus .qaac.cvbr96k.mp4 .aotuv.q2.2.ogg .lame3.99.5.v5.mp3 .faac.abr96k.mp4 .faac.vbr30q.mp4

  

Download logs

You can download a ZIP file containing all logs. Encryption keys and parsed text included.

Contact

Kamedo2: Twitter@kamedo2