Results of the public multiformat listening test (July 2014)

These are the summary results of the multiformat listening test.

You can download a ZIP file containing all results for all samples.

How to interpret the charts: Each chart is drawn with 6 codecs on the X axis and the rating given (1.0 to 5.0) on the Y axis. The mean rating given to each codec is indicated by the middle point of each vertical I-shaped line segment. Each I-shaped segment represents the 95% confidence interval of the mean rating (using bootstrap analysis) for each codec. This analysis is almost identical to the one used in previous listening tests.

Important note: These plots represent group preferences (for the particular group of people who participated in the test). Individual preferences vary somewhat. The best codec for a person is dependent on his/her own preferences and the type of sound he/she encodes.

Plot of the complete result (40 tracks, 339 results):

Closeup of the important section (40 tracks, 339 results):

Donators and contributions (38 donators, 33 valid donators, 363 results, 339 valid results):

kiavan92 is the top contributor.

Ratings by donators (top 12 donators with 10 or more results, 270 results):

Ratings submitted by kiavan92 (38 results)

Ratings submitted by Alexxander (35 results)

Ratings submitted by anonymous2 (35 results)

Ratings submitted by win32vb (32 results)

Ratings submitted by Kamedo2 (25 results)

Ratings submitted by Steve Forte Rio (24 results)

Ratings submitted by IgorC (21 results)

Ratings submitted by Caspian (16 results)

Ratings submitted by Gainless (13 results)

Ratings submitted by Joungmin Lee (11 results)

Ratings submitted by 2Bdecided (10 results)

Ratings submitted by singlethread (10 results)

Other than these top donators, we received important results from Hex144, timuer, Panther025, naylor83, HS, TL, psychoticmeow, yourlord, NeoteriX, johnb, DH, LedHed8, and 14 anonymous donators.

List of all valid donator's contributions

Ratings by set of tracks:

Per track results

A page with graphs and comments for each track individually is available.

Bitrate table

The codecs and settings were calibrated to provide ~96kbps on a large variety of music, except for the MP3 and anchors.

Codec bitrates scatter plot

These are the bitrates used by the codecs for the tracks in the test:

Sample	Length	AAC	Opus	Ogg	MP3 V5	FAAC 96	FAAC q30
--------------------------------------------------------------------------
1	18.191 	103	82	97	144	98	58	SinceAlways
2	20.155 	106	111	107	137	98	54	Waiting
3	11.879 	114	112	124	169	98	55	velvet
4	10.031 	101	133	101	156	98	60	trumpet
5	15.957 	101	122	134	155	98	51	girl
6	29.933 	99	116	111	144	97	54	Can't Wait Until Tonight (Dry Wurlitzer Mix)
7	23.450 	85	131	79	77	83	41	35_SQAM_glockenspiel_cut
8	30.000 	108	100	105	128	98	44	Robots_old
									
9	20.000 	109	103	116	148	98	48	Asleep__4.11-4.31_
10	20.178 	100	103	89	120	98	54	Greatest_Love_of_All_2min57
11	20.000 	102	92	96	131	98	55	Hey Tonight
12	20.000 	100	120	94	131	98	54	Severance__1.31-1.51_
13	18.924 	109	104	88	108	98	48	Shinsho_pool_3min45_4min4
14	22.000 	100	90	107	131	98	56	SlavesOfFear
15	20.000 	109	104	122	146	98	49	The Chastising of Renegade
16	22.304 	99	124	96	127	98	55	TrosYGareg
									
17	9.028 	95	96	99	121	98	46	4-Sound-English-male.441
18	9.707 	104	119	113	154	99	48	9-Have-big-expensive-car.441
19	7.840 	95	100	104	116	98	47	12-German-male-speech.441
20	8.566 	99	94	100	129	98	52	15-Good-evening.441
21	9.656 	105	112	89	133	98	52	21-classic.441
22	8.720 	101	133	121	176	99	45	24-Greensleeves-Korean-male-speech.441
23	9.184 	99	98	95	125	98	53	25-This-is-the-end.441
24	9.966 	105	104	112	128	99	48	27-last-song-drums-and-trampets.441
									
25	28.505 	106	101	94	120	98	53	bonhemian_rhapsody
26	14.986 	107	104	122	145	98	53	clapton_44k
27	12.701 	107	104	102	135	98	48	Coral
28	20.106 	101	111	85	141	98	47	ExitMusic
29	5.007 	116	97	143	160	98	54	liberate
30	29.849 	103	93	102	131	98	55	NewYorkCity
31	30.002 	99	97	106	134	98	49	sandman
32	22.092 	109	114	133	138	98	60	take_your_finger_frin_my_head
									
33	29.932 	106	113	119	137	98	51	Changes
34	13.742 	111	121	108	126	98	53	Girl_In_The_Fire__Sample_
35	30.762 	110	106	112	143	98	52	Hotel California
36	29.234 	103	105	89	123	98	56	Jupiter, the Bringer of Jolity
37	25.481 	110	104	114	147	98	48	Last_Of_The_Mohicanz__Sample_
38	24.671 	110	96	100	130	98	48	Only Time
39	29.789 	99	97	117	135	98	59	Through The Fire And Flames
40	16.403 	103	103	99	144	98	56	With Love (Outro)
--------------------------------------------------------------------------
Mean	18.973	103.7 	106.7 	106.1 	135.5 	97.6 	51.7
Unit	second	kbps	kbps	kbps	kbps	kbps	kbps
Sample	Length	AAC	Opus	Ogg	MP3 V5	FAAC 96	FAAC q30

Bitrate vs ratings 2D scatter plot

Bootstrap analysis:

bootstrap.py --compare-all -p 1000000 -s 1000000 aq23.txt
bootstrap.py v1.0 2011-02-03
Copyright (C) 2011 Gian-Carlo Pascutto
License Affero GPL version 3 or later <http://www.gnu.org/licenses/agpl.html>

Reading from: C:\Users\PCC\Downloads\friedman124\aq23.txt
Read 6 treatments, 40 samples => 15 comparisons
Means:
AppleAAC      Opus       Ogg     MP3V5   FAAC96k   FAACq30
   4.400     4.653     4.235     4.237     2.652     1.194

Unadjusted p-values:
          Opus      Ogg       MP3V5     FAAC96k   FAACq30
AppleAAC  0.000*    0.023*    0.016*    0.000*    0.000*
Opus      -         0.000*    0.000*    0.000*    0.000*
Ogg       -         -         0.981     0.000*    0.000*
MP3V5     -         -         -         0.000*    0.000*
FAAC96k   -         -         -         -         0.000*

Opus is better than AppleAAC (p=0.000)
Ogg is worse than AppleAAC (p=0.023)
Ogg is worse than Opus (p=0.000)
MP3V5 is worse than AppleAAC (p=0.016)
MP3V5 is worse than Opus (p=0.000)
FAAC96k is worse than AppleAAC (p=0.000)
FAAC96k is worse than Opus (p=0.000)
FAAC96k is worse than Ogg (p=0.000)
FAAC96k is worse than MP3V5 (p=0.000)
FAACq30 is worse than AppleAAC (p=0.000)
FAACq30 is worse than Opus (p=0.000)
FAACq30 is worse than Ogg (p=0.000)
FAACq30 is worse than MP3V5 (p=0.000)
FAACq30 is worse than FAAC96k (p=0.000)

p-values adjusted for multiple comparison:
          Opus      Ogg       MP3V5     FAAC96k   FAACq30
AppleAAC  0.000*    0.044*    0.043*    0.000*    0.000*
Opus      -         0.000*    0.000*    0.000*    0.000*
Ogg       -         -         0.981     0.000*    0.000*
MP3V5     -         -         -         0.000*    0.000*
FAAC96k   -         -         -         -         0.000*

Opus is better than AppleAAC (p=0.000)
Ogg is worse than AppleAAC (p=0.044)
Ogg is worse than Opus (p=0.000)
MP3V5 is worse than AppleAAC (p=0.043)
MP3V5 is worse than Opus (p=0.000)
FAAC96k is worse than AppleAAC (p=0.000)
FAAC96k is worse than Opus (p=0.000)
FAAC96k is worse than Ogg (p=0.000)
FAAC96k is worse than MP3V5 (p=0.000)
FAACq30 is worse than AppleAAC (p=0.000)
FAACq30 is worse than Opus (p=0.000)
FAACq30 is worse than Ogg (p=0.000)
FAACq30 is worse than MP3V5 (p=0.000)
FAACq30 is worse than FAAC96k (p=0.000)

ANOVA analysis:

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 40
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              239         385.20
Testers (blocks)    39           4.52
Codecs eval'd        5         369.44   73.89   1282.39  0.00E+000
Error              195          11.24    0.06
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.106

Means:

Opus     AppleAAC MP3V5    Ogg      FAAC96k  FAACq30
  4.65     4.40     4.24     4.24     2.65     1.19

---------------------------- p-value Matrix ---------------------------

         AppleAAC MP3V5    Ogg      FAAC96k  FAACq30
Opus     0.000*   0.000*   0.000*   0.000*   0.000*
AppleAAC          0.003*   0.002*   0.000*   0.000*
MP3V5                      0.976    0.000*   0.000*
Ogg                                 0.000*   0.000*
FAAC96k                                      0.000*
-----------------------------------------------------------------------

Opus is better than AppleAAC, MP3V5, Ogg, FAAC96k, FAACq30
AppleAAC is better than MP3V5, Ogg, FAAC96k, FAACq30
MP3V5 is better than FAAC96k, FAACq30
Ogg is better than FAAC96k, FAACq30
FAAC96k is better than FAACq30

Formatted ratings

For ANOVA analysis, Bootstrap analysis and graphmaker 4.

AppleAAC Opus Ogg MP3V5 FAAC96k FAACq30
%genre Chris's set of samples
4.860 4.960 4.660 4.420 2.630 1.330 
4.490 4.580 4.030 4.290 2.610 1.070 
4.382 4.345 4.373 4.373 2.464 1.127 
4.270 4.840 4.240 3.910 2.440 1.040 
3.238 4.492 3.954 4.077 2.577 1.238 
3.767 4.433 3.378 4.578 2.567 1.056 
4.558 4.700 4.425 4.492 3.067 1.242 
4.533 4.911 4.667 3.611 2.633 1.056 
%genre Randomly picked from album libraries
4.908 4.885 4.785 4.885 2.946 1.062 
4.367 4.667 4.322 4.267 2.967 1.100 
4.567 4.400 4.133 4.056 2.811 1.200 
3.950 4.650 4.000 4.283 2.717 1.367 
4.544 4.811 4.289 3.878 2.922 1.400 
4.313 4.913 4.850 4.738 2.850 1.225 
4.430 4.760 4.710 4.270 2.880 1.180 
4.250 4.488 4.150 3.963 2.488 1.238 
%genre Mpeg Sequences
4.820 4.970 3.770 4.470 3.010 1.220 
4.044 4.711 4.278 4.167 2.589 1.122 
4.800 4.820 3.590 4.090 2.710 1.190 
4.514 4.314 4.243 4.186 2.486 1.186 
4.571 4.686 3.957 3.943 2.886 1.429 
4.317 5.000 4.233 4.400 2.917 1.000 
4.413 4.888 4.200 4.563 2.700 1.225 
4.343 4.571 3.971 4.057 2.771 1.000 
%genre Picked from previous 2011 AAC test
4.529 4.629 4.043 3.714 2.486 1.257 
4.214 4.286 4.457 4.300 2.414 1.043 
4.500 4.940 4.530 4.620 2.570 1.290 
4.433 4.933 4.133 3.800 2.333 1.000 
4.429 4.243 4.614 4.500 2.486 1.486 
4.067 4.800 4.133 4.483 2.383 1.000 
4.325 4.925 4.400 4.725 2.663 1.363 
4.657 4.043 3.700 4.129 2.529 1.214 
%genre Picked from new HA thread 103989
4.213 4.363 4.425 4.250 2.538 1.188 
4.500 4.363 4.125 4.025 2.938 1.100 
4.586 4.814 4.129 4.457 2.729 1.229 
4.417 4.483 3.800 4.100 2.733 1.300 
4.211 4.678 4.667 4.322 2.267 1.178 
4.686 4.514 4.200 4.186 2.529 1.114 
4.475 4.650 4.675 3.913 2.375 1.513 
4.500 4.650 4.175 3.988 2.488 1.175
%samples	SinceAlways
%samples	Waiting
%samples	velvet
%samples	trumpet
%samples	girl
%samples	Can't Wait Until Tonight (Dry Wurlitzer Mix)
%samples	35_SQAM_glockenspiel_cut
%samples	Robots_old
%samples	Asleep__4.11-4.31_
%samples	Greatest_Love_of_All_2min57
%samples	Hey Tonight
%samples	Severance__1.31-1.51_
%samples	Shinsho_pool_3min45_4min4
%samples	SlavesOfFear
%samples	The Chastising of Renegade
%samples	TrosYGareg
%samples	4-Sound-English-male.441
%samples	9-Have-big-expensive-car.441
%samples	12-German-male-speech.441
%samples	15-Good-evening.441
%samples	21-classic.441
%samples	24-Greensleeves-Korean-male-speech.441
%samples	25-This-is-the-end.441
%samples	27-last-song-drums-and-trampets.441
%samples	bonhemian_rhapsody
%samples	clapton_44k
%samples	Coral
%samples	ExitMusic
%samples	liberate
%samples	NewYorkCity
%samples	sandman
%samples	take_your_finger_frin_my_head
%samples	Changes
%samples	Girl_In_The_Fire__Sample_
%samples	Hotel California
%samples	Jupiter, the Bringer of Jolity
%samples	Last_Of_The_Mohicanz__Sample_
%samples	Only Time
%samples	Through The Fire And Flames
%samples	With Love (Outro)
%link_path tracks/ .wav .qaac.cvbr96k.mp4 .opus1.1.b96k.opus .aotuv.q2.2.ogg .lame3.99.5.v5.mp3 .faac.abr96k.mp4 .faac.vbr30q.mp4

Post screening:

Invalid results were discarded according to the following criteria, which were made public at the beginning of the test:

If a listener ranked the low anchor at 5.0 on a sample, the listener's results for that sample were discarded.
If a listener ranked the mid-low anchor at 5.0 on a sample, the listener's results for that sample were discarded.
If a listener ranked the low anchor higher than the mid-low anchor on a sample, the listener's results for that sample were discarded.
If a listener ranked the reference worse than 4.5 on a sample, the listener's results for that sample were discarded.
If a listener ranked the reference worse than 5.0 on 25% or more of submitted results, all of that listener's results were discarded.
If a listener submitted 25% or more invalid results, all of the listener's results were discarded.

Track download:

Chris's set of samples
	track name	Apple AAC	Opus 1.1	Ogg Vorbis	MP3 V5	FAAC 96k	FAAC q30
1	SinceAlways	4.86	4.96	4.66	4.42	2.63	1.33
2	Waiting	4.49	4.58	4.03	4.29	2.61	1.07
3	velvet	4.38	4.35	4.37	4.37	2.46	1.13
4	trumpet	4.27	4.84	4.24	3.91	2.44	1.04
5	girl	3.24	4.49	3.95	4.08	2.58	1.24
6	Can't Wait Until Tonight (Dry Wurlitzer Mix)	3.77	4.43	3.38	4.58	2.57	1.06
7	35_SQAM_glockenspiel_cut	4.56	4.70	4.43	4.49	3.07	1.24
8	Robots_old	4.53	4.91	4.67	3.61	2.63	1.06
Randomly picked from album libraries
9	Asleep__4.11-4.31_	4.91	4.89	4.79	4.89	2.95	1.06
10	Greatest_Love_of_All_2min57	4.37	4.67	4.32	4.27	2.97	1.10
11	Hey Tonight	4.57	4.40	4.13	4.06	2.81	1.20
12	Severance__1.31-1.51_	3.95	4.65	4.00	4.28	2.72	1.37
13	Shinsho_pool_3min45_4min4	4.54	4.81	4.29	3.88	2.92	1.40
14	SlavesOfFear	4.31	4.91	4.85	4.74	2.85	1.23
15	The Chastising of Renegade	4.43	4.76	4.71	4.27	2.88	1.18
16	TrosYGareg	4.25	4.49	4.15	3.96	2.49	1.24
Mpeg Sequences
17	4-Sound-English-male.441	4.82	4.97	3.77	4.47	3.01	1.22
18	9-Have-big-expensive-car.441	4.04	4.71	4.28	4.17	2.59	1.12
19	12-German-male-speech.441	4.80	4.82	3.59	4.09	2.71	1.19
20	15-Good-evening.441	4.51	4.31	4.24	4.19	2.49	1.19
21	21-classic.441	4.57	4.69	3.96	3.94	2.89	1.43
22	24-Greensleeves-Korean-male-speech.441	4.32	5.00	4.23	4.40	2.92	1.00
23	25-This-is-the-end.441	4.41	4.89	4.20	4.56	2.70	1.23
24	27-last-song-drums-and-trampets.441	4.34	4.57	3.97	4.06	2.77	1.00
Picked from previous 2011 AAC test
25	bonhemian_rhapsody	4.53	4.63	4.04	3.71	2.49	1.26
26	clapton_44k	4.21	4.29	4.46	4.30	2.41	1.04
27	Coral	4.50	4.94	4.53	4.62	2.57	1.29
28	ExitMusic	4.43	4.93	4.13	3.80	2.33	1.00
29	liberate	4.43	4.24	4.61	4.50	2.49	1.49
30	NewYorkCity	4.07	4.80	4.13	4.48	2.38	1.00
31	sandman	4.33	4.93	4.40	4.73	2.66	1.36
32	take_your_finger_frin_my_head	4.66	4.04	3.70	4.13	2.53	1.21
Picked from new HA thread 103989
33	Changes	4.21	4.36	4.43	4.25	2.54	1.19
34	Girl_In_The_Fire__Sample_	4.50	4.36	4.13	4.03	2.94	1.10
35	Hotel California	4.59	4.81	4.13	4.46	2.73	1.23
36	Jupiter, the Bringer of Jolity	4.42	4.48	3.80	4.10	2.73	1.30
37	Last_Of_The_Mohicanz__Sample_	4.21	4.68	4.67	4.32	2.27	1.18
38	Only Time	4.69	4.51	4.20	4.19	2.53	1.11
39	Through The Fire And Flames	4.48	4.65	4.68	3.91	2.38	1.51
40	With Love (Outro)	4.50	4.65	4.18	3.99	2.49	1.18
	track name	Apple AAC	Opus 1.1	Ogg Vorbis	MP3 V5	FAAC 96k	FAAC q30

Contact

Kamedo2: Twitter@kamedo2