Results of the public multiformat listening test (July 2014)
These are the summary results of the multiformat listening test . The purpose was to test Opus, AAC and Ogg Vorbis at 96 kbps against a classic MP3 128 kbps, and find out which codec produces the best sound quality.
Summary
The sound quality of Opus, AAC, Ogg Vorbis and MP3 was evaluated by the public blind listening test. As a standard procedure of the ABC/HR methodology, the lossy audio files were paired with lossless original before the test. Thirty eight donators from public rated their perceived fidelity of those lossy files. Thirty five diverse music tracks and five speech tracks were used as test tracks.
The Opus encoder 1.1, the AAC encoder of iTunes, and the aoTuV at 96 kbps and the LAME at 128 kbps were tested. The most common response was on or between "Imperceptible" and "Perceptible, but not annoying".
The Opus codec was the clear winner, and the AAC encoded by Apple iTunes won the second place. The Ogg Vorbis was the third. The MP3, which was allowed to use 30kbps more bitrate than those contenders, was tied to Ogg Vorbis in joint third place.
FAAC (libfaac), which was known to be inferior before the test, was used as low quality anchors.
Contents
How to interpret the charts
Each chart is drawn with 6 codecs on the X axis and the rating given (1.0 to 5.0) on the Y axis.
The mean rating given to each codec is indicated by the middle point of each vertical I-shaped line segment.
Each I-shaped segment represents the 95% confidence interval of the mean rating (using bootstrap analysis) for each codec.
Ratings of all tracks (40 tracks)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
95%
Confidence
Interval
0.1.9-win32
iTunes 11, qaac 2.41
aoTuV b6.03
LAME 3.99.5
faac-1.28-mod from rarewares
--bitrate 96
--cvbr 96
-q2.2
-V5
-b 96
-q 30
107kbps
104kbps
106kbps
136kbps
98kbps
52kbps
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings
3.2
3.4
3.6
3.8
4.0
4.2
4.4
4.6
4.8
5.0
Opus
AAC
Ogg
MP3
All sets of tracks (5 sets, each 8 tracks)
Ratings of sets of tracks
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
0.1.9-win32
iTunes 11, qaac 2.41
aoTuV b6.03
LAME 3.99.5
faac-1.28-mod from rarewares
--bitrate 96
--cvbr 96
-q2.2
-V5
-b 96
-q 30
107kbps
104kbps
106kbps
136kbps
98kbps
52kbps
C
C
C
C
C
C
T
T
T
T
T
T
M
M
M
M
M
M
R
R
R
R
R
R
A
A
A
A
A
A
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Set name Opus 1.1 AAC Apple OggVorbis MP3 LAME FAAC 96k FAAC q30
Set name Opus 1.1 AAC Apple OggVorbis MP3 LAME FAAC 96k FAAC q30
Chris's set of tracks 4.66 4.26 4.22 4.22 2.62 1.15
Tracks randomly picked from album libraries 4.70 4.42 4.41 4.29 2.82 1.22
Mpeg Sequences 4.75 4.48 4.03 4.24 2.76 1.17
Tracks picked from previous 2011 AAC test 4.60 4.39 4.25 4.28 2.48 1.21
Tracks picked from new HA thread 103989 4.56 4.45 4.28 4.16 2.58 1.23
Ratings of each track
1
1
2
2
3
3
4
4
5
5
Chris's set of tracks
SinceAlways
Guitar
Waiting
Pops
velvet
Perc.
trumpet
Wind
girl
Techno
Can't Wait Until Tonight (Dry Wurlitzer Mix)
Pops
35_SQAM_glockenspiel_cut
Inst.
Robots_old
Techno
Tracks randomly picked from album libraries
Asleep__4.11-4.31_
Strings
Greatest_Love_of_All_2min57
Pops
Hey Tonight
Rock
Severance__1.31-1.51_
Pops
Shinsho_pool_3min45_4min4
Pops
SlavesOfFear
Metal
The Chastising of Renegade
Rock
TrosYGareg
Pops
Mpeg Sequences
4-Sound-English-male.441
Voice
9-Have-big-expensive-car.441
Pops
12-German-male-speech.441
Voice
15-Good-evening.441
Voice
21-classic.441
Classic
24-Greensleeves-Korean-male-speech.441
Voice
25-This-is-the-end.441
Voice
27-last-song-drums-and-trampets.441
Perc.
Tracks picked from previous 2011 AAC test
bonhemian_rhapsody
Rock
clapton_44k
Guitar
Coral
Pops
ExitMusic
Vocal
liberate
Metal
NewYorkCity
Pops
sandman
Rock
take_your_finger_from_my_hair
Rock
Tracks picked from new HA thread 103989
Changes
Rock
Girl_In_The_Fire__Sample_
Rock
Hotel California
Rock
Jupiter, the Bringer of Jollity
Classic
Last_Of_The_Mohicanz__Sample_
Electro
Only Time
Newage
Through The Fire And Flames
Metal
With Love (Outro)
Techno
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Chris's set of tracks
SinceAlways
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Waiting
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
velvet
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
trumpet
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
girl
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Can't Wait Until Tonight (Dry Wurlitzer Mix)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
35_SQAM_glockenspiel_cut
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Robots_old
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Tracks randomly picked from album libraries
Asleep__4.11-4.31_
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Greatest_Love_of_All_2min57
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Hey Tonight
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Severance__1.31-1.51_
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Shinsho_pool_3min45_4min4
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
SlavesOfFear
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
The Chastising of Renegade
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
TrosYGareg
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Mpeg Sequences
4-Sound-English-male.441
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
9-Have-big-expensive-car.441
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
12-German-male-speech.441
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
15-Good-evening.441
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
21-classic.441
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
24-Greensleeves-Korean-male-speech.441
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
25-This-is-the-end.441
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
27-last-song-drums-and-trampets.441
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Tracks picked from previous 2011 AAC test
bonhemian_rhapsody
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
clapton_44k
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Coral
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
ExitMusic
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
liberate
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
NewYorkCity
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
sandman
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
take_your_finger_from_my_hair
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Tracks picked from new HA thread 103989
Changes
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Girl_In_The_Fire__Sample_
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Hotel California
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Jupiter, the Bringer of Jollity
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Last_Of_The_Mohicanz__Sample_
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Only Time
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Through The Fire And Flames
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
With Love (Outro)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Donators, tracks and contributions (363 results, 339 valid results)
Ratings of each donator
1
1
2
2
3
3
4
4
5
5
kiavan92
(38)
Alexxander
(35)
anonymous2
(35)
win32vb
(32)
Kamedo2
(25)
Steve Forte Rio
(24)
IgorC
(21)
Caspian
(16)
Gainless
(13)
Joungmin Lee
(11)
2Bdecided
(10)
singlethread
(10)
Hex144
(8)
anonymous9
(7)
timuer
(7)
anonymous3
(5)
anonymous5
(4)
HS
(4)
naylor83
(4)
Panther025
(4)
anonymous1
(3)
anonymous10
(3)
psychoticmeow
(3)
TL
(3)
yourlord
(3)
anonymous7
(2)
johnb
(2)
NeoteriX
(2)
anonymous11
(1)
anonymous12
(1)
anonymous4
(1)
anonymous8
(1)
DH
(1)
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings of the top 12 donators
Ratings submitted by kiavan92 (38 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings submitted by Alexxander (35 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings submitted by anonymous2 (35 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings submitted by win32vb (32 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings submitted by Kamedo2 (25 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings submitted by Steve Forte Rio (24 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings submitted by IgorC (21 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings submitted by Caspian (16 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings submitted by Gainless (13 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings submitted by Joungmin Lee (11 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings submitted by 2Bdecided (10 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Ratings submitted by singlethread (10 results)
1
Very
annoying
2
Annoying
3
Slightly
annoying
4
Perceptible,
but not
annoying
5
Impercep-
tible
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
Bitrates of tracks
In each codec, settings were calibrated to provide ~96kbps on a large variety of music on average, except for the MP3 and anchors.
Track Bitrate [kbps]
0
20
40
60
80
100
120
140
160
0.1.9-win32
iTunes 11, qaac 2.41
aoTuV b6.03
LAME 3.99.5
faac-1.28-mod from rarewares
--bitrate 96
--cvbr 96
-q2.2
-V5
-b 96
-q 30
107kbps
104kbps
106kbps
136kbps
98kbps
52kbps
Opus
AAC
Ogg
MP3
FAAC 96k
FAAC q30
These are the file size based bitrates.
Sample Length Opus AAC Ogg MP3 FAAC 96 FAAC q30
--------------------------------------------------------------------------
1 18.191 82 103 97 144 98 58 SinceAlways
2 20.155 111 106 107 137 98 54 Waiting
3 11.879 112 114 124 169 98 55 velvet
4 10.031 133 101 101 156 98 60 trumpet
5 15.957 122 101 134 155 98 51 girl
6 29.933 116 99 111 144 97 54 Can't Wait Until Tonight (Dry Wurlitzer Mix)
7 23.45 131 85 79 77 83 41 35_SQAM_glockenspiel_cut
8 30 100 108 105 128 98 44 Robots_old
9 20 103 109 116 148 98 48 Asleep__4.11-4.31_
10 20.178 103 100 89 120 98 54 Greatest_Love_of_All_2min57
11 20 92 102 96 131 98 55 Hey Tonight
12 20 120 100 94 131 98 54 Severance__1.31-1.51_
13 18.924 104 109 88 108 98 48 Shinsho_pool_3min45_4min4
14 22 90 100 107 131 98 56 SlavesOfFear
15 20 104 109 122 146 98 49 The Chastising of Renegade
16 22.304 124 99 96 127 98 55 TrosYGareg
17 9.028 96 95 99 121 98 46 4-Sound-English-male.441
18 9.707 119 104 113 154 99 48 9-Have-big-expensive-car.441
19 7.84 100 95 104 116 98 47 12-German-male-speech.441
20 8.566 94 99 100 129 98 52 15-Good-evening.441
21 9.656 112 105 89 133 98 52 21-classic.441
22 8.72 133 101 121 176 99 45 24-Greensleeves-Korean-male-speech.441
23 9.184 98 99 95 125 98 53 25-This-is-the-end.441
24 9.966 104 105 112 128 99 48 27-last-song-drums-and-trampets.441
25 28.505 101 106 94 120 98 53 bonhemian_rhapsody
26 14.986 104 107 122 145 98 53 clapton_44k
27 12.701 104 107 102 135 98 48 Coral
28 20.106 111 101 85 141 98 47 ExitMusic
29 5.007 97 116 143 160 98 54 liberate
30 29.849 93 103 102 131 98 55 NewYorkCity
31 30.002 97 99 106 134 98 49 sandman
32 22.092 114 109 133 138 98 60 take_your_finger_frin_my_head
33 29.932 113 106 119 137 98 51 Changes
34 13.742 121 111 108 126 98 53 Girl_In_The_Fire__Sample_
35 30.762 106 110 112 143 98 52 Hotel California
36 29.234 105 103 89 123 98 56 Jupiter, the Bringer of Jollity
37 25.481 104 110 114 147 98 48 Last_Of_The_Mohicanz__Sample_
38 24.671 96 110 100 130 98 48 Only Time
39 29.789 97 99 117 135 98 59 Through The Fire And Flames
40 16.403 103 103 99 144 98 56 With Love (Outro)
--------------------------------------------------------------------------
Mean 18.973 106.7 103.7 106.1 135.5 97.6 51.7
Unit second kbps kbps kbps kbps kbps kbps
Sample Length Opus AAC Ogg MP3 FAAC 96 FAAC q30
Bitrate vs ratings 2D scatter plot
Opus 3.2 3.4 3.6 3.8 4.0 4.2 4.4 4.6 4.8 5.0 64 80 96 112 128 144 160 176 192 bitrate [kbps] Ratings AAC 3.2 3.4 3.6 3.8 4.0 4.2 4.4 4.6 4.8 5.0 64 80 96 112 128 144 160 176 192 bitrate [kbps] Ogg 3.2 3.4 3.6 3.8 4.0 4.2 4.4 4.6 4.8 5.0 64 80 96 112 128 144 160 176 192 bitrate [kbps] Ratings MP3 3.2 3.4 3.6 3.8 4.0 4.2 4.4 4.6 4.8 5.0 64 80 96 112 128 144 160 176 192 bitrate [kbps]
Listen and download tracks
The Links on the left side are the lossless original tracks, and the right side are the lossy tracks.
Post screening
Invalid results were discarded according to the following criteria, which had been made public prior to the test:
If a listener ranked the low anchor at 5.0 on a sample, the listener's results for that sample were discarded.
If a listener ranked the mid-low anchor at 5.0 on a sample, the listener's results for that sample were discarded.
If a listener ranked the low anchor higher than the mid-low anchor on a sample, the listener's results for that sample were discarded.
If a listener ranked the reference worse than 4.5 on a sample, the listener's results for that sample were discarded.
If a listener ranked the reference worse than 5.0 on 25% or more of submitted results, all of that listener's results were discarded.
If a listener submitted 25% or more invalid results, all of the listener's results were discarded.
The discarded results were less than 7% of all results.
Bootstrap analysis
bootstrap.py --compare-all -p 1000000 -s 1000000 aq23.txt
bootstrap.py v1.0 2011-02-03
Copyright (C) 2011 Gian-Carlo Pascutto
License Affero GPL version 3 or later <http://www.gnu.org/licenses/agpl.html>
Reading from: C:\Users\PCC\Downloads\friedman124\aq23.txt
Read 6 treatments, 40 samples => 15 comparisons
Means:
AppleAAC Opus Ogg MP3V5 FAAC96k FAACq30
4.400 4.653 4.235 4.237 2.652 1.194
Unadjusted p-values:
Opus Ogg MP3V5 FAAC96k FAACq30
AppleAAC 0.000* 0.023* 0.016* 0.000* 0.000*
Opus - 0.000* 0.000* 0.000* 0.000*
Ogg - - 0.981 0.000* 0.000*
MP3V5 - - - 0.000* 0.000*
FAAC96k - - - - 0.000*
Opus is better than AppleAAC (p=0.000)
Ogg is worse than AppleAAC (p=0.023)
Ogg is worse than Opus (p=0.000)
MP3V5 is worse than AppleAAC (p=0.016)
MP3V5 is worse than Opus (p=0.000)
FAAC96k is worse than AppleAAC (p=0.000)
FAAC96k is worse than Opus (p=0.000)
FAAC96k is worse than Ogg (p=0.000)
FAAC96k is worse than MP3V5 (p=0.000)
FAACq30 is worse than AppleAAC (p=0.000)
FAACq30 is worse than Opus (p=0.000)
FAACq30 is worse than Ogg (p=0.000)
FAACq30 is worse than MP3V5 (p=0.000)
FAACq30 is worse than FAAC96k (p=0.000)
p-values adjusted for multiple comparison:
Opus Ogg MP3V5 FAAC96k FAACq30
AppleAAC 0.000* 0.044* 0.043* 0.000* 0.000*
Opus - 0.000* 0.000* 0.000* 0.000*
Ogg - - 0.981 0.000* 0.000*
MP3V5 - - - 0.000* 0.000*
FAAC96k - - - - 0.000*
Opus is better than AppleAAC (p=0.000)
Ogg is worse than AppleAAC (p=0.044)
Ogg is worse than Opus (p=0.000)
MP3V5 is worse than AppleAAC (p=0.043)
MP3V5 is worse than Opus (p=0.000)
FAAC96k is worse than AppleAAC (p=0.000)
FAAC96k is worse than Opus (p=0.000)
FAAC96k is worse than Ogg (p=0.000)
FAAC96k is worse than MP3V5 (p=0.000)
FAACq30 is worse than AppleAAC (p=0.000)
FAACq30 is worse than Opus (p=0.000)
FAACq30 is worse than Ogg (p=0.000)
FAACq30 is worse than MP3V5 (p=0.000)
FAACq30 is worse than FAAC96k (p=0.000)
ANOVA analysis
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis
Number of listeners: 40
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings
Source of Degrees Sum of Mean
variation of Freedom squares Square F p
Total 239 385.20
Testers (blocks) 39 4.52
Codecs eval'd 5 369.44 73.89 1282.39 0.00E+000
Error 195 11.24 0.06
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.106
Means:
Opus AppleAAC MP3V5 Ogg FAAC96k FAACq30
4.65 4.40 4.24 4.24 2.65 1.19
---------------------------- p-value Matrix ---------------------------
AppleAAC MP3V5 Ogg FAAC96k FAACq30
Opus 0.000* 0.000* 0.000* 0.000* 0.000*
AppleAAC 0.003* 0.002* 0.000* 0.000*
MP3V5 0.976 0.000* 0.000*
Ogg 0.000* 0.000*
FAAC96k 0.000*
-----------------------------------------------------------------------
Opus is better than AppleAAC, MP3V5, Ogg, FAAC96k, FAACq30
AppleAAC is better than MP3V5, Ogg, FAAC96k, FAACq30
MP3V5 is better than FAAC96k, FAACq30
Ogg is better than FAAC96k, FAACq30
FAAC96k is better than FAACq30
Formatted ratings
The ratings parsed for ANOVA analysis, Bootstrap analysis and graphmaker 5 .
Opus AAC Ogg MP3 FAAC96k FAACq30
%feature 10 0.1.9-win32 iTunes 11, qaac 2.41 aoTuV b6.03 LAME 3.99.5 faac-1.28-mod from rarewares faac-1.28-mod from rarewares
%feature 11 --bitrate 96 --cvbr 96 -q2.2 -V5 -b 96 -q 30
%feature 12 107kbps 104kbps 106kbps 136kbps 98kbps 52kbps
%genre Chris's set of tracks
4.960 4.860 4.660 4.420 2.630 1.330
4.580 4.490 4.030 4.290 2.610 1.070
4.345 4.382 4.373 4.373 2.464 1.127
4.840 4.270 4.240 3.910 2.440 1.040
4.492 3.238 3.954 4.077 2.577 1.238
4.433 3.767 3.378 4.578 2.567 1.056
4.700 4.558 4.425 4.492 3.067 1.242
4.911 4.533 4.667 3.611 2.633 1.056
%genre Tracks randomly picked from album libraries
4.885 4.908 4.785 4.885 2.946 1.062
4.667 4.367 4.322 4.267 2.967 1.100
4.400 4.567 4.133 4.056 2.811 1.200
4.650 3.950 4.000 4.283 2.717 1.367
4.811 4.544 4.289 3.878 2.922 1.400
4.913 4.313 4.850 4.738 2.850 1.225
4.760 4.430 4.710 4.270 2.880 1.180
4.488 4.250 4.150 3.963 2.488 1.238
%genre Mpeg Sequences
4.970 4.820 3.770 4.470 3.010 1.220
4.711 4.044 4.278 4.167 2.589 1.122
4.820 4.800 3.590 4.090 2.710 1.190
4.314 4.514 4.243 4.186 2.486 1.186
4.686 4.571 3.957 3.943 2.886 1.429
5.000 4.317 4.233 4.400 2.917 1.000
4.888 4.413 4.200 4.563 2.700 1.225
4.571 4.343 3.971 4.057 2.771 1.000
%genre Tracks picked from previous 2011 AAC test
4.629 4.529 4.043 3.714 2.486 1.257
4.286 4.214 4.457 4.300 2.414 1.043
4.940 4.500 4.530 4.620 2.570 1.290
4.933 4.433 4.133 3.800 2.333 1.000
4.243 4.429 4.614 4.500 2.486 1.486
4.800 4.067 4.133 4.483 2.383 1.000
4.925 4.325 4.400 4.725 2.663 1.363
4.043 4.657 3.700 4.129 2.529 1.214
%genre Tracks picked from new HA thread 103989
4.363 4.213 4.425 4.250 2.538 1.188
4.363 4.500 4.125 4.025 2.938 1.100
4.814 4.586 4.129 4.457 2.729 1.229
4.483 4.417 3.800 4.100 2.733 1.300
4.678 4.211 4.667 4.322 2.267 1.178
4.514 4.686 4.200 4.186 2.529 1.114
4.650 4.475 4.675 3.913 2.375 1.513
4.650 4.500 4.175 3.988 2.488 1.175
%samples SinceAlways Guitar
%samples Waiting Pops
%samples velvet Perc.
%samples trumpet Wind
%samples girl Techno
%samples Can't Wait Until Tonight (Dry Wurlitzer Mix) Pops
%samples 35_SQAM_glockenspiel_cut Inst.
%samples Robots_old Techno
%samples Asleep__4.11-4.31_ Strings
%samples Greatest_Love_of_All_2min57 Pops
%samples Hey Tonight Rock
%samples Severance__1.31-1.51_ Pops
%samples Shinsho_pool_3min45_4min4 Pops
%samples SlavesOfFear Metal
%samples The Chastising of Renegade Rock
%samples TrosYGareg Pops
%samples 4-Sound-English-male.441 Voice
%samples 9-Have-big-expensive-car.441 Pops
%samples 12-German-male-speech.441 Voice
%samples 15-Good-evening.441 Voice
%samples 21-classic.441 Classic
%samples 24-Greensleeves-Korean-male-speech.441 Voice
%samples 25-This-is-the-end.441 Voice
%samples 27-last-song-drums-and-trampets.441 Perc.
%samples bonhemian_rhapsody Rock
%samples clapton_44k Guitar
%samples Coral Pops
%samples ExitMusic Vocal
%samples liberate Metal
%samples NewYorkCity Pops
%samples sandman Rock
%samples take_your_finger_from_my_hair Rock
%samples Changes Rock
%samples Girl_In_The_Fire__Sample_ Rock
%samples Hotel California Rock
%samples Jupiter, the Bringer of Jollity Classic
%samples Last_Of_The_Mohicanz__Sample_ Electro
%samples Only Time Newage
%samples Through The Fire And Flames Metal
%samples With Love (Outro) Techno
%link_path tracks/ .wav .opus1.1.b96k.opus .qaac.cvbr96k.mp4 .aotuv.q2.2.ogg .lame3.99.5.v5.mp3 .faac.abr96k.mp4 .faac.vbr30q.mp4
Download logs
You can download a ZIP file containing all logs . Encryption keys and parsed text included.
Contact
Kamedo2: Twitter@kamedo2