Lossless audio codec comparison revision 5 - CDDA part

Introduction

This document compares the performance of lossless audio codecs on CDDA material, which are PCM audio files with a bit depth of 16 bit per sample and a samplerate of 44.1kHz.

The test corpus consists of 64 sources of very different sound. An attempt was made to create a balanced though extensive set covering many possible sounds. While most sources contain music, not all do.

Method

To compare the performance of each codec, the following steps are followed for each combination of corpus file, codec and codec setting:

  • A WAV file is placed on a large enough ramdisk
  • An MD5sum is calculated for the WAV file excluding its header
  • The WAV file is encoded by the chosen codec provided with the required settings. The amount of CPU time required to do this conversion is measured and the resulting filesize is recorded
  • The encoded file is decoded by the chosen codec. The amount of CPU time required to do this conversion is measured
  • An MD5sum is calculated for the resulting decoded file excluding its header
  • The MD5sum of the provided WAV file and the decoded file are compared

The following codecs and settings are used:

Codec Settings used
FLAC -0, -1, -2, -3, -4, -5, -6, -7, -8
ALAC --fast, [-]
WavPack -f, [-], -h, -hh, -x4f, -x4, -x4h, -x4hh
TAK -p0, -p0e, -p0m, -p1, -p1e, -p1m, -p2, -p2e, -p2m, -p3, -p3e, -p3m, -p4, -p4e, -p4m
OptimFROGpresets 0 through 10 and max
Monkeys Audio -c1000, -c2000, -c3000, -c4000, -c5000
La [-], -high
MP4ALS -a -o5, -a, -a -o20, -a -o40, -a -b, -a -b -o20, -a -b -o40, -a -b -o1023, -7
WMA [-]
Shorten [-]
TTA [-]

For each codec, the latest Windows binary provided by the author of the codec is used, no specially tuned compiles are used. In case of ALAC, encoding is done with refalac64 as provided by QAAC. In case of WMA, encoding is done with WMAEncode.exe (which uses the encoder provided by Windows 10) and decoding with FFmpeg 5.0.

Measurements are made on a Windows 10 machine with a AMD A4-5000 CPU with 4GB of RAM. This CPU has all x86 instruction set extensions up to and including AVX (i.e. lacks AVX2). Measuring the CPU-time used is done with timer64.exe, part of the 7-max/7-benchmark suite.

Timing is done per track. This measured time is divided by the content length, i.e. the execution time of the encoding or decoding process in seconds divided by the playback length of the track or chapter in seconds. The result of this division is called CPU-usage. The filesize of the encoded file is divided by the filesize of the original WAV file to calculate a compression.

Results per source are obtained by averaging the compression and CPU-usage so each track or chapter contributes the same amount to the average, i.e. length of the track or chapter is not incorporated.

The average of all sources is obtained by averaging the results per source, again without any weighing. The total results are therefore not influenced by the length of the corpus content, each source contributes an equal amount to the average.

Results (PDF)

The best results will be obtained in the bottom left corner of the graphs: this represents the best compression (smallest file size) and the lowest CPU-usage (fastest compression and/or decompression).

In the graph each codec is represented by a group of markers connected by a line. Each combination of settings as mentioned in the previous table corresponds to one marker, in the order listed in the table. The first combination of settings is (usually) the fastest and the last the slowest. Therefore, the marker closest to the upper left corner of the graph corresponds to the first listed combination of settings, and the marker closest to the lower right corner corresponds to the last listed combination of settings.

Discussion

Looking at the average of all sources, there is not a single clear winner. In case maximum compression is desired, OptimFROG does best, with the highest compressing settings having quite a clear margin over the next best competitor. When a balance between compression and speed is desired, TAK seems to hold a good spot, with its decoding speed only second to FLAC while achieving much higher compression.

When looking at the results per source, the results shift a bit, but large shifts are only found for peculiar sources. For example, OptimFROG does exceptionally well on the diffuse sound sources. Other big difference are visible on mono audio encoded as stereo (with the two channels carrying bit-for-bit exact same content) where it can be clearly seen that Shorten and some presets of refalac and FLAC do not perform stereo decorrelation.

Other peculiar results can be spotted in the graph for Ryodi Ikeda - Dataplex, where Monkey Audio's insane preset (which is the slowest and usually best compressing) is outperformed by its other presets. Something similar happens with La and OptimFROG. Also Junichi Masuda Go Ichinose - Pokemon Gold stands out as being different.

Sources used

Solo instruments
Artist - AlbumMain instrumentationCommentYear
Jeroen van Veen - Einaudi: Waves The Piano Collection Piano2013
Jean Guillou - The Great Organ of St. Eustache, Paris Church organ2006
Bobby McFerrin - The Voice Voice1984
Dan Brown - Angels & Demons VoiceAudiobook2009
Carmen Cavallaro - Songs of Our Times, Song Hits of 1921 Piano[5][1]1942
Konrad Junghanel - Europaische Lautenmusik Volume 1 11-string baroque lute1989
Nicolò Paganini - 24 Capricci (Salvatore Accardo) Violin1978
 
Acoustic ensembles and bands
Loosely sorted by size of ensemble
Artist - AlbumMain instrumentationCommentYear
Alela Diana - The Pirate's Gospel (tracks 1, 2, 6, 7, 10 and 11)Voice and guitar with occasional vocal stacking2004
Fredrik Fors and Sveinung Bjelland - Les Nouveaux MusiciensClarinet and piano2004
Rosenberg Trio - Djangolists2 guitars and double bass2010
Szakcsi - Viragom, Viragom2 Voices, piano, flute1998
Berlage Saxophone Quartet - SaxoFOLK4 saxophones2014
Apocalyptica - Inquisition Symphony4 cello's1998
Flanders Recorder Quartet - Armonia di Flauti4 recorders1998
Dave Brubeck Quartet - Time OutSaxophone, piano, double bass, drums1959
Equilibrium - Rekreatur (Acoustic bonus CD)Flute, accordion, guitar, bass guitar, drumsVery fast paced2010
Dvorak - String Quartet No. 32 violins, alto violin, cello1993
Alison Krauss & Union Station - New FavouriteVoice, guitar, steel guitar, fiddle, upright bass2001
RiverBend Bluegrass Live at Jacoby Arts Center on 2021-07-29Voice, mandolin, banjo, violin, steel guitar, double bassLive, [1]2021
Piet Jeegers Clarinet Choir - Volume 415 clarinets (1x E♭, 8x B♭, 2x Alto, 2x Bass, 1x Contra Alto, 1x Contra Bass)2002
 
Amplified ensembles and bands
Loosely sorted by amount of 'overdrive'
Artist - AlbumMain instrumentationCommentYear
Coldplay - ParachutesVoice, electric guitar, bass guitar, drums2000
Michael Bublé - Michael Bublé Meets Madison Square GardenVoice, electric guitar, piano, drums, brass, crowd2009
Albert King - Years Gone ByVoice, 2 electric guitars, keyboard, bass guitar, drumsVinyl rip1969
Status Quo - Whatever You WantVoice, 2 electric guitars, bass guitar, drumsVinyl rip1979
Muse - HAARPVoice, overdriven guitar, bass guitar, drumsLive2008
Iron Maiden - PowerslaveVoice, 2 overdriven guitars, bass guitar, drums1984
Animals as Leaders - Weightless2 overdriven 8-string guitars, drums2011
In Flames - Soundtrack to Your EscapeGrowling vocals, 2 overdriven guitars, bass guitar, drums2004
Nightwish - WishmasterVoice, overdriven guitar, bass guitar, keyboard, drums2000
Epica - The Holographic PrincipleClean and growling vocals, guitar, synth, bass guitar, drums, orchestra and choir2016
 
Big bands
Artist - AlbumMain instrumentationCommentYear
Bert Kaempfert - A Swingin' SafariBig band with string section and amplified vocals1962
Dutch Swing College Band - Digital DutchDixieland band[3]1982
Fanfare Ciocarlia - Baro Biao; World Wide WeddingFanfare1999
Glenn Miller - The Best Of The Lost Recordings And Secret BroadcastsBig band[4]1943
John Allmark Jazz Orchestra Live at The Met on 2019-05-20Big bandLive, [1]2019
Ida Gotkovsky - Works for Symphonic BandSymphonic Band (also known as concert band or wind orchestra)1989
 
Orchestral and choral
Loosely sorted by size of ensemble
Artist - AlbumMain instrumentationCommentYear
Confido Domino Minsk - Sacred Choral Music From White RussiaChoir1994
Vivaldi - Four Seasons (Amsterdam Baroque Orchestra, cond. Ton Koopman)Baroque orchestra (small)1996
Dario Marianelli - Pride & PrejudicePiano and chamber orchestraFilm score2005
J.S. Bach - MagnificatBaroque orchestra (small), vocal soloists and choirLive2012
Michael Giacchino - The Incredibles'Processed' orchestra, very upfront brass [7]Film score2004
Nicholas Hooper - Harry Potter and the Order of the Phoenix'Processed' orchestra and choir [7]Film score2004
Tchaikovsky - The Nutcracker (Kirov Orchestra, cond. Valery Gergiev)Ballet orchestra (intermediately sized)1998
Stravinsky - Le Sacre du Printemps / Petrouchka (Isreal Philharmonic Orchestra cond. Leonard Bernstein)Large symphonic orchestra1983
Verdi - Requiem (Berliner Philharmoniker cond. Claudio Abbado)Large symphonic orchestra, vocal soloists and very large choir2001
Mahler - Symphony no.8 (Concertgebouw orchestra cond. Bernard Haitink)Very large symphonic orchestra, vocal soloists and exceptionally large choir [6]1971
 
Electronic music
Loosely sorted by how 'irregular' music is
Artist - AlbumDescriptionCommentYear
The Ambient Visitor - Across the Milky WayVery slow ambient music, drone2021
Kraftwerk - AutobahnAnalog synthesizers, vocoder1974
Fatboy Slim - You've Come a Long Way, BabySamplers (vocal and drum), synthesizers1998
Owl City - Ocean EyesAudibly pitch corrected voice, synthesizers, drum computers2009
Taylor Swift - 1989Stacked voice, synthesizers, drum computers2014
Lil Wayne - Tha Carter IIIVoice, samplers, keyboard, drum computer2008
Xzibit - At the Speed of LifeVoice, samplers, drum computer1996
Daft Punk - Alive 2007Sampling, vocoded voiceLive2007
Junichi Masuda, Go Ichinose - Pokemon GoldGame Boy emulator (2x square wave, noise generator, wavetable)1999
The Prodigy - Invaders Must DieVoice, samplers2009
Ubiktune - SOUNDSHOCK: FM FUNK MADDNESS!!FM Synthesizers2011
Skrillex - BangarangHighly 'overdriven' synthesizers, vocal samples, drum synth2011
Merzbow - Cloud Cock OO Grand Very noisy music 1989
 
Other
Artist - AlbumDescriptionCommentYear
Ryoji Ikeda - dataplex Dominated by transients 2005
Horacio Vaggione - Fluides Very short sounds (microsound) 2018
N/ARecordings of diffuse soundfields produced by choirs, ensembles and orchestras [2]Own recordings2015
N/ARarewares test samples (https://www.rarewares.org/test_samples/)2005
N/AVarious recordings of ambient sounds taken from Freesound.org [8]

[1] Available on archive.org

[2] Consists of: Recording of a choir in a small church; Recording of a (different) choir (part of recording is a capella, part of recording accompanied by a small ensemble) in a small theater; Recording of music directors exam, featuring first a small, later an intermediately sized and finally a very large choir in a concert hall; Recording of a choir, orchestra and solists in a intermediately sized church; All recordings with boundary microphones spaced 3 meters apart on a balcony, as far from the performers as possible

[3] 'Launch title' for first CD players

[4] Professional vinyl rip and restoration from a radio transcription disc

[5] Professional vinyl rip from a 78rpm commercial release

[6] Mahler's symphony no.8 is nicknamed 'Symphony of a Thousand' for its exceptionally large orchestra and choir

[7] Orchestra has been recorded in seperate sections (with sound barriers) so mix can be varied and some instruments are (at times) mixed in much louder or quieter than the really are. Also, for Harry Potter and the Order of the Phoenix, choir had been dubbed over orchestra. See https://youtu.be/0sASbCxnjAg for Harry Potter and the Order of the Phoenix and https://www.youtube.com/watch?v=zA1XcaQI5eU for Incredibles

[8] Sound include birdsong, traffic and rain