A blind A/B test plays two versions of the same audio without telling you which is which. Match their loudness, play the same short passage from both in shuffled order, pick one each round, and look at the file names only at the end. Run at least 10 rounds. Picking the same file 9 or more times in 10 is unlikely to happen by guessing (p ≈ 0.011).
Why test blind
When you know which file is the new revision, the expensive master or the lossless file, you expect it to sound better, and you tend to hear it that way. A sighted comparison, like a normal A/B test of a mix, measures your ears and your expectations together. A blind test removes the labels, so only the sound is left to choose by.
Blind testing is worth the effort when a decision is close or costly. Typical cases are approving one of two masters, choosing between two mix revisions late in a project, and checking whether a high-bitrate MP3 is good enough to send instead of a WAV.
Blind A/B, ABX and double blind
These tests answer different questions.
- Blind A/B preference test. You hear two unlabelled versions and pick the one you prefer. It answers "which one do I like better, when I can't see the names?"
- ABX test. You hear A and B, then an unknown X that is one of them, and say which one X is. It answers "can I hear a difference at all?" It is the usual test for lossless against lossy audio.
- Double blind test. Neither the listener nor the person running the test knows which file plays. Software that shuffles the files and hides the names makes a test double blind, because nobody in the room can give a hint.
A preference test can't prove that you hear no difference. If your picks split evenly, either the versions sound the same to you or you like them equally. For mixing and mastering decisions that is usually enough, because the question is which one to keep.
How to run a blind A/B test
- Match the loudness. Measure each file's integrated LUFS and turn the louder one down to the quieter one. A difference of about 1 dB biases the choice, and in a blind test it also gives the louder file away. The level matching guide shows how.
- Align the files. Both versions need to start at the same sample, so each round plays the same moment in the song.
- Pick short passages. A phrase or two of the section you care about is enough. Long passages make you compare memories instead of sounds.
- Shuffle every round. The order of the two versions has to change at random from round to round, or you learn the pattern.
- Decide what you are judging. Pick one quality per test, such as the vocal level or the low end, and judge only that.
- Hide the names until the end. Don't look at results between rounds, and don't change headphones or volume halfway.
ABSeq has a Blind Test that runs these steps for tracks A and B. Each round plays the same short passage from both, shuffled and labelled Version A and Version B, and you pick the one you prefer. You set 1 to 50 rounds and the seconds per passage, Auto Gain is on by default, and the file names appear only in the results. The help guide covers how to set up a blind test. It is a preference test, not an ABX test.
How many rounds you need
One round tells you nothing. With two versions, a guess lands on either one half the time, so a single pick is a coin flip. Over many rounds, guessing averages out near half, and a consistent preference stands out.
| Rounds | Same pick needed | Chance by guessing |
|---|---|---|
| 10 | 9 or more | about 1.1% (p ≈ 0.011) |
| 16 | 12 or more | about 3.8% (p ≈ 0.038) |
The same numbers apply to ABX, where you count correct answers instead of picks. In 16 ABX trials, 12 or more correct has under a 5% chance of coming from guessing.
More rounds give a clearer answer but tire your ears. If you need more than 16, split them into two sessions with a break between them.
Reading the results
- A clear preference, such as 9 of 10. You hear a difference and prefer one version. Keep that one, or work out what it does better.
- A split, such as 6 of 10. The difference is too small for you to prefer either one on this system. Pick on other grounds, such as file size or cost.
- A clear preference for the version you expected to lose. Check the level match first. If the levels are equal, the change you made may have hurt the track.
Write down the system you tested on. A result on headphones doesn't carry over to a car or a phone speaker, so repeat close calls on a second system.
Blind testing lossless against MP3 or AAC
To test whether you can hear a lossy encode, compare the lossless file (WAV or FLAC) with an MP3 or AAC made from the same master. Match the loudness and check the alignment as for any other test. An ABX test is the usual choice here, because the question is whether you hear a difference, not which one you like.
Check the true peak of the master first. A master at 0 dBFS sample peak can have inter-sample peaks above 0 dBTP, which can clip when the file is encoded to AAC or MP3. In that case you may hear the clipping, not the codec. The true peak guide explains the difference, and the guide to comparing two audio files covers other format checks.
Common mistakes
- Skipping the level match, so the louder file wins or gives itself away.
- Running three or four rounds and treating the result as final.
- Checking the score halfway and then listening for the version you think is winning.
- Using passages so long that you compare your memory of each one.
- Changing the volume or the headphones partway through.
Questions
What is a blind A/B test in audio?
A blind A/B test plays two versions of the same audio without telling you which is which, and you pick the one you prefer in each round. The file names appear only at the end, so your choice comes from the sound rather than from knowing which file is new or expensive.
What is the difference between an ABX test and an A/B test?
An A/B preference test asks which of two unlabelled versions you prefer. An ABX test plays A, B and an unknown X, and asks which one X is, so it tests whether you can hear a difference at all. ABX is the usual test for lossless against lossy audio.
How many rounds should a blind test have?
At least 10. In 10 rounds, picking the same file 9 or more times has about a 1.1% chance of happening by guessing. In 16 rounds, 12 or more has about a 3.8% chance. One to five rounds can't separate a real preference from luck.
Can you hear lossless audio in a blind test?
It depends on the listener, the music, the encode and the playback system, so the answer comes from your own test. Run an ABX test with loudness matched and the files aligned, and count correct answers. In 16 trials, 12 or more correct is unlikely to be guessing.
What is a double blind test in audio?
In a double blind test, neither the listener nor the person running the test knows which file is playing, so nobody can give a hint by accident. Software that shuffles the files and hides their names until the end runs a double blind test by itself.