I think a big part of the reason there's still so much debate around things like audibility differences in DACs, cables, software, etc etc is simply because a lot of people misunderstand placebo/expectation bias themselves.
There's a few key points:
1) Placebo doesn't mean you're lying or stupid
2) Expectation bias goes both ways
3) Placebo affects EVERYONE regardless of your experience, hearing ability, gear, age, knowledge etc
4) A difference cannot be 'too big/obvious to be placebo'
5) Sighted testing isn't reliable, and controlled blind testing is actually really hard to do. Most 'blind tests' are not actually blind.
6) Expectation bias can affect you even if you don't consciously expect a particular outcome
1) Placebo doesn't mean you're lying or stupid.
This is probably the biggest one. Some people seem to think that when someone says 'it's probably just placebo', that they're accusing the other person of lying. Placebo is a crazy powerful influence.
When you say "I can hear a difference in cables", I BELIEVE YOU. In fact I'm pretty certain that yes, you ARE hearing a perhaps even very clear and obvious difference.
What I'm not certain about is whether that difference you're hearing is because of a genuine difference between the cables, or because your mind is playing tricks on you.
Secondly, this doesn't mean 'you're stupid'. EVERYONE is susceptible to placebo. You are, I am, and everyone reading this is. It has nothing to do with intelligence and it is not something that we can be trained to overcome.
2) Expectation bias goes both ways
"Expectation bias occurs when an individual's expectations about an outcome influence perceptions of one's own or others' behaviour" - ScienceDirect.com
Obviously the main argument for the existence of audible differences in things like cables is simply that....well...people say they can hear them.
And as explained above, I'm CERTAIN that they are hearing a difference. BUT, in order to show that the difference being heard is due to a genuine difference between cables and not just placebo, you MUST do a controlled blind test to eliminate placebo as a factor. You must entirely remove your ability to discern which cable is which by any other means than hearing. And then do enough runs to ensure that the result was not just obtained by chance. (See point 5)
If you don't, then whilst yes the difference might seem huge and obvious to you, neither you nor I actually know that it really exists.
Contrarily, many people will use an argument such as 'Well I tried both and I couldn't hear a difference so you must be imagining it/it must be placebo'.
This argument forgets that expectation bias and placebo goes both ways. If you believe cables or DACs or anything at all doesn't make a difference, then it's quite likely you won't hear a difference when you compare. Because expectation bias goes both ways!
I could go get swimming lessons. But if I fundamentally believe swimming teachers can't teach you to swim, they can't really force me to float now can they?
Blind testing can show that something is audible, it cannot be used to prove a negative/prove that something is NOT audible.
3) Placebo affects EVERYONE regardless of your experience, hearing ability, gear, age, knowledge etc
This is again quite a common counter-argument to the suggestion that a difference might be caused by placebo. People will list off a resume of their experience, knowledge, intelligence, hearing ability, wealth etc, but none of this matters.
EVERYONE is susceptible to placebo. It doesn't matter how good your hearing is, what you've heard before, or how much experience/training you have.
Placebo affects everyone including you, me, Albert Einstein and there's a reason why even in fields like medicine where effects should be more obvious, blind testing and control groups are used. Because sighted testing is NOT reliable.
Another related argument being that 'it can't be expectation bias because I went in expecting the opposite to happen'. Unfortunately again, this doesn't matter. What you consciously expect doesn't negate expectation bias. For placebo to be a potential factor you simply have to be AWARE of what you're listening to.
To test properly, you have to fully remove your knowledge of what is being listened to.
4) A difference cannot be 'too big/obvious' to be placebo
Related to the previous point, nothing is 'too obvious' to be placebo.
Even outside of audio there have been plenty of other people that have even taken advantage of just how powerful placebo and expectation bias can be.
Take a look at faith healers for example. People will go from being in such chronic pain they can't walk to suddenly running and crying across a stage as a 'miracle' or the pain of their cancer fades away.
If you want a couple interesting resources on the power of suggestibility, I would HIGHLY recommend watching 'an honest liar'. A documentary about the life of James Randi, who fun fact: Had an outstanding prize of $1,000,000 available to anyone able to demonstrate supernatural powers, INCLUDING the ability to hear differences in speaker cables. No one was able to pass....
His debunking of various faith healers, magicians and psychics in often very high profile and elaborate ways are both entertaining and fascinating as he explains how all of it works and why they work so well.
His TED Talk is also excellent:
I had quite an entertaining personal experience with someone similar, a stage hypnotist called Martin Taylor. (He was the one who Derren Brown credits as inspiring him into his current line of work).
He described himself as 'the hypnotist who doesn't use hypnosis'.
In his act he with complete and utter transparency explains how 'hypnotism' works. How there are no trances, there are no magic or intricate brain-melting methods used to get people to do his bidding. No, people will do the things he says, hear the things he tells them to hear, smell the things he tells them to smell simply because the power of suggestion and expectation can be pushed to extraordinary limits. Especially when there's a room full of a couple hundred people watching. Our brains are fascinating and flawed and can be exploited.
He comes up and quite clearly explains that 'hypnosis' doesn't exist and that none of it is real. And yet 5 minutes later he has people with their hands stuck together unable to pull them apart, people shouting things in response to trigger words, falling asleep on command and smelling a horrifically unpleasant smell that doesn't actually exist. He's excellent. And a great example of how no, nothing is 'too obvious' or big to not be placebo. The people on stage didn't ACTUALLY have their hands stuck together. They were just told that it was impossible to pull them apart and due to a mix of suggestion and pressure from both themselves and others, they therefore were unable to do so.
5) Sighted testing isn't reliable, and controlled blind testing is actually really hard to do. Most 'blind tests' are not actually blind.
I've already talked above about how sighted testing isn't reliable. No matter how 'obvious' a difference is or how clear it seems to you. If you are testing sighted, then you cannot be sure of what is causing that difference.
But the other issue is that there are a number of 'blind tests' that get shared in various places....that aren't actually blind. It's actually REALLY hard to do a blind test properly.
Let's take the example of cables.
- You have to ensure that the switching itself is random.
This is the 'double blind' part you might also hear. You cannot have a human determine whether cables should or should not be swapped during the test, because our own pattern prediction can skew results. Switching must be determined by something as close to truly random as possible, such as a random number generator.
- You can't let the participant(s) know whether cables have been swapped.
You need them to leave the room, not be able to hear what is happening inside, and make sure that the delay between runs is fixed. Otherwise even subconsciously they might have an indication of whether cables have been swapped based purely on how long it took before they were able to come back in, or even the sound of the cables being plugged in. Some cables will sound different when being moved/plugged in than others. So just having their view hidden is insufficient.
Just these two alone already make it a fair bit more involved than you might expect, but the third point is the biggest one:
- You MUST do a large number of runs, else even if you got all of them right, your result is not actually statistically significant.
If you do a low number of runs, then even if you get most or all of them right, there is a large chance that you could have just gotten that result by guessing.
For example if you get 8/10 right, there is still a 5.5% chance you just got that by guessing.
If you keep the proportion of correct answers the same (80%), but double the number of runs to 20, then 16/20 correct reduces that probability of the result being just luck to 0.6%! MUCH lower.
Do 50 runs, and get 40/50 correct, and there is only a 0.001% chance you could have obtained that by guessing.
So many blind tests online either do not appropriately control for possible tells or indicators. Or only do a small number of runs.
7 or 8/10 might seem significant, that's almost all of them right! But actually, there is still a noteworthy chance it was just luck. Do more runs, and get a better result, not much point doing something half-heartedly and getting inconclusive results when you could just do it a bit longer and get something much more concrete.
6) Expectation bias can affect you even if you don't consciously expect a particular outcome
As mentioned earlier, expectation bias only requires you to be AWARE of what is being listened to. Either directly via sighted testing, or just getting an indicator that might skew probability of results, such as the sound of cables being swapped or the time it took to do so.
What you believe going into the test doesn't actually determine whether Expectation bias is/isn't a factor. It ALWAYS is