[PointyFox]

I did three blind level-matched tests between a Schiit Modi 2 and a Schiit Bifrost Multibit and no one was able to tell a difference. People usually had a preference, but they were all over the place and often contradicted themselves, often preferring one thing to itself when the same DAC was used twice in a row.

 

[Maxx134]

I’m also going to disagree with your narrative of the utilization of blind testing - one of the benefits is to eliminate existing biases, not to simply appeal to open minded novices. Though blind testing certainly works just fine on the open minded

Yeah was speaking in general so probably off.
I do feel all the data we have so far from blind testing is very valuable.

I just whish in some cases it would limit or weed out any doubtful or "stressed" testers.

Everything that I compare using careful controls sounds the same.

Ok so may I ask did you use same headphones? Speakers?

It could be that any "common denominator" can be an influence to your outcome.

 

[bigshot]

Ok so may I ask did you use same headphones? Speakers? It could be that any "common denominator" can be an influence to your outcome.

I don't want to introduce all the transducer error and room effects with my speaker system, so I use headphones for my careful comparisons. I use Oppo PM-1s.

I'm looking for clear differences, but I can't find any. I look at the specs and measurements and compare them to the thresholds of audibility and I can't see any reason why they should sound different. I'm just looking for something that verifies an actual clear audible difference and no one seems to be able to provide me with it. Only subjective impressions with no control over bias.

To be honest, sound reproduction seems to be more of a religion to people who claim to hear differences than a science.

 

[Maxx134]

I did three blind level-matched tests between a Schiit Modi 2 and a Schiit Bifrost Multibit and no one was able to tell a difference. People usually had a preference, but they were all over the place and often contradicted themselves, often preferring one thing to itself when the same DAC was used twice in a row.

I would suggest a new approach.
Instead of having them guess randomly at anything.

Switch Tell them a change of dac has been made.
This eliminates the initial doubt in their minds, and allows for the brain to work on what the change is.

The object of this proposition, is to eliminate the variable of doubt in their minds.
So that they can focus on the change, and then ask them which dac they think it is.

To me, just randomly telling them to guess if there was any change at all, will defeat the challenge as dead in its tracks from the outset, as it allows the minds imagination to go wild.

But thats only my opinion.

 

[PointyFox]

I would suggest a new approach.
Instead of having them guess randomly at anything.

Switch Tell them a change of dac has been made.
This eliminates the initial doubt in their minds, and allows for the brain to work on what the change is.

The object of this proposition, is to eliminate the variable of doubt in their minds.
So that they can focus on the change, and then ask them which dac they think it is.

To me, just randomly telling them to guess if there was any change at all, will defeat the challenge as dead in its tracks from the outset, as it allows the minds imagination to go wild.

But thats only my opinion.

That's exactly what I did. I even faked switching cables during the tests. Most of the times I actually switched DACs but sometimes I just pretended to. I didn't inform them of any results until after the testing was complete. They thought I was just switching from one to the other each time, but didn't know which I was switching to since I randomized it. So they thought I was doing A then B, or B then A.

 

[Maxx134]

I use Oppo PM-1s.

I owned PM1 , then PM2.
Really great cans.
The construction quality alone far exceeds most cans.
Have you heard of the "Audio Zenith PMx2"?
A bit more expensive but you may like them.
I heard them and preferred them over the stock versions.
Anyways I sidetracked.

Maybe in the future someone will find a way to bridge the subjective and objective.
I find it most necessary to use measurements especially in modding headphones.

 

[bfreedma]

I would suggest a new approach.
Instead of having them guess randomly at anything.

Switch Tell them a change of dac has been made.
This eliminates the initial doubt in their minds, and allows for the brain to work on what the change is.

The object of this proposition, is to eliminate the variable of doubt in their minds.
So that they can focus on the change, and then ask them which dac they think it is.

To me, just randomly telling them to guess if there was any change at all, will defeat the challenge as dead in its tracks from the outset, as it allows the minds imagination to go wild.

But thats only my opinion.

Nothing wrong with a different approach, but what you’re suggesting would produce a different data set than what the blind test in discussion would develop. The goal of the blind test protocol isn’t for the subject to identify a preference, but to simply identify that a change has been made and to see if that change can consistently be demonstrated to be identifiable. Removing the variability of the test by always changing (in this case) the DAC wouldn’t develop accurate data.

If the controlled testing as described above indicates subjects can identify a change in DACs in a statistically significant way, then a preference identification consistency test would be an interesting follow on.

 

[PointyFox]

Nothing wrong with a different approach, but what you’re suggesting would produce a different data set than what the blind test in discussion would develop. The goal of the blind test protocol isn’t for the subject to identify a preference, but to simply identify that a change has been made and to see if that change can consistently be demonstrated to be identifiable. Removing the variability of the test by always changing (in this case) the DAC wouldn’t develop accurate data.

If the controlled testing as described above indicates subjects can identify a change in DACs in a statistically significant way, then a preference identification consistency test would be an interesting follow on.

I tried killing two birds with one stone and asked them their preference each time. Sometimes they were different DACs, sometimes the same. The only thing that was statistically significant was that they couldn't identify differences between DACs and didn't favor one over the other consistently. There was about a 54% preference for the much cheaper DAC, but this wasn't significant.

 

[bigshot]

I owned PM1 , then PM2. Really great cans. The construction quality alone far exceeds most cans.
Have you heard of the "Audio Zenith PMx2"?

Maybe in the future someone will find a way to bridge the subjective and objective.
I find it most necessary to use measurements especially in modding headphones.

I was asked by Oppo to be part of a group to evaluate the PM-1 and HA-1 before they went to market. I corresponded with the designer of the PM-1s and asked him a lot of questions and did some tests and let him know what I thought. At the end of the testing period they liked my input so they gave them to me. I don't think my evaluation of the HA-1 went over quite so well, but they let me keep that too. I would normally not invest that much money in headphones. My normal listening is done on speakers. But I'm very satisfied with the Oppo. They only deviated from my ideal response curve by a couple of dB in only one small range. Best cans I've ever heard. Never heard the AZPMx2. Are they as comfortable as the PM-1s? Most of the testers I worked with were satisfied with the sound and focused on the comfort. I was the opposite. I thought they were very comfortable and focused on the response.

I find that measurements are very useful. But it's also important to establish a threshold of audibility with controlled listening tests. We can measure a lot of things we can never hear. It's important to keep measurements in the context of the specs of human ears.

 

[castleofargh]

the oxygen masks will drop automatically in case of a depressurization of the cabin, the emergency exits are; 2 on the front, 2 on the wings and 2 in the back. if you look at the top of the page, you'll notice the topic is about cables.


@Elecroestatico if I may use a made up story(any similarity to somebody in this topic would be totally not surprising ^_^):
so there are those problems to solve for a job interview. some clearly have a hard time doing so and give up saying most are impossible to solve. I come and declare "lol! I solved them, it's really obvious. but it's ok, we're not all born with the same brain and knowledge. so if it's impossible for some, then that's just how it is. perfectly natural and expected". so far I seem pretty legit and reasonable. somewhat cocky but hey if I did solve the all stuff maybe I deserve to brag a little.
but then, at the end when the employee comes to get the work done, my copy is empty and I tell him that it's all in my head, all solved accurately. he can trust me 130%, I'm the guy for the job. what do you expect that dude to think of me?
later in the hallway, when asked how I solved the problems by fellow examinees, I answer that I went with my guts like I did for decades because I believe it's just as reliable, if not, more reliable than following the accepted method. again, what do you expect those guys to think of me?

if you don't want to properly demonstrate the claims you make. there is a very effective ancestral technique involving not making claims with absolute confidence in the first place. it works wonders.
in here where we play pretend to care about facts more than the size of our own penis. we really don't enjoy reading empty claims. so we ask people to "put up or shut up", hoping that it was just a misunderstanding where the guy making the claim will correct his sentence into an opinion or an impression. or that he will stick to his claim and back it up with reasonable evidence that can be shared with us.

now if you wish to troll all the guys who claim there are no differences between DACs in general or cables in general, I'm with you. because of course nobody can prove that, so nobody should claim it. simple enough. logic doesn't apply only to people we disagree with.


a listening test proving audible differences between gears, has to be an actual test(captain obvious), not something where we know at all time what's going on, where an impression magically morphs into a fact. also it has to be an experience testing hearing and hearing alone. again, duh! but somehow most audiophiles fail to qualify for those so very obvious requirements of a listening test.
if I see 2 different products, obviously I feel a difference, I know there are some, I see some. it would be weird not to get the impression that they are indeed different. but that's not a listening test.
and deciding to willingly focus on sound does not effectively isolate sound from everything going on in our head. if you're not convinced, there are so many experiments demonstrating as much, most can be done to trick friends and family while making my point.
a favorite of this section is the McGurk effect because it's so self explanatory:


it should be pretty obvious that a blind testing, given how annoying it is to set up and run properly, wouldn't be systematically used by scientists when testing impressions if there was an easy effective alternative. like telling the subjects "ok now you only concentrate on the stuff we test, and just tell me if you pass the test, I'll take your word for it because you know what you heard". "oh and BTW, if you somehow were to believe you passed, you could consider yourself special and belonging to the elite. it would also reassure you about all the money you spent. just think about that, no risk to be biased into making stuff up just to get the result you really want to get so very much. and certainly we have no reason to set up a test that would identify when people happen to think they perceive differences but really don't. we can fully trust them to know better than being biased like newbies".
in any domain, this is a lame sarcastic joke. in audio forums, the majority argues very seriously that it's exactly how a listening test must be done. ignorance and group thinking can really achieve impressive results.



so to summarize:
- a test should involve actual testing, not be me asking to myself how much I agree with what I think I feel.
- if we have X independent variables in a test about audio, and most aren't audio variables, we're not testing audio! in fact we're probably not getting any meaningful data.
- if you can't set up a proper listening test by yourself, that's perfectly understandable. and if you can but don't want to, again, we all very much understand. that shiit is hard and boring at the same time. no shame, no judgement, I often don't properly test stuff myself ^_^. but then, let's agree that we're not qualified to pass judgements on the results of those tests we didn't do. pretty simple request IMO.




no it's not a long post!

 

[Steve999]

So my takeaway is you interrupted me listening to Duke Ellington for me to hear someone go “ba ba ba” over and over again. Thanks!

Seriously, that is a freaked out illusion. Is that posted in the top stickie thread (that thread is about the weather, so your post would be a little OT, but I think that video should be the first thing in the top sticky anyway). It looks like it was nice weather out there. And the illusion makes a very major point in a much different way than a lot of dry reading or arguing or diligent Q&A or tedious a/b/x testing will. I think it would make a great introduction to this sub-forum if people wonder what we’re about here.

the oxygen masks will drop automatically in case of a depressurization of the cabin, the emergency exits are; 2 on the front, 2 on the wings and 2 in the back. if you look at the top of the page, you'll notice the topic is about cables.


@Elecroestatico if I may use a made up story(any similarity to somebody in this topic would be totally not surprising ^_^):
so there are those problems to solve for a job interview. some clearly have a hard time doing so and give up saying most are impossible to solve. I come and declare "lol! I solved them, it's really obvious. but it's ok, we're not all born with the same brain and knowledge. so if it's impossible for some, then that's just how it is. perfectly natural and expected". so far I seem pretty legit and reasonable. somewhat cocky but hey if I did solve the all stuff maybe I deserve to brag a little.
but then, at the end when the employee comes to get the work done, my copy is empty and I tell him that it's all in my head, all solved accurately. he can trust me 130%, I'm the guy for the job. what do you expect that dude to think of me?
later in the hallway, when asked how I solved the problems by fellow examinees, I answer that I went with my guts like I did for decades because I believe it's just as reliable, if not, more reliable than following the accepted method. again, what do you expect those guys to think of me?

if you don't want to properly demonstrate the claims you make. there is a very effective ancestral technique involving not making claims with absolute confidence in the first place. it works wonders.
in here where we play pretend to care about facts more than the size of our own penis. we really don't enjoy reading empty claims. so we ask people to "put up or shut up", hoping that it was just a misunderstanding where the guy making the claim will correct his sentence into an opinion or an impression. or that he will stick to his claim and back it up with reasonable evidence that can be shared with us.

now if you wish to troll all the guys who claim there are no differences between DACs in general or cables in general, I'm with you. because of course nobody can prove that, so nobody should claim it. simple enough. logic doesn't apply only to people we disagree with.


a listening test proving audible differences between gears, has to be an actual test(captain obvious), not something where we know at all time what's going on, where an impression magically morphs into a fact. also it has to be an experience testing hearing and hearing alone. again, duh! but somehow most audiophiles fail to qualify for those so very obvious requirements of a listening test.
if I see 2 different products, obviously I feel a difference, I know there are some, I see some. it would be weird not to get the impression that they are indeed different. but that's not a listening test.
and deciding to willingly focus on sound does not effectively isolate sound from everything going on in our head. if you're not convinced, there are so many experiments demonstrating as much, most can be done to trick friends and family while making my point.
a favorite of this section is the McGurk effect because it's so self explanatory:


it should be pretty obvious that a blind testing, given how annoying it is to set up and run properly, wouldn't be systematically used by scientists when testing impressions if there was an easy effective alternative. like telling the subjects "ok now you only concentrate only the stuff we test, and just tell me if you pass the test, I'll take your word for it because you know what you heard". "oh and BTW, if you somehow were to believe you pass, you could consider yourself special and belonging to the elite. it would also reassure you about all the money you spent. just think about that, no risk to be biased into making stuff up just to get the result you really want to get so very much. and certainly we have no reason to set up a test that would identify when people happen to think they perceive differences but really don't. we can fully trust them to know better than being biased like newbies".
in any domain, this is a lame sarcastic joke. in audio forums, the majority argues very seriously that it's exactly how a listening test must be done. ignorance and group thinking can really achieve impressive results.



so to summarize:
- a test should involve actual testing, not be me asking to myself how much I agree with what I think I feel.
- if we have X independent variables in a test about audio, and most aren't audio variables, we're not testing audio! in fact we're probably not getting any meaningful data.
- if you can't set up a proper listening test by yourself, that's perfectly understandable. and if you can but don't want to, again, we all very much understand. that shiit is hard and boring at the same time. no shame, no judgement, I often don't properly test stuff myself ^_^. but then, let's agree that we're not qualified to pass judgements on the results of those tests we didn't do. pretty simple request IMO.




no it's not a long post!

 

[Glmoneydawg]

the oxygen masks will drop automatically in case of a depressurization of the cabin, the emergency exits are; 2 on the front, 2 on the wings and 2 in the back. if you look at the top of the page, you'll notice the topic is about cables.


@Elecroestatico if I may use a made up story(any similarity to somebody in this topic would be totally not surprising ^_^):
so there are those problems to solve for a job interview. some clearly have a hard time doing so and give up saying most are impossible to solve. I come and declare "lol! I solved them, it's really obvious. but it's ok, we're not all born with the same brain and knowledge. so if it's impossible for some, then that's just how it is. perfectly natural and expected". so far I seem pretty legit and reasonable. somewhat cocky but hey if I did solve the all stuff maybe I deserve to brag a little.
but then, at the end when the employee comes to get the work done, my copy is empty and I tell him that it's all in my head, all solved accurately. he can trust me 130%, I'm the guy for the job. what do you expect that dude to think of me?
later in the hallway, when asked how I solved the problems by fellow examinees, I answer that I went with my guts like I did for decades because I believe it's just as reliable, if not, more reliable than following the accepted method. again, what do you expect those guys to think of me?

if you don't want to properly demonstrate the claims you make. there is a very effective ancestral technique involving not making claims with absolute confidence in the first place. it works wonders.
in here where we play pretend to care about facts more than the size of our own penis. we really don't enjoy reading empty claims. so we ask people to "put up or shut up", hoping that it was just a misunderstanding where the guy making the claim will correct his sentence into an opinion or an impression. or that he will stick to his claim and back it up with reasonable evidence that can be shared with us.

now if you wish to troll all the guys who claim there are no differences between DACs in general or cables in general, I'm with you. because of course nobody can prove that, so nobody should claim it. simple enough. logic doesn't apply only to people we disagree with.


a listening test proving audible differences between gears, has to be an actual test(captain obvious), not something where we know at all time what's going on, where an impression magically morphs into a fact. also it has to be an experience testing hearing and hearing alone. again, duh! but somehow most audiophiles fail to qualify for those so very obvious requirements of a listening test.
if I see 2 different products, obviously I feel a difference, I know there are some, I see some. it would be weird not to get the impression that they are indeed different. but that's not a listening test.
and deciding to willingly focus on sound does not effectively isolate sound from everything going on in our head. if you're not convinced, there are so many experiments demonstrating as much, most can be done to trick friends and family while making my point.
a favorite of this section is the McGurk effect because it's so self explanatory:


it should be pretty obvious that a blind testing, given how annoying it is to set up and run properly, wouldn't be systematically used by scientists when testing impressions if there was an easy effective alternative. like telling the subjects "ok now you only concentrate only the stuff we test, and just tell me if you pass the test, I'll take your word for it because you know what you heard". "oh and BTW, if you somehow were to believe you pass, you could consider yourself special and belonging to the elite. it would also reassure you about all the money you spent. just think about that, no risk to be biased into making stuff up just to get the result you really want to get so very much. and certainly we have no reason to set up a test that would identify when people happen to think they perceive differences but really don't. we can fully trust them to know better than being biased like newbies".
in any domain, this is a lame sarcastic joke. in audio forums, the majority argues very seriously that it's exactly how a listening test must be done. ignorance and group thinking can really achieve impressive results.



so to summarize:
- a test should involve actual testing, not be me asking to myself how much I agree with what I think I feel.
- if we have X independent variables in a test about audio, and most aren't audio variables, we're not testing audio! in fact we're probably not getting any meaningful data.
- if you can't set up a proper listening test by yourself, that's perfectly understandable. and if you can but don't want to, again, we all very much understand. that shiit is hard and boring at the same time. no shame, no judgement, I often don't properly test stuff myself ^_^. but then, let's agree that we're not qualified to pass judgements on the results of those tests we didn't do. pretty simple request IMO.




no it's not a long post!

mcgurk makes me wonder if we are too quick to stick a placebo label on things?It is placebo....but the effect is very real.

 

[Steve999]

mcgurk makes me wonder if we are too quick to stick a placebo label on things?It is placebo....but the effect is very real.

You are making me think on a lazy Saturday morning. Stop it. I think we are really using the term placebo in audio by analogy only, and in this case the analogy of placebo is stretched to the breaking point. I would classify it more as illusion than placebo. And I do think we are too quick to use the placebo analogy too quickly, and perjoratively at that.

 

[castleofargh]

So my takeaway is you interrupted me listening to Duke Ellington for me to hear someone go “ba ba ba” over and over again. Thanks!

Seriously, that is a freaked out illusion. Is that posted in the top stickie thread (the thread about the weather, so your post would be a little OT, but I think that video should be in the top sticky anyway. It looks like it was nice weather out there. And the illusion makes a very major point in a much different way than a lot of dry reading or tedious a/b/x testing will. I think it would make a great introduction to this sub-forum if people wonder what we’re about here.

admit that the dude has mad flow.

off topic:
I don't know what should be made as a sticky or not, before recently we had nothing at all, and some suggestions got me to try 2 topics. I honestly don't know if that was a good thing or not. sure testing claims and myth got many more posts, but almost none about actual tests, so...
if you're talking about the "Useful and Important links" in intro to the sub section, I have to confess that I can't edit those myself and had nothing to do with them. I can ask an admin daddy to put the vid in a link, but I'm not sure that it really has meaning on its own. maybe someone should make a nice little topic about it with some brainstorming ideas related to the general phenomenon and why we're so annoying with our removal of extra variables in tests? but I suck at that, I'm the guy who turns a simple idea into 100000 boring words. even in French I tend to do that.

mcgurk makes me wonder if we are too quick to stick a placebo label on things?It is placebo....but the effect is very real.

of course it's very real, and it's the same as placebo. we perceive a different sound because we see something different, but what really causes the change in perceived sound is how our brain has assumed that a given mouth and tong movement equates to a certain sound. we believe that if we see that movement, we will hear that sound, and even when we don't, we get somehow the placebo effect. in this case it has a lot to do with speech recognition and patterns, but the 3 big ideas are still there:
1/ expecting something can be enough to make us feel like it's happening.
2/ our impressions are a giant mix of all sorts of senses, ideas, beliefs, habits... so when we decide that those same impressions entirely and accurately correlate to hearing alone, we're fooling ourselves.
3/ knowing there is the possibility of bias doesn't reduce the impact of that bias. I hate that one so very much, but it just cannot be ignored.

 

[Steve999]

I’m not really seeing the definition of or analogy to placebo holding up here with the McGurk guy. So I double-checked myself. I suppose the point is arguable though. I am actually even more convinced we way over-use the concept of placebo here in general. The more you research, the more strained the analogy looks.

Placebo

For other uses, see Placebo (disambiguation).

Placebos are typically inert tablets, such as sugar pills
A placebo (/pləˈsiːboʊ/ plə-SEE-boh) is a substance or treatment of no intended therapeutic value.[1]Common placebos include inert tablets (like sugar pills), inert injections (like saline), sham surgery,[2]and other procedures.[3]

In drug testing and medical research, a placebo can be made to resemble an active medication or therapy so that it functions as a control; this is to prevent the recipient(s) or others from knowing (with their consent) whether a treatment is active or inactive, as expectations about efficacy can influence results.[4][5] In a clinical trial any change in the placebo arm is known as the placebo response, and the difference between this and the result of no treatment is the placebo effect.[6]

A placebo may be given to a person in a clinical context in order to deceive the recipient into thinking that it is an active treatment. The use of placebos as treatment in clinical medicine is ethically problematic as it introduces deception and dishonesty into the doctor–patient relationship.[7] Placebos have no impact on disease itself; they can only affect the person's perception of their own condition.[8]

Historically, an influential 1955 study entitled The Powerful Placebo established the idea that placebo effects were clinically important,[9]and were a result of the brain's role in physical health, but a 1997 review of the study found "no evidence [...] of any placebo effect in any of the studies cited".[10] Subsequent research has found that placebos are not a useful means of therapy.[11]