[purrin]

I've discussed with peers whether this post belongs in the sound science forum or here, and I decided to post it here for the following reasons:
 

1. This study is really about the subjective experience first. And how objective measurements back up those subjective experiences rather than the reverse.
2. This study does not subscribe to the fringes of the objectivist or subjectivist belief systems
3. Frequency response and CSD waterfall plots are commonplace in the speaker world. Respected journalists like John Atkinson have been using these kinds of measurements for decades. Such measurements can hardly be considered esoteric science in today's Information Age world.
4. Only "popular" headphones were selected for use in the study. The measurements shown pertain directly to these headphones.

 
That being said, moderators should move this post to the sound science forum if they feel it's more appropriate.
 

---

 
[size=medium]Do Objective Headphone Measurements Correlate to the Audiophile's Subjective Experience?

There are those who feel that any attempt to scientifically correlate headphone measurements to the complexities of the subjective experience is futile. This post is not for those people, as they have already made up their minds.

There should be no tension between subjective and objective evaluation. Those who feel this way are likely on the fringes of their belief systems, or have a misunderstanding of the limitations or unreasonable expectations of objective measurements. Others may feel that measurements are better suited for engineers and too complex for the layman to understand. This is a good argument, but one that I’ve seen fall apart once the layman is gradually exposed to more data and becomes familiar enough with measurement patterns, wiggly lines so to speak, to correlate them with actual experience.

Some people may wholly discount measurements if they feel slighted because their favorite headphones do not measure perfectly, i.e. in a flat straight line. This is yet another misunderstanding of how measurements work. While there are those who prefer a technically perfect measurement, most people prefer a little “sugar and spice”. The flat straight line is simply an artificial reference point. Those who have developed specific tastes will likely prefer certain kinds measurement patterns. Oppositely, they may also be adverse to certain kinds of measurement patterns.

Finally, there are those who have unreasonable expectations, thinking that because measurements cannot tell the entire story, they are of little use. It’s true that frequency response, cumulative spectral decays, and non-linear distortion measurements do not tell the whole story. But they do say a lot and this assertion is best illustrated.

While there’s no guarantee that a good measuring headphone will sound good to every person, there is a strong correlation between good measurements and good sound and bad measurements to bad sound. Of course there is the issue of how to interpret measurements, but we will leave that up to the reader, and in a follow up report.

A pool of thirty headphone hobbyists were asked to rate on a five point scale seven headphones. They were not told of the purpose of this study. The headphones in the study were selected because of their popularity in order to increase the likelihood that the hobbyists were familiar with them, and hence could provide a rating. Each hobbyist was asked to rate the headphone as follows without any consideration to price:[/size]
 

• [size=medium]It’s a good sounding headphone[/size]
• [size=medium]It’s good, but with some reservations about certain aspects[/size]
• [size=medium]It’s OK. Nothing special.[/size]
• [size=medium]I don’t like it, but I can see why others do[/size]
• [size=medium]It’s a bad sounding headphone[/size]

[size=medium]For now, the models of these seven headphones will not be identified. It’s probable that many readers will be able to guess.




Headphone A
Responses: 14
Average Rating: 4.07 (1 = good, 5 = bad)
Standard Deviation 1.14
Comments:[/size]

1. [size=medium]glaring, sibilant upper-end. I do like other facets of it[/size]
2. [size=medium]heard it once at a meet..walked away thinking "people actually paid nearly XX for this?? Maybe they will buy my dog poop for 100 bucks per pound as a statement piece"[/size]
3. [size=medium]the XX was your sacrifice to the Headphone Gods to give you a good YY[/size]

 
[size=small]

[/size]



As a side note, to illustrate why measurements are important - they keep us honest. A reviewer once said of this headphone:

The XX’s are the flattest sounding, most extended headphones I have ever heard. No headphone I know of will equal them from top to bottom for evenness of performance... I have never heard or thought I would hear headphones this extended and flat in both directions in my lifetime, but here they are.
 
 
 

Headphone B
Responses: 28
Average Rating: 1.78 (1 = good, 5 = bad)
Standard Deviation 0.83
Comments:
 

1. [size=medium]i wanted to love this HP when i owned it years ago, but ultimately found it a bit dull and lifeless.[/size]
2. [size=medium]and despite the XX’s flaws, I still like them quite a bit. Switching to them coming from sterile sounding cans makes music fun to listen to again.[/size]
3. [size=medium]I think it's not the best headphone in every aspect but still a very capable headphone with a pleasant sound signature.[/size]

 
[size=small]

[/size]



 
 

Headphone C
Responses: 23
Average Rating: 2.00 (1 = good, 5 = bad)
Standard Deviation 0.74
Comments:
 

1. [size=medium]very airy and clear, wide soundstage,[/size]
2. [size=medium]It's a damn good-sounding headphone. It's not perfect, but as AA said, EQ works well to correct its flaws.[/size]
3. [size=medium]I can appreciate it from a technical standpoint but don't really want it near me. Like a professional weightlifter.[/size]

 
[size=small]

[/size]



 
 

Headphone D
Responses: 20
Average Rating: 2.95 (1 = good, 5 = bad)
Standard Deviation 0.94
Comments:
 

1. [size=medium]love the instrument separation and bass - especially the double bass as i find most recordings or HPs bloat this instrument.  requires EQing and/or careful amp matching [/size]
2. [size=medium]I like the clarity and separation from it, and I'm actually okay with the overall FR. Don't like the weird bump in the mids though, and the tonality is definitely artificial.[/size]
3. [size=medium]properly amped, this is a good one. I gave it to a good friend when I decided it's not for me, and he's still happy with it, and I wouldn't give bad headphones as presents [/size]

 
[size=small]

[/size]

 
 
 

Headphone E
Responses: 20
Average Rating: 3.20 (1 = good, 5 = bad)
Standard Deviation 0.89
Comments:
 

1. [size=medium]bad sounding headphone.  Pain = bad.  I think it's about as simple as that. [/size]
2. [size=medium]The XX is like a beautiful person with a disease. I know there may very well be repercussions, but I just don't care to think that far ahead. Plus I know I've been bad, and I need to be punished.[/size]
3. [size=medium]It was my main headphone for years, so I'm quite used to its sound signature. [/size]

 
[size=small]

[/size]





Headphone F
Responses: 16
Average Rating: 4.00 (1 = good, 5 = bad)
Standard Deviation 0.89

Comments:
 

1. [size=medium]I feel they impart a certain body to music, but it's kind of weightless and not planted... kind of like an elephant in zero gravity. Aside from that I found them to be pretty thoroughly "meh."[/size]
2. [size=medium]I bought these after the usual hype from the likes of YY, ZZ, and a few others kept harping on them.  I do not consider them in the same class in the Audeze and HD800.  They are thoroughly middling to my ears.[/size]
3. [size=medium]I've heard several and they were all warm, bright, and metallic.  All at the same time.  Kinda like when my Dremel's cut off wheel shoots a stream of aluminum particles in my face.[/size]

 
[size=small]

[/size]




 

Headphone G
Responses: 17
Average Rating: 1.65 (1 = good, 5 = bad)
Standard Deviation 0.49
Comments:
 

1. [size=medium]I hate to nitpick this headphone because I found it so appealing. For the price I was expecting an ortho succubus that would be everything I wanted it in headphone.[/size]
2. [size=medium]I'd buy it for $500-$600[/size]

 
[size=small]

[/size]

 

I was initially planning on writing an analysis, but I eventually decided against it. I thought it would be more entertaining for readers to figure it out on their own.
 
Hopefully it should be rather clear by now that headphones with less high frequency anomalies and cleaner waterfall plots tend to sound better to most people. Ultimately readers should decide for themselves if these measurements mean anything to them personally. It should noted that a number of participants stated that they did could not vote for rating #1 ([size=medium]It’s a good sounding headphone)[/size] because they did not consider any headphone, no matter how good, to to absolutely free of any issues.

 

[KG Jag]

Do you mean choice "A" instead of "#1"--or are you referring to your 5 point rating scale?

 

[ocswing]

Quote:

Do you mean choice "A" instead of "#1"--or are you referring to your 5 point rating scale?

 
Pretty sure he is referring to the rating scale. "It should be noted that a number of participants stated that they could not vote for choice #1 "[size=medium]It’s a good sounding headphone" [/size]because they did not consider any headphone, no matter how good, to be absolutely free of any issues.

 

[colgatetotal]

Being an audiology student, there are a few things I take into account when looking at these graphs.
 
- Listening level
The fletcher-munson curve is a reality. Some headphones sound better at different listening levels than others. Your survey does not detail the tracks used for listening, nor does it detail the average dBSPL of all the tracks used. I find that I can raise the volume of headphones with a darker, flatter response to levels far beyond what I'm used to. But what's the point when I never listen at those levels anyway? I find that my most comfortable listening level is around the 65-70dbSPL range, and at that level I would find that a flat response to be lacking sub-bass and extended treble.
 
- Model of the head used for the measurements
The head used when taking these measurements probably assumes a standard adult-sized head. My head is asymmetrical -- the ear canal for my right ear is of a different length than my left, so as a result I cannot wear IEMs without experiencing a balance shift. As the length of the ear canal changes, so too does the typical natural resonance of 2700hz. Fitted hats come in a variety of sizes, so it's also natural that a headphone will tend to certain people and not others due to physical needs. What sounds great to you will not necessarily sound great to a child or teenager.
 
- Why would the microphone used for the measurements be perfect?
You can calibrate a microphone all you want. But the microphone at perfect calibration still wouldn't have a super-straight flat response. The qualities of the microphone would affect these graphs tremendously.
 
Ultimately, while the CSD and frequency response graphs are useful to some extent, they lack certain pieces of information that allow me (and a few other people I know) to make a conclusion. I feel that people are now taking a look at these graphs and are taking them out of context, and that they're mainly used for "affirmative justification" -- that is, they have a biased notion going into the demo, and then they say "see? I was right all along" when shown the CSD/frequency response graph.

 

[hans030390]

Aren't most (or all) of these participants members on your forum? I would argue the forum draws in people that are more concerned about objective measurements than most on HF, which, in turn, would lead to the correlation of better measurement = better subjective experience. In that regard, this study could be considered somewhat biased.
 
Don't get me wrong, as I love what you do with your forum and check there daily! I don't mean to undermine this study either. Thanks for your contributions!

 

[purrin]

This subjectivist vs. objective concern is absolute nonsense started by nwavguy and his minions who are on a holy crusade. There is no battle between the forces of good and evil. Besides, the participants were only asked about their subjective opinion of the headphone, and were not told of the the purpose of the study. If you look at the standard deviations, you will note that there was a good amount of variance in opinions for some of the headphones.

 

[TMRaven]

B looks like an HD600/650 and C an HD800.  B wins over C! 
 
Headphone G looks like one of the Hifiman HE-500 or 400 but I could be wrong.

 

[Curly21029]

I pay attention to measurements and do find them to be valuable... to an extent.  There's one statement that I read far too often in reviews that utilize graphs: "they sound better then they measure."  As detailed by colgatetotal above, there are numerous variables that must be taken into account where measurements are concerned and it would be foolish to be solely reliant on them when formulating an opinion of performance.  Another consideration is the objectivity of the person taking the measurement. (the human factor most definitely DOES come into play here)  Therefore, I feel that it's most beneficial to hold charts/graphs not as a means of proving "honesty," but rather as a point of reference to take into account that's akin to any other opinion.
 
In addition, purrin, you may as well list the model with the corresponding measurements.  Since the above graphs have already been published elsewhere, it's a given that they will be identified in short order.

 

[hans030390]

Quote:

This subjectivist vs. objective concern is absolute nonsense started by nwavguy and his minions who are on a holy crusade. There is no battle between the forces of good and evil. Besides, the participants were only asked about their subjective opinion of the headphone, and were not told of the the purpose of the study. If you look at the standard deviations, you will note that there was a good amount of variance in opinions for some of the headphones.

 
That's true, but I think it would have been more accurate to pull info from many people on HF as well. The communities between HF and your forum are quite different, which could potentially skew the results compared to polling a more broad category of people.

 

[TMRaven]

Not entirely unbiased either unless they were blindfolded and told to rate the different headphones being put on their ears.  But of course they could still cary some bias over based on familiar fit alone (this thing weighs a ton it must be LCD-2!  Guess I'll rate it this way.) 

 

[purrin]

I would certainly hope that people wouldn't to solely rely on measurements when formulating an opinion. That would be dumb in my opinion. I would hope that people find measurements useful. 
 
For example, I wished measurements such as these were available when I purchased the K701 many years ago. It would have saved me a lot of time: wondering whether my ears were defective; hearing anomalies which weren't there; or burning in the darn thing for 400+ hours. Measurements of the K701 and (one or two other headphones that I was familiar) with would have certainly saved me a lot of time and money (vs. listening people on HF insisting they were neutral.)
 
This is what I meant by "keeping us honest." Headphone measurements (used in relative comparison) are more precise and less subject to the whims of varied opinions of people or people's moods. Not that the reviewer was being dishonest or was "on-the-take". But then again, I would seriously wonder about that statement in regards to the ED8. At the very least, in my opinion, it would imply incompetence as an audio reviewer.

 

[Questhate]

Quote:

 
For example, I wished measurements such as these were available when I purchased the K701 many years ago. It would have saved me a lot of time: wondering whether my ears were defective; hearing anomalies which weren't there; or burning in the darn thing for 400+ hours. Measurements of the K701 and (one or two other headphones that I was familiar) with would have certainly saved me a lot of time and money (vs. listening people on HF insisting they were neutral.)

 
Yep... and not to mention wasting money upgrading other parts of your chain to rule out gear synergy. One look at the upward tilt of the FR on the K701 will let you know that the headphone is bass-lite and NOT because "your amp isn't good enough". 
 
I like that you emphasized in this article that a "perfect flat line" is not necessarily the goal. I personally don't look at them as proof that "my headphones are objectively better than yours", but rather how certain attributes of headphone sound that I prefer tend to manifest themselves in measurements -- and letting that help guide how I spend my hard-earned money. Just another significant data point for reference.
 
And as far as sample group bias, if you polled seasoned HF members who have heard most of the headphones posted here, would the results be that much different? I'm not saying that the bias doesn't exist (which it does), but may be slightly overblown. The only results that I can see differ from these are regarding the T1, and possibly a greater standard deviation with the ED8. Both of those headphones have QC issues with unit-to-unit variation, so factor that in as well. 

 

[purrin]

Quote:

 
That's true, but I think it would have been more accurate to pull info from many people on HF as well. The communities between HF and your forum are quite different, which could potentially skew the results compared to polling a more broad category of people.

 
I was considering having a more open poll, but in the end, I decided not to use the "average" or "novice" Head-Fi'er for the following reasons:
 

• The novice Head-Fi'er may not have had exposure to many of the headphones in the poll. For example, asking someone who has never driven a Corvette or a 911 Turbo to rate the cars is not a good idea.
• The novice Head-Fi'er may not have had exposure to a sufficiently wide variety of headphones to have a good grasp of good and bad headphones in a relative sense. For example, if you own Beats and an SR80, it's possible the SR80s are the greatest headphone in the world to you.
• The participants were well aware that they were not to comment on a headphone unless they felt confident about it. This could not be guaranteed from the average Head-Fi crowd. Even in this thread, we see people cite their "credentials" and speak of things they have little practical experience with, e.g. "calibrated measurement microphones affecting the results of the graphs tremendously." or shift the topic / intent (which already is very tightly constrained) of the study in order to at least partially discredit it.
• Finally, all of the participants are still HF active members.

 
I fully admit the study is biased. All studies are. If you feel it's too "elitist", that's fine. I would be more than willing to rename the post to "Do Objective Headphone Measurements Correlate To the Advanced Headphone Hobbyist's Subjective Experience."
 
Will post the identities of the headphones in a bit along with the measurements. Hopefully things will make a bit more sense now.

 

[purrin]

[size=medium]Ultrasone ED8
Responses: 14
Average Rating: 4.07 (1 = good, 5 = bad)
Standard Deviation 1.14
Comments:[/size]

1. [size=medium]glaring, sibilant upper-end. I do like other facets of it[/size]
2. [size=medium]heard it once at a meet..walked away thinking "people actually paid nearly XX for this?? Maybe they will buy my dog poop for 100 bucks per pound as a statement piece"[/size]
3. [size=medium]the ED8 was your sacrifice to the Headphone Gods to give you a good YY[/size]

 
[size=medium]

[/size]



As a side note, to illustrate why measurements are important - they keep us honest. A reviewer once said of this headphone:

The XX’s are the flattest sounding, most extended headphones I have ever heard. No headphone I know of will equal them from top to bottom for evenness of performance... I have never heard or thought I would hear headphones this extended and flat in both directions in my lifetime, but here they are.
 
 
 

HD650
Responses: 28
Average Rating: 1.78 (1 = good, 5 = bad)
Standard Deviation 0.83
Comments:
 

1. [size=medium]i wanted to love this HP when i owned it years ago, but ultimately found it a bit dull and lifeless.[/size]
2. [size=medium]and despite the HD650’s flaws, I still like them quite a bit. Switching to them coming from sterile sounding cans makes music fun to listen to again.[/size]
3. [size=medium]I think it's not the best headphone in every aspect but still a very capable headphone with a pleasant sound signature.[/size]

 
[size=medium]

[/size]



 
 

HD800
Responses: 23
Average Rating: 2.00 (1 = good, 5 = bad)
Standard Deviation 0.74
Comments:
 

1. [size=medium]very airy and clear, wide soundstage,[/size]
2. [size=medium]It's a damn good-sounding headphone. It's not perfect, but as AA said, EQ works well to correct its flaws.[/size]
3. [size=medium]I can appreciate it from a technical standpoint but don't really want it near me. Like a professional weightlifter.[/size]

 
[size=medium]

[/size]



 
 

AKG K701
Responses: 20
Average Rating: 2.95 (1 = good, 5 = bad)
Standard Deviation 0.94
Comments:
 

1. [size=medium]love the instrument separation and bass - especially the double bass as i find most recordings or HPs bloat this instrument.  requires EQing and/or careful amp matching[/size]
2. [size=medium]I like the clarity and separation from it, and I'm actually okay with the overall FR. Don't like the weird bump in the mids though, and the tonality is definitely artificial.[/size]
3. [size=medium]properly amped, this is a good one. I gave it to a good friend when I decided it's not for me, and he's still happy with it, and I wouldn't give bad headphones as presents[/size]

 
[size=medium]

[/size]

 
 
 

Grado RS-1
Responses: 20
Average Rating: 3.20 (1 = good, 5 = bad)
Standard Deviation 0.89
Comments:
 

1. [size=medium]bad sounding headphone.  Pain = bad.  I think it's about as simple as that.[/size]
2. [size=medium]The RS1 is like a beautiful person with a disease. I know there may very well be repercussions, but I just don't care to think that far ahead. Plus I know I've been bad, and I need to be punished.[/size]
3. [size=medium]It was my main headphone for years, so I'm quite used to its sound signature.[/size]

 
[size=medium]

[/size]





Beyer T1
Responses: 16
Average Rating: 4.00 (1 = good, 5 = bad)
Standard Deviation 0.89

Comments:
 

1. [size=medium]I feel they impart a certain body to music, but it's kind of weightless and not planted... kind of like an elephant in zero gravity. Aside from that I found them to be pretty thoroughly "meh."[/size]
2. [size=medium]I bought these after the usual hype from the likes of YY, ZZ, and a few others kept harping on them.  I do not consider them in the same class in the Audeze and HD800.  They are thoroughly middling to my ears.[/size]
3. [size=medium]I've heard several and they were all warm, bright, and metallic.  All at the same time.  Kinda like when my Dremel's cut off wheel shoots a stream of aluminum particles in my face.[/size]

 
[size=medium]

[/size]


 
 

LCD2 revision1
Responses: 17
Average Rating: 1.65 (1 = good, 5 = bad)
Standard Deviation 0.49
Comments:
 

1. [size=medium]I hate to nitpick this headphone because I found it so appealing. For the price I was expecting an ortho succubus that would be everything I wanted it in headphone.[/size]
2. [size=medium]I'd buy it for $500-$600[/size]

 
[size=medium]

[/size]
 

 

[Focker]

 

 ​

I was initially planning on writing an analysis, but I eventually decided against it. I thought it would be more entertaining for readers to figure it out on their own.  
 

 
I think it's great to discuss what you did here, but I really don't think there is anything that can be inferred or concluded based on this study.