Considering the repeatability of the HD800 and HD800S measurements at Innerfidelity, how similar in subbass frequency response the HD800 and HD800S are and that the Innerfidelity has lower distortion measurements at subbass frequencies of headphones like the Audeze LCD2 none of this would be relevant to the discrepancy.
I think people are overestimating how good your equipment needs to be do these bass measurements. As long as the environment is quiet, you repeat test for consistency and you measure at a high enough SPL you will be ok with subbass distortion measurements at -40dB (1%). People are perfectly capable of pulling out <1% distortion measurements with their DIY foam flat baffle setups and shown as much consistently across the <8Khz frequency range. If they couldn't the noise would be obvious in the measurements.
The discrepancy between FFT and THD+N measurements I think is going to be the issue. Innerfidelity has three different HD800 measurements taken at different times and multiple HD800S measurements all showing consistently the same thing. I think people are grasping at straws if they think it's a simple error considering how long Tyll has done this for. We are after all not talking about one outlier here but a consistent pattern.
It's not as simple as
thinking you're in a quiet-enough room to measure low levels of distortion. If it was that easy, why bother to isolate at all? Look again at Dan Foley's article in audioXpress:
Test and Measurement: “I Can Hear It. Why Can’t I Measure It?”
Look at the NC (noise criterion) curves, in which the measured sound pressure at each octave band is plotted. In a typical suburban home, noise levels of 30 dBA to 38 dBA can be expected, with a noise spectrum corresponding to the NC30 curve. You can see, then, if you're measuring THD -- and especially if you're measuring THD+N (total harmonic distortion plus noise) -- isolation is critically important, especially at the low levels we're examining here. Notice the shape of the NC curves as you move lower in frequency, and also keep in mind that plot only goes down to 63 Hz.
"No, but it's quiet where I am," you might say. You might think that, because perhaps you live in a yurt in the middle of the desert, far away from any civilization -- and/or it might
seem that way because our ears follow
A-weighting curves, so we don't tend to hear the low frequency energy in our environments at those levels. Microphones, however,
will "hear" it.
Again, if it didn't matter, why bother isolating? Why would companies like Sennheiser, Shure, Sony, Audeze, Focal, and other manufacturers bother to isolate? I just received an email from OSSIC today updating Kickstarter participants on their shipping progress. In the email was this photo (which looks to me like a production test rig):
Why would Microsoft need the quietest place on earth? Why would
Roush's NVH (noise, vibration, and harshness) labs go through the expense of building hemi-anechoic chambers large enough to drive trucks into? If you want to do acoustical measurements, isolation
does matter if you want to work toward eliminating external influence on the device under test and the measurement system.
This is why we went to considerable expense and trouble to have a lab-grade isolation enclosure built, for as much acoustic and vibration isolation as we could reasonably afford and fit into our office. While I've seen and experienced better isolation at several labs and manufactories than we can achieve here, I don't personally know of anyone doing consumer-facing headphone measurements that has gone to the lengths we have to isolate the headphones being measured. When I want to measure the harmonic distortion of these headphones, I want to do the best I can to make sure I'm only measuring
harmonic distortion.
As for the
DIY measurement rigs, I'm not going to venture to explain their results. While they obviously have a lot in common
when you look at them (since they're mostly guided by the same sets of instructions), their results are not as compelling or authoritative to me, certainly when it comes to distortion measurements, for the reasons I've discussed in this thread already.
If the precision was there in the DIY-type rigs, acoustical engineers the world over would save themselves the grief of trying to secure more budget from management (an age-old engineering ritual) to expand or improve their labs. I think it's very cool that an increasing number of enthusiasts are interested in audio measurements, and I think it's fun to share results and findings with friends. However, when you're going to post conclusions based on those measurements to thousands (even millions) of people, I do think we should expect and encourage greater precision.
Here's an example of what I'm talking about, this time with a completely different kind of audio measurement -- not acoustical, but this one measuring audio
electronics (a digital-to-analog converter in this case):
An inexpensive (sub-$100) USB DAC was released. Here's a measurement of it that was posted online by an enthusiast:
This was posted for many thousands to see. It looks fine at first blush. Here's what the poster said about it:
Noise floor is higher here because I'm using a balanced probe with a /10 divider (less spurious garbage, but at the expense of moving the noise floor up 10db). I wouldn't worry about it because really, anything under 100db from the main signal... yeah, only John Atkinson cares to interpret stuff like that.
Now even if you agree with that sentiment, let's take a look to see if the DAC's noise floor is in fact only 10 dB lower:
The pink trace is the one previously posted and overlaid on this plot, and the blue trace is the same measurement done here on our APx555. You can see that DAC's noise floor is not only 10 dB lower than the pink plot, it's actually closer to
35 dB lower through most of the audio band in this measurement
. Is that pink measurement (or even a hypothetical one only 10 dB below that) a fair representation of the DAC's noise floor? I don't think so.
Now let's get back to acoustical measurements and the HD800 and HD800S:
Remember, the belief that Sennheiser is deliberately engineering 2nd order harmonic distortion into the HD800S started with one measurement in particular --
an FFT that showed the HD800 as having higher 3rd order than 2nd order distortion with a 40 Hz sine as the stimulus signal. Due to this, the difference in H2 between the two measured headphones (using this stimulus signal) was shown to be
20 dB -- and it's from that significant difference that this theory originated,
and then propagated for the past couple of years.
At the ALMA conference, I talked to a few veteran acoustical engineers about this very subject (one of whom is a headphone audio engineer who has designed one of the lowest distortion headphone products I've yet measured), and another at CES (another experienced headphone engineer), none of whom work for Sennheiser. All of them said that they found it
extremely unlikely that Sennheiser would do such a thing with this type of headphone (which is also consistent with feedback from Sennheiser).
This is why we focused more on the Sennheiser
HD800 measurements. We had two used, older units, and later received three brand new units.
None exhibited the higher H3 (versus H2) with a 40 Hz sine stimulus --
all showed substantially higher H2 than H3. Again,
the original FFT measurement showing higher H3 (still the only one I've seen to show that) was the primary reason
so many people have assumed this theory to be true for nearly two years, which is why we performed the same test so many times.
We also ran the same test on an HD800 and HD800S at the ALMA conference last week, one with Audio Precision and other with Brüel & Kjær. I placed and adjusted the headphones in the Audio Precision measurements. However, at Brüel & Kjær, I simply placed the headphones on their new 5128 HATS (head and torso simulator), and the gentleman there took over for the final seatings and measurements. Lest you wonder about his qualifications, the man at Brüel & Kjær's exhibit who did this also helped set up and monitor the measurement system that confirmed Microsoft's anechoic chamber as the record-setting
quietest place on earth -- I suspect, then, he's qualified to measure a headphone. Neither AP's nor B&K's measurements showed higher H3 than H2 with the HD800 (or HD800S), both being consistent with what we'd shown before that, and what we've shown since.
Finally, then, here are our HD800 and HD800S THD plots for all the new units (one HD800S and three HD800's). Unless putting these headphones on a human is part of Sennheiser's manufacturing process, none of these headphones had been worn by anyone before the measurements were performed.
The first graph is total harmonic distortion (THD). The second graph is distortion product ratio (H2). In each graph, there are four measurements per headphone. The gray lines are HD800's (serial numbers 49282, 49759, 49760), and the black plots are the HD800S (serial number 13134).
Here are our FFT measurements, posted in the order they appeared in this thread:
(Above) Sennheiser HD800 S/N 00342
(Above) Sennheiser HD800 S/N 00342 and Sennheiser HD800S S/N 00205, Seating 1
(Above) Sennheiser HD800 S/N 00342 and Sennheiser HD800S S/N 00205, Seating 2
(Above) HD800S (S/N 13134) and HD800 (S/N 00279):
80 Hz (H2) Δ = 3.331 dBSPL (HD800S > HD800)
120 Hz (H3) Δ = 0.598 dBSPL (HD800 > HD800S)
(Above) HD800S (S/N 13134) versus HD800 (S/N 00342):
80 Hz (H2) Δ = 1.920 dBSPL (HD800 > HD800S)
120 Hz (H3) Δ = 3.611 dBSPL (HD800S > HD800)
UPDATE: Here are three new FFT's (measured on 2018-01-19) showing the brand new HD800's and HD800S. These are overlays of the headphones being driven at both 90 dBFS and 105 dBFS (at 40 Hz). For the sake of this discussion, the most salient point is that all three of the HD800's (as well as the HD800S) show substantially higher H2 than H3 at both 90 dBSPL (40 Hz) and 105 dBSPL (40 Hz).
Again, at ALMA International's annual conference a couple of weeks ago, I visited the exhibits of Brüel & Kjær (B&K) and Audio Precision (AP) to measure HD800 (S/N 00342) and HD800S (S/N 13134) using their headphone test fixtures. Again, these measurements are consistent with our findings so far, and do not support the original theory (that Sennheiser deliberately added H2 distortion to the HD800S).
The following are the measurements taken by Brüel & Kjær at the conference. I placed the headphones on the head, and Vince Rey from Brüel & Kjær did the final positioning and conducted the measurements using their latest HATS (head and torso simulator) and their PULSE analysis software. Here are the results:
Because it was in an exhibit hall, these B&K measurements were done at 100 dB (40 Hz sine).
Here's the measurement of the same headphones at the Audio Precision exhibit, which was performed by AP's Ryan O'Connor and me, also at 100 dBSPL:
UPDATE: We also ran FFT's on the GRAS KEMAR 45BB-12, as some were curious about using a completely different measurement fixture in our system, and what the results would be from that. Keep in mind that in addition to being a different type of fixture, the 45BB-12 also has unique ear simulators within it called the GRAS 43BB low-noise ear simulators. The GRAS 43BB simulators are highly sensitive, very low-noise, and extend the lower dynamic range below the threshold of human hearing. You'll see, then, that the noise floor is lower through much the audio band than our previous FFT's, which is why the floor will look different than our previous such plots.
Also keep in mind that the GRAS 45CA (because of its flat "cheeks") can (as I said
in this video about headphone measurements) result in more measured bass than we'd get with KEMAR with some headphones, and the HD800 / HD800S are among those headphones this is true with. As a result, to achieve a certain bass frequency dBSPL requires substantially more output with KEMAR if you're within the KEMAR's bass roll-off zone with these headphones (and the 39 Hz sine stimulus we used for this test
is in that zone). So, for example, I only needed >175 mVrms to achieve 90 dBSPL (40 Hz) with the 45CA, I had to go to >335 mVrms to reach 90 dBSPL (39 Hz) on KEMAR.
This next set consists of four FFT measurements, each comparing the HD800 (S/N 49759) and HD800S (S/N 13134): 39 Hz @ 90 dBSPL, 39 Hz @ 100 dBSPL, 78 Hz @ 90 dBSPL, and 78 Hz @ 100 dBSPL. Here they are:
To summarize the measurements: We now have a variety of measurements that include two different HD800S's (one of which is brand new, and one of which is early production, neither of which has ever been modified), and five different stock HD800's (also never modified, which may be relevant), three of which are brand new, one of which is early production, and one of which is
early production.
At our office, we have measured with two completely different kinds of test fixtures (GRAS 45CA and GRAS KEMAR) with two completely different kinds of ear simulators (GRAS RA0401 and GRAS 43BB), both sets of which have been calibrated.
We also have included FFT measurements of HD800 S/N 00342 and HD800S S/N 13134 done by others. Ryan O'Connor from Audio Precision and I did one set at ALMA's annual conference using the Audio Precision AECM206 test fixture. At that same event, Vince Rey from Brüel & Kjær did the other set on B&K's new 5128 HATS. (I had to let Vince handle this one, as I had no experience using B&K's PULSE measurement software.)
Most notably, none of the 40 Hz (or 39 Hz) FFT's -- whether by us, B&K, or AP (the latter two at ALMA) -- show HD800 H3 > H2, as
the original FFT in question does.
Again, at the ALMA conference and CES, I talked to several veteran acoustical engineers (including headphone engineers) about this very subject (none of whom work for Sennheiser). And, again, all of them said that they found it
extremely unlikely that Sennheiser would do such a thing with this type of headphone (which is also consistent with feedback from Sennheiser).
It seems pretty clear to me at this point -- based on the measurements (by us and others), based on feedback from Sennheiser, and based on feedback from several acoustical engineers in the audio industry not in the employ of Sennheiser -- that the assumption that Sennheiser did deliberately
"add second harmonic distortion to the [H800S's] low notes to 'bloom' them up a bit" is not accurate. While I do personally feel (in listening) that the HD800S does have an overall fuller tone than the HD800 does, it would seem that it would be due to methods of tuning other than this, with a resulting headphone that in my opinion sounds no less precise.
Except where otherwise stated, the measurements in this post were made with the following system:
NOTE 2018-01-17 21:20 EST: I made a mistake on the DAC FFT plot. My FFT length and sample rate were not the same as the other FFT, which resulted in a noise floor on my plot on the graph I posted that's about 10 dB lower than it should be. I'm posting a corrected FFT shortly (and will also correct the values stated when I do).
NOTE 2018-01-17 21:50 EST: I posted the corrected DAC measurement above with the corrected FFT length and sample rate, and also corrected the values in the accompanying paragraph. (
Click here to see the original graph that, for the purpose of a fair comparison, was not configured correctly.) In the future, for that type of measurement, we may use power spectral density (PSD) or amplitude spectral density (ASD) -- which we'll explain later, in the proper context -- as those will allow us to present noise data in an FFT spectrum with amplitude values independent of FFT length. Sorry about the error, guys.
NOTE 2018-01-22 04:18 EST: FFT's -- including ones by others with two other fixtures -- have been added to this post (above).
NOTE 2018-01-23 17:18 EST: We added FFT measurements from the GRAS KEMAR 45BB-12 (
click here to see these above).