Headphone Measurements: Different Setups, Different Results
Oct 2, 2015 at 3:30 AM Post #16 of 45
  All measurements are very, very wide approximations...
 
 
https://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CCUQFjAAahUKEwiy4O-NmKPIAhULfRoKHXeoDik&url=http%3A%2F%2Faudiology-online.co.uk%2Fwp-content%2Fuploads%2F2014%2F07%2FSubjective-variance-of-calibrated-audiometer-systems.pdf&usg=AFQjCNHBo92Z0vON8VimfPZ2fmD0RJP9OQ


how about staying on topic? this link talks about measuring the hearing of people, where the vastly unreliable part is the people. that's not what any of the devices in the OP are for.
 
Oct 2, 2015 at 5:09 AM Post #17 of 45
 
how about staying on topic? this link talks about measuring the hearing of people, where the vastly unreliable part is the people. that's not what any of the devices in the OP are for.


Because you need an argument to continue the discussion lol.
 
"This test was performed on a cohort of university employees and students, to measure
the variation in audiogram data that can be experienced when using different traceably
calibrated audiometer systems"
 
Nov 18, 2015 at 11:43 AM Post #18 of 45
After 20 years of reading, or trying to read the graphs in HiFi News and Stereophile I have come to the 
grand conclusion that they should just be largely ignored. 
 
Frequency response graphs for headphones that were neutral, consistent and accurate 
could be useful, but I'm not sure we currently have that.
 
Good engineers use their ears not graphs, eg Chord.
Plus I always assume every graph produced has an agenda.
 
Nov 18, 2015 at 11:51 AM Post #19 of 45
Tools can't have agendas. Only people. Engineers love tools to help them do their jobs consistently and repeatable.

I think measurements are a wonderful tool into understanding why I'm hearing what I'm hearing, to understand why I like what I like, and to help me look for other headphones and in-ears that fit my preferred sound. But one must also understand the measuring process, the different setups used and the different compensation methodologies used and preferred by the measurer. I honestly do not understand the fear of measurements. Just because some may abuse it or read too much into it, doesn't mean it doesn't provide useful information.
 
Nov 18, 2015 at 8:43 PM Post #20 of 45
I have to agree. in my experience very few people can read a measurement for what it really is. it's certainly not the measurement's fault. just like if I read Japanese and misunderstand the meaning of the sentence, I'll be the one to blame, not Japanese language.
 
I've seen far to many people thinking that a frequency response graph is supposed to say everything, or getting mad because they think their headphone is flat but the graph isn't.....
it's mostly about ignorance and misinterpretation, not about measurement. a measurement offers a conditional truth and never claimed to do more. the conditions are that a given headphone placed in a given position was measured by a given measurement gear.
you can have the same headphone model and have several db of differences from manufacturing difficulties, another few DB differences from the way you wear it or how old are the pads. and of course the measurement gear and the calibration applied are mighty important in knowing what we can and cannot conclude about the measurement.
 
a measurement is a piece of information, never the all story. it's people's fault if they don't understand what to do of it. if there were no measurements, I would have left the audio hobby long ago. it's often the last thread of sanity left when discussing a FOTM device.
 
Dec 26, 2015 at 5:59 AM Post #22 of 45
YMMV is definitely the ruling paradigm here.....
wink_face.gif

 
May 26, 2017 at 2:16 AM Post #24 of 45
Subscribed. Hopefully we can have a more affordable and reasonably useful approach for IEM and headphone measurement, so more can do their own measurement for a more balance discussions between subjective impressions and sharing measurement results.
 
Jun 27, 2017 at 10:13 PM Post #25 of 45
Hi all,

Wanted to toss my $0.02 into the mix...

For those that know who I am, then you know I have a bit of a reputation in the industry (good). For those that don't, I can guarantee you've heard my work if you've ever listened to live or recorded music. From recording microphones to studio monitors to PA systems to consumer systems, I've designed audio systems and transducers for pretty much all the big players - and done so many, many times (not to mention lots of headphones and IEMs along the way).

In my experience, measurement correlation between systems is hard to first establish and even more difficult to maintain. Datasets within a measurement system/location/team can be fairly consistent over time as long as the equipment is rigorously maintained, processes are slavishly followed, and the team cares greatly about consistency. Otherwise - all bets are off.

In production of audio systems, we use "golden samples" - we use a very small number of selected reference units that are deemed as "ideal". A day's production usually starts with the online production test systems measuring the golden sample, then tolerances are set accordingly to that measurement. All production must pass within the tolerance window, and the few (typically one or two a week) that essentially have no deviation from the golden sample are culled out and reserved as future, replacement "golden samples" (the tolerance can be discussed later, but suffice to say it is probably an order of magnitude larger than most HeadFi'ers would expect).

In other words, we use physical representations to calibrate against, rather than abstract numbers and concepts. It all comes down to how measurements can change from not just system to system or operator to operator but day to day. Temperature and humidity can affect measurements in significant manners. Environmental noise can - and definitely will! - corrupt measurements. Different mountings of DUTs (Device Under Test) by operators will affect measurements.

In essence, after installing literally hundreds (perhaps over a thousand) acoustic test systems, at dozens of factories in dozens of countries, I can confidently say that expecting consistency between two or more systems is a fool's errand. Won't happen.

Measurements are a great way to confirm you are getting what you expect, and to document where you are. And they are relevant within the same local world (equipment, team, environment). They can be used to guide design of product by a team, a team that is familiar and experienced with what measurement X really means in terms of what they are designing.

So with that, measurement correlation between different teams is never really expected, nor should it be. In fact, I start to get nervous if things line up too well! Great consistency tells me either the wrong settings are being used (we're not looking at a fine enough level of detail - we're oversmoothing/over-interpolating), or some fudging is going on to make things look closer than they really are.

All that said - don't expect measurements from one person to closely track that of another. Look at how products vary inside each measurement set, and assume the variances are at least relative - that is much more instructive. If one system is hot or cold in the treble, it will be consistently hot or cold, and you'll see that as you compare larger datasets between different systems. That is what we should pay attention to, rather than a few cherry-picked comparisons.

Now back to continue the designing a new headphone, over here in Foster Tokyo...
 
Jun 28, 2017 at 11:48 AM Post #26 of 45
Hi all,

Wanted to toss my $0.02 into the mix...

For those that know who I am, then you know I have a bit of a reputation in the industry (good). For those that don't, I can guarantee you've heard my work if you've ever listened to live or recorded music. From recording microphones to studio monitors to PA systems to consumer systems, I've designed audio systems and transducers for pretty much all the big players - and done so many, many times (not to mention lots of headphones and IEMs along the way).

In my experience, measurement correlation between systems is hard to first establish and even more difficult to maintain. Datasets within a measurement system/location/team can be fairly consistent over time as long as the equipment is rigorously maintained, processes are slavishly followed, and the team cares greatly about consistency. Otherwise - all bets are off.

In production of audio systems, we use "golden samples" - we use a very small number of selected reference units that are deemed as "ideal". A day's production usually starts with the online production test systems measuring the golden sample, then tolerances are set accordingly to that measurement. All production must pass within the tolerance window, and the few (typically one or two a week) that essentially have no deviation from the golden sample are culled out and reserved as future, replacement "golden samples" (the tolerance can be discussed later, but suffice to say it is probably an order of magnitude larger than most HeadFi'ers would expect).

In other words, we use physical representations to calibrate against, rather than abstract numbers and concepts. It all comes down to how measurements can change from not just system to system or operator to operator but day to day. Temperature and humidity can affect measurements in significant manners. Environmental noise can - and definitely will! - corrupt measurements. Different mountings of DUTs (Device Under Test) by operators will affect measurements.

In essence, after installing literally hundreds (perhaps over a thousand) acoustic test systems, at dozens of factories in dozens of countries, I can confidently say that expecting consistency between two or more systems is a fool's errand. Won't happen.

Measurements are a great way to confirm you are getting what you expect, and to document where you are. And they are relevant within the same local world (equipment, team, environment). They can be used to guide design of product by a team, a team that is familiar and experienced with what measurement X really means in terms of what they are designing.

So with that, measurement correlation between different teams is never really expected, nor should it be. In fact, I start to get nervous if things line up too well! Great consistency tells me either the wrong settings are being used (we're not looking at a fine enough level of detail - we're oversmoothing/over-interpolating), or some fudging is going on to make things look closer than they really are.

All that said - don't expect measurements from one person to closely track that of another. Look at how products vary inside each measurement set, and assume the variances are at least relative - that is much more instructive. If one system is hot or cold in the treble, it will be consistently hot or cold, and you'll see that as you compare larger datasets between different systems. That is what we should pay attention to, rather than a few cherry-picked comparisons.

Now back to continue the designing a new headphone, over here in Foster Tokyo...

Hi Dan,
Thanks for that. Enjoyed the information, perspective, and inside view into the process.
Hoping that you can offer more insight into the design development process when you have time.
For now, I'm looking forward to the purchase of the Be.
 
Jul 23, 2017 at 2:30 PM Post #28 of 45
So I decided to do something...:wink:

I grabbed my 45CA, took some measurements, and built a decent facsimile in CAD. It can be downloaded from here. You can 3D print it on pretty much any 3D printer (I use a Lulzbot Mini at home). You need 6 qty 6mm x 25mm bolt and nut sets, and 6 qty x 20mm bolt and nut sets.

This is designed to use my 711-compliant couplers, but could be modified for other 711-compliant (IEC 60318-4) microphone couplers and mics. Using a non-711 compliant mic will compromise accuracy above ~6 kHz. But this will work with most 711-compliant couplers.

It's available in STL (if you want to just print it out), Alibre (the 3D parametric CAD I use - highly recommended), Solidworks (which Alibre can natively read and write), and STEP (for those using something else). So you can import and edit and change as you will. It is released gratis! If you re-release or share, I'd appreciate a hat tip back...

This is designed around an 85 percentile head. Proper width, angle of the headphone sides, and height of the headband. Not adjustable as-is, but could be made so.

Hopefully releasing a simple, 3D printable, adaptable headphone mount based on a multi-thousand dollar piece of equipment will encourage those with lots of different home-brew systems to try something a bit more consistent with all other measurements that are being made. Using this platform, and doing an adapter for a Pocket CLIO (quite easy to do), you would have something that anyone with access to a printer could build up as a test fixture for under $800 complete (with $700 of that being the CLIO). That would yield a very consistent measurement platform for hobbyists, and still well within the budget of what most hobbyists easily spend in 6 months of different gear. If you're serious about the hobby, spending a few hundred towards good measurement gear (and the CLIO is just that) is quite reasonable.

EDIT: One reason I wanted to post this, is because of the constant push for "science! Measurements!" Well, the scientific method is REALLY, REALLY good, and so is the sharing of data. But sharing data WITHOUT the ability to reproduce the data is interesting, but it's not really hard-core science. You want to publish your data AND the way you gathered the data, so others can reproduce and replicate your results. Otherwise you have created some interesting observations, but not really hard-core data.

Hopefully the scientific-end of the hobbyist community can move closer to a reproducible standard, one that would allow a wider variety of measurements to be made and improve the quality and consistency of all the data.

Dan
 
Last edited:
Dec 22, 2017 at 5:24 PM Post #29 of 45
anybody got to play with the MiniDSP EARS already?
so far my impressions(after a few hours, so nothing absolute):
- the DF compensation seemed mostly fine on my HD650. the one for IEMs not so fine IMO, but then again any small placement change creates a cataclysmic event, it seems even worst than with basic couplers or tubes. but admittedly I only tried a couple IEMs so far and as we all know with some practice things usually improve significantly.
not sure if the RAWs are usable or calibrated in any way. they don't look great to me so far, but personally I only really care about variations between measurements, so I don't mind.

- the outer ear is the same eight as mine, not soft like a real ear but still fairly soft(big chunk of soft silicone). the inside space where most IEM shells are resting is too small, even my mother doesn't have ears that small. I can't place a SE215 correctly because the shell can't fit. the anti tragus and anti helix area just aren't far enough in the back for the shell of the IEM to fit in the cavity area. so good luck fitting all the TOTL IEMs with their 57drivers/side and massive shells.

- it's driver-less (default windoz USB audio drivers), on my win7 I get the mic boost stuck at +24dB in the properties and spent the day trying to deal with that without success. it's likely to be the value applied as I can only use the 0 gain setting on the EARS(little switches on the device to adjust the gain in 6dB increments). default is 18dB and I couldn't use it because I clipped the signal with half my IEMs when I set them at 90dB SPL@1khz.(calibration in REW using ER4SR's certificate and a multimeter). so I hope there is an easy fix to that.
if I'm bored while letting my liver rest after this WE, I will try to run a few measurements with IMM6, Veritas, my Frankenstein combo and this to show the variations I get and how fun:rage: resonances can be. it should make for a good illustration of this topic, even if it only involves poor people's gears ^_^.
 
Dec 24, 2017 at 11:02 AM Post #30 of 45
got a little free time this morning so I went on with it. here are my cheap amateur measurement rigs in a few examples (not enough to really get a pattern, just enough to make a mess ^_^).
gear used:
-output: Odac/O2
- IEMs: ER4SR(sealed deep insertion), XBA_C10(sealed, can't go deep), Fiio EX1 AKA rebranded Titan1(super mega vented and dynamic driver).
-inputs: that's where it gets fun.
option 1@@@@: MiniDSP EARS (USB)
282A1617.jpg




option 2@@@@: IMM6 made to use on cellphones and tablets.
imm-6_photo_1_2.jpg

I fed mine with a cellphone for the DC voltage, but recorded into a Scarlett 2i2.TBH as far as FR is concerned, I get almost the same results with a cheap 30$ Startech USB ADC, or even with my laptop's mic input.
a silicone tube is added to be able to measure IEMs with a seal while at a more appropriate distance(see the hundred measurements done by @crinacle using this little cheap mic).


option 3 @@@@: EMC8000 stuck into a fake IEC711 coupler, massively DIY as the 2 didn't fit by default. (XLR, needs 48v phantom power).
895.jpg
+
10012510_thumb.jpg
I have the coupler on the right



option 4@@@@: Vibro Veritas v1. plugged into a 30$ startech USB ADC/DAC.
150713_InnerFideliBits_Photo_VibroVeritas.jpg
+
ICUSBAUDIO2D.main.jpg







the color code follows the measurement rigs in the graphs for pretend clarity^_^. so green is always something measured with the IMM6.
I used the same source and output level for each group of measurements, first set the IEM to output 90dB SPL@1khz, then calibrate each rigs so REW would read 90dB@1khz. I also tried my best to get the resonance spikes around the same frequency when it was physically possible. no calibration, compensation or smoothing was applied. the EARS comes with a DF compensation but it would make no sense to use it for that comparison against RAWs.

Etymotic ER4SR into various rigs:
er4sr.jpg


Sony XBA_c10:
XBA_C10.jpg


Fiio EX1 (old one):
EX1.jpg


to my ears, the EX1 has a lot more subs than the ER4SR or the XBA_c10. I hear the ER4SR with an attenuated low end(I EQ it with a +6dB boost in the subs stating around 100hz). the XBA_c10 feels really close to my idea of flat. and the EX1 has the very sort of boosted low end that I adore. so while the quantities are clearly wrong, my Frankenstein rig with the EMC8000+coupler is the only one to follow the general direction of what I'm hearing. the others IMO always show too much subs on sealed stuff and not enough on vented IEMs. and while I can compensate the EMC8000+coupler to push all the low end up by some value and stick closer to perceived response, it's not something I can really control on the other rigs. the only choice would be to have a sealed calibration and a vented calibration. else the results between IEMs really don't follow subjective impressions. so that's something to check for people trying to set up their little amateur system.


all in all, for frequency response graphs only, I'd suggest to stick to the IMM6 and some app. it's cheap, small, practical, and with a little testing done on silicone tubes of various lengths, we can really work out something with most of the resonances close to how they are on graphs from expensive dummy heads. the EARS has the very obvious advantage to make for easy measurements of headphones without making our own "dummy box" for the head. but I'll wait to see if I can solve my issues before advising to buy one(if you get it for IEM measurements, prepare the tweezers to go fishing for tips).
for distortions and stuff like that, there are plenty of microphones with lower noise floor and better overall fidelity than an IMM6. but if cheap stuff could do it all with great results, there would be no need for people to invest in the kind of rig @jude has for HeadFi. at some point, we always have to face reality. pro gears have different requirements and costs. but to have some fun and learn a great deal through experimentation, any cheap solution has something to offer IMO.
 

Attachments

  • variations using IMM6 of each IEM as calib.jpg
    variations using IMM6 of each IEM as calib.jpg
    234.6 KB · Views: 0

Users who are viewing this thread

Back
Top