A perfect demonstration of why AB and ABX tests give false negatives

Subjects that don't have their own home
User avatar
Dave the bass
Amstrad Tower of Power
Posts: 12273
Joined: Tue May 22, 2007 4:36 pm
Location: NW Kent, Darn Sarf innit.

#16 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Dave the bass »

The best image I've ever heard so far was from some commercially available Thoriated Tungsten behemoth many years ago in a HiFi shop here in Dartford. I don't know what the amp was, what was being played or what the speakers were but it was one those stand out moments, almost as if it didn't matter, I was totally wrapped in that sound picture. Awethumne feeling.

I've had summat almost equal here at home, but not quite.... this is the killer.... I totally removed all the furniture for a weekend in the lounge and lived like a nobby no-mates nutter with just one seat in between the Saburos connected to a 3 stage project amp using 2C22 -> summat and 2A3 (as usual) in SE. It was fun. Impractical, but fun.
"The fat bourgeois and his doppelganger"
User avatar
Nick
Site Admin
Posts: 15708
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#17 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

Also interesting, as I would have expected that without furniture the imaging would have been worst as there would be more reflections from walls and stuff.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
User avatar
Dave the bass
Amstrad Tower of Power
Posts: 12273
Joined: Tue May 22, 2007 4:36 pm
Location: NW Kent, Darn Sarf innit.

#18 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Dave the bass »

Yup, total opposite. Dunno if you remember how we had things set up when you came down a long time ago, possibly Steely Dan weekend? One Sab had to fire over the arm of a Sofa and I used to hear the HF suffering as a result of it but still folk insist on sitting down when they 'popped-in', meh!

Well, everything that was in that room I removed and got some impressive sounds and imagining.

At Eggy on my 1st visit Scott had some single driver tall horns that did brilliant imaging IMO. Doobie Bros 3ft in-front of the speakers and filling all the area in between. 2nd Eggy our-Ed had some FE 167's in a slightly curved horn that tickled my Spidey-Image sensors too. Amp wise and sources, no idea.

That was the catalyst for me building a few single driver speaker projects. Sound Image-driven projects in retrospect.
"The fat bourgeois and his doppelganger"
lindsayt
User
Posts: 40
Joined: Sat Jul 22, 2017 8:34 am

#19 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by lindsayt »

Interesting radio link, Nick.

If you play the highly distorted version and the non distorted version it's obvious to anyone listening which is the less distorted version.

With hi-fi if you're comparing 2 components / systems with relatively large sonic differences, that'll still be detected by the listeners regardless of what order they're played in, or any filling in done by the brain.

If there's subtle differences between components I'm quite happy to accept that there may be some brain filling in going on. Especially if the listener is under some stress in a blind test, where he may look like a fool if he gets it "wrong".

But then I'd also argue that if the differences between 2 components are so subtle that I'd struggle to tell them apart in an AB test, then is it worth me spending any money on? Pareto and all that.
User avatar
Nick
Site Admin
Posts: 15708
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#20 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

But then I'd also argue that if the differences between 2 components are so subtle that I'd struggle to tell them apart in an AB test, then is it worth me spending any money on? Pareto and all that.
Thats a strangely inverted argument. The point I was making is that because of the way the brain works (as shown in the clip) even large differences can (will be) be masked by a AB test.

The equivalent would be saying that because you can't tell a kg of lead and a kg of gold apart by sight in a room with no light it it, there is no difference in value between the two.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
User avatar
Dr Bunsen Honeydew
Old Hand
Posts: 1358
Joined: Mon Sep 21, 2009 10:50 am
Location: Muppet Labs

#21 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Dr Bunsen Honeydew »

The argument about AB and ABX tests in hi-fi terms is they are feckin' waste of time, and an excuse to avoid reality. Or an excuse to continue marketing poor product as with Harbeth.

You see in order to understand you have to test AB and ABX testing, in reality it is mind muddling bullshit for normal people. I have had to be subject to it so many times I KNOW what a nonsense it is, but how do I implant that knowledge in you lot, simply I can't, especially with the feckin' eeeediots who think I am the devil incarnate.

So all I ask is don't waste my time with it, or using it to judge NVA, it judges nothing but the users intelligence.
Neal
Shed dweller
Posts: 2299
Joined: Fri Aug 03, 2007 10:57 am
Location: From the land of the Bodgers

#22 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Neal »

That link Nick explains why I've chased my tail at times even with a sighted AB test, thanks for posting it.
Only the Sith deal in absolutes.
User avatar
pre65
Amstrad Tower of Power
Posts: 21373
Joined: Wed Aug 22, 2007 11:13 pm
Location: North Essex/Suffolk border.

#23 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by pre65 »

Dr Bunsen Honeydew wrote: Mon Jul 24, 2017 3:54 pm The argument about AB and ABX tests in hi-fi terms is they are feckin' waste of time, and an excuse to avoid reality. Or an excuse to continue marketing poor product as with Harbeth.

So all I ask is don't waste my time with it, or using it to judge NVA, it judges nothing but the users intelligence.
I'm getting confused here. :oops:

Is a "bake off" (horrible term) not a form of A-B testing ?

If not, why ?
The only thing necessary for the triumph of evil is for good men to do nothing.

Edmund Burke

G-Popz THE easy listening connoisseur. (Philip)
User avatar
Nick
Site Admin
Posts: 15708
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#24 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

pre65 wrote: Mon Jul 24, 2017 5:58 pm
Dr Bunsen Honeydew wrote: Mon Jul 24, 2017 3:54 pm The argument about AB and ABX tests in hi-fi terms is they are feckin' waste of time, and an excuse to avoid reality. Or an excuse to continue marketing poor product as with Harbeth.

So all I ask is don't waste my time with it, or using it to judge NVA, it judges nothing but the users intelligence.
I'm getting confused here. :oops:

Is a "bake off" (horrible term) not a form of A-B testing ?

If not, why ?
They are if they are reduced to the horrible play the same track on every bit of kit process. And are then pointless for the reason given above.

What term would you prefer BTW, audition?
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
User avatar
pre65
Amstrad Tower of Power
Posts: 21373
Joined: Wed Aug 22, 2007 11:13 pm
Location: North Essex/Suffolk border.

#25 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by pre65 »

Nick wrote: Mon Jul 24, 2017 6:12 pm
They are if they are reduced to the horrible play the same track on every bit of kit process. And are then pointless for the reason given above.

What term would you prefer BTW, audition?
Audition would be preferable, or perhaps phono fest. Any other suggestions for a different word, or phrase ?
The only thing necessary for the triumph of evil is for good men to do nothing.

Edmund Burke

G-Popz THE easy listening connoisseur. (Philip)
lindsayt
User
Posts: 40
Joined: Sat Jul 22, 2017 8:34 am

#26 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by lindsayt »

Nick wrote: Mon Jul 24, 2017 3:34 pm
But then I'd also argue that if the differences between 2 components are so subtle that I'd struggle to tell them apart in an AB test, then is it worth me spending any money on? Pareto and all that.
Thats a strangely inverted argument. The point I was making is that because of the way the brain works (as shown in the clip) even large differences can (will be) be masked by a AB test.

The equivalent would be saying that because you can't tell a kg of lead and a kg of gold apart by sight in a room with no light it it, there is no difference in value between the two.
I think that the clip shows that large differences will be detected. With the 2 clips it's obvious that the one with the missing sections is a highly distorted version of the original. When listening to the distorted clip for a second time after hearing the non-distorted one I was very much aware that I was filling in the blanks and listening to the distorted version.

If on playing the distorted clip a second time, I was unable to tell whether I was being played the distorted version or not then yes that would indicate that large differences can be masked by the brain. But that was not the case with this example. Every time I heard the distorted version I was fully aware that I was listening to an annoyingly grossly distorted recording.
User avatar
Nick
Site Admin
Posts: 15708
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#27 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

lindsayt wrote: Mon Jul 24, 2017 6:34 pm If on playing the distorted clip a second time, I was unable to tell whether I was being played the distorted version or not then yes that would indicate that large differences can be masked by the brain. But that was not the case with this example. Every time I heard the distorted version I was fully aware that I was listening to an annoyingly grossly distorted recording.
Yes, of course, but that was not the point I was making, not that you could not tell the distorted version from the non distorted version, but than once you heard the undistorted version, the distorted version sounded very different, and you could not go back to the situation where the distorted version was unintelligible.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
User avatar
Nick
Site Admin
Posts: 15708
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#28 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

pre65 wrote: Mon Jul 24, 2017 6:30 pm
Nick wrote: Mon Jul 24, 2017 6:12 pm
They are if they are reduced to the horrible play the same track on every bit of kit process. And are then pointless for the reason given above.

What term would you prefer BTW, audition?
Audition would be preferable, or perhaps phono fest. Any other suggestions for a different word, or phrase ?
I dont know why you would prefer Audition, and its not a festival, its a comparison of phono stages, (or of any things).
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
User avatar
pre65
Amstrad Tower of Power
Posts: 21373
Joined: Wed Aug 22, 2007 11:13 pm
Location: North Essex/Suffolk border.

#29 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by pre65 »

Nick wrote: Mon Jul 24, 2017 6:41 pm
I dont know why you would prefer Audition, and its not a festival, its a comparison of phono stages, (or of any things).
Well, was Egg fest a festival ?

The choice of the word "audition" was your suggestion Nick, but at the end of the day you call it what you like. :?
The only thing necessary for the triumph of evil is for good men to do nothing.

Edmund Burke

G-Popz THE easy listening connoisseur. (Philip)
User avatar
andrew Ivimey
Social Sevices have been notified
Posts: 8307
Joined: Mon Jun 11, 2007 8:33 am
Location: Bedford

#30 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by andrew Ivimey »

Listen In.

Aud It.

As in, to have a '... ...'
Philosophers have only interpreted the world - the point, however, is to change it. No it isn't ... maybe we should leave it alone for a while.
Post Reply