A perfect demonstration of why AB and ABX tests give false negatives

Subjects that don't have their own home
User avatar
Nick
Site Admin
Posts: 15707
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#1 A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

Saw a version of this on QI last night. Perfect example of why any AB comparison will normally "show" there is no difference between two bits of kit, and so (for example) why all amplifiers sound the same.

https://www.theatlantic.com/technology/ ... ar/373036/
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
User avatar
Dave the bass
Amstrad Tower of Power
Posts: 12273
Joined: Tue May 22, 2007 4:36 pm
Location: NW Kent, Darn Sarf innit.

#2 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Dave the bass »

Coooooooooooooooool! I like it.

I understand that my brain can make an unintelligible sequence of sounds into a coherent sentence with meaning, that Soundcloud clip just proved it to me. I'm pleased.

But, I don't agree that its "A perfect demonstration of why AB and ABX tests give false negatives" because, when I'm listening to music on bits of kit I'm not trying to unravel a sentence. What I listen to is the music, often with no words so I don't think I'm doing any 'unravelling'.

I'm struggling to get at what I'm trying to explain for the mo. I need to do more word-thinking.
"The fat bourgeois and his doppelganger"
User avatar
Nick
Site Admin
Posts: 15707
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#3 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

Lets say we have two bits of kit, A and B. AB and ABX testing is used to see if its possible to tell two things apart, if the result is that the testee can't tell them apart, then its declared that the two bits of kit are equivalent so as one is not different to the other it can't be better or preferred.

Lets say B is actually a better less distorted (or in some way better sounding thing), if we hear A first, and then B, we think we have heard a improvement, so we say, ahh, that sounded better, lets go back to A again. But now we can't unhear the improvement, so now going back to A we still hear what we thought was a improvement going from A to B, and we begin to doubt or first thought. Now we go from A to B again, and still hear no improvement, the more we do it the more it becomes clear that we can't hear any difference, and our first thought of a improvement was false.

So even though there was a difference that if we come back to it a few days later, we will hear again, a comparison test shows us that there isnt.

In the video, once you have heard the real clip, you can't hear what you heard first, not matter how hard you try.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
User avatar
pre65
Amstrad Tower of Power
Posts: 21373
Joined: Wed Aug 22, 2007 11:13 pm
Location: North Essex/Suffolk border.

#4 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by pre65 »

As I did not know what ABX testing was I looked it up.

I found this article interesting.

http://hometheaterreview.com/why-do-aud ... x-testing/
The only thing necessary for the triumph of evil is for good men to do nothing.

Edmund Burke

G-Popz THE easy listening connoisseur. (Philip)
User avatar
Nick
Site Admin
Posts: 15707
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#5 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

That article is a perfect example of what I suggest the video destroys.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
Cressy Snr
Amstrad Tower of Power
Posts: 10552
Joined: Wed May 30, 2007 12:25 am
Location: South Yorks.

#6 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Cressy Snr »

Eg,
I listen to amplifier A and it sounds good
I listen to amplifier B and it sounds more detailed "ooh, I haven't heard that triangle before"
I listen to amplifier A and now the triangle I hadn't heard on that amp is there, large as life.
I listen to amplifier B again and now it sounds the same as amplifier A - Bugger!
Sgt. Baker started talkin’ with a Bullhorn in his hand.
User avatar
Dave the bass
Amstrad Tower of Power
Posts: 12273
Joined: Tue May 22, 2007 4:36 pm
Location: NW Kent, Darn Sarf innit.

#7 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Dave the bass »

Got it! Thanks all. OK, its about not being able to undo your hearing recognitions.
"The fat bourgeois and his doppelganger"
User avatar
Nick
Site Admin
Posts: 15707
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#8 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

Dave the bass wrote: Sun Jul 23, 2017 1:16 pm Got it! Thanks all. OK, its about not being able to undo your hearing recognitions.
Yes, as is said in the video, the brain will use all the information it has including past experience to understand a signal.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
User avatar
Dave the bass
Amstrad Tower of Power
Posts: 12273
Joined: Tue May 22, 2007 4:36 pm
Location: NW Kent, Darn Sarf innit.

#9 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Dave the bass »

Cool, got that.

but here's a non-confrontational 'but'...

Part of what I listen to/for in the sound reproduction overall is the sound images in front of me. Now, I don't think the brain can be tricked here because you either hear/see it in your minds eye or you don't.
"The fat bourgeois and his doppelganger"
User avatar
Nick
Site Admin
Posts: 15707
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#10 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

But once you have "seen" it, its very much harder to not see it.

But the meaning (IMHO) of the video is that

Hearing a difference means there is a difference

Not hearing a difference is not proof of the absence of a difference.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
User avatar
Dave the bass
Amstrad Tower of Power
Posts: 12273
Joined: Tue May 22, 2007 4:36 pm
Location: NW Kent, Darn Sarf innit.

#11 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Dave the bass »

Nick wrote: Sun Jul 23, 2017 3:44 pm But once you have "seen" it, its very much harder to not see it.
Agreed in the case of unravelling and 'oooo... I didn't here that before' kinda moments and perhaps 'tone' too but I don't agree in the case of imaging because thats just it, it either is or isn't there for whatever reason.

I don't think its a brain/memory related 'thing'. It's happening in the 'now', or maybe thats just the way I hear stuff, I think I'm quite sensitive (note to self - "you little Princess DTB!") to imaging. I know we're all different and hear different things.
"The fat bourgeois and his doppelganger"
User avatar
Nick
Site Admin
Posts: 15707
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#12 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

Dave the bass wrote: Sun Jul 23, 2017 3:53 pm
Nick wrote: Sun Jul 23, 2017 3:44 pm But once you have "seen" it, its very much harder to not see it.
Agreed in the case of unravelling and 'oooo... I didn't here that before' kinda moments and perhaps 'tone' too but I don't agree in the case of imaging because thats just it, it either is or isn't there for whatever reason.

I don't think its a brain/memory related 'thing'. It's happening in the 'now', or maybe thats just the way I hear stuff, I think I'm quite sensitive (note to self - "you little Princess DTB!") to imaging. I know we're all different and hear different things.
Interesting, I wonder if that means you are particularly sensitive to phase errors?
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
User avatar
Ray P
No idea why I do this anymore
Posts: 6294
Joined: Thu Nov 22, 2007 5:18 pm
Location: Somerset

#13 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Ray P »

Thanks for posting the link Nick, it gives me some understanding of my own audio experiences.

Coincidentally, I took myself into Waterstones yesterday and started browsing this book, which seems to give some interesting general insights;



I'll be getting a copy as a birthday present shortly...
Sorry, I couldn't resist!
User avatar
Ali Tait
Eternally single
Posts: 4373
Joined: Fri Jun 08, 2007 8:10 pm
Location: Galashiels

#14 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Ali Tait »

Nick wrote: Sun Jul 23, 2017 6:52 pm
Dave the bass wrote: Sun Jul 23, 2017 3:53 pm
Nick wrote: Sun Jul 23, 2017 3:44 pm But once you have "seen" it, its very much harder to not see it.
Agreed in the case of unravelling and 'oooo... I didn't here that before' kinda moments and perhaps 'tone' too but I don't agree in the case of imaging because thats just it, it either is or isn't there for whatever reason.

I don't think its a brain/memory related 'thing'. It's happening in the 'now', or maybe thats just the way I hear stuff, I think I'm quite sensitive (note to self - "you little Princess DTB!") to imaging. I know we're all different and hear different things.
Interesting, I wonder if that means you are particularly sensitive to phase errors?
I'd say the same as Dave, it's the imaging I pick up on first. Maybe explains why I've caught reversed phase on a few occasions?
User avatar
Nick
Site Admin
Posts: 15707
Joined: Sun May 06, 2007 10:20 am
Location: West Yorkshire

#15 Re: A perfect demonstration of why AB and ABX tests give false negatives

Post by Nick »

Good example how we all listen slightly differently. For me imaging is not that important, as long as there is good separation of the sounds my head adds the imaging (as it does in mono). So its probably low noise and maybe secondly, low higher order distortion that’s important to me.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
Post Reply