I agree with your last point completely and the studies were not designed to look at the impact on performance. The studies are contrary evidence to your first point - the musicians were adamant about their ability to tell the difference. To be clear I am not aware of any similar study outside of violins.
They actually conducted two studies. The first was criticized because it was conducted in a hotel room. The second study, which reproduced the results of the first, was conducted in a concert hall.
I've watched videos on those studies.
There's one interesting thing to note about A/B tests that don't get mentioned - there's an inherent problem when it comes to listening because ear memory after very few back and forths between A and B becomes confusing. This is also a matter addressed in the literature.
Long term listening tests are actually more accurate.
It reminds me of the 'cable controversy'. Two years ago I did some recordings with an acoustic guitar, a Neumann mic, a 1073 preamp, and two cables; one from Mogami, one from Sommer that cost quite a bit more. The Mogami is very good cable.
The preamp went straight into the interface. The mic position was not moved a millimeter.
There was a significant enough difference between the two cables that I wound up buying three of the Sommers for use with my condenser mics, at a cost of around $750. I wouldn't have done it without testing.
I posted links to the results on a forum I had for a while. All of the people who posted responses said that I wasn't alone in hearing it, all were able to hear the difference.
Several years before, my excellent studio tech, with a EE from Michigan, insisted I couldn't hear the difference between the fat speaker cables I was using and zip cord. It was impossible, he said. He insisted we do an A/B test. I had to leave the room so he could set it up with a sheet and a couple of mic stands behind my analog console, so he could change the wires without me seeing what he was doing.
In 10 runs, I got 9 right, the last was the one I couldn't because my ears were fatigued. As you know, 9 out of 10 is statistically significant.
My tech literally stormed out of the studio, shouting, "That's impossible. Something must have gone wrong!" Seriously. The guy got in his mobile fix-everything truck, and drove off mad.
Um...OK.
A similar thing happened when I was recording some ads in Holland at Wisseloord, Polygram's studio at the time. We had recorded everything on a Sony digital open-reel DASH machine, at the insistence of my clients. That was the first high-resolution digital multitrack, a $200,000 machine. The recordings sounded great. BTW, we chose the studio because the clients insisted on finding one to record with.
I mean, OK, send me to Europe for a week to work in a world-class studio! Yay!
The agency producer wanted the fade out at the end to be shorter, this after the musicians had gone back to London.
Trouble was, making it shorter cut off the tail of the last chord's reverb, and he couldn't deal with it. So Hans, the engineer said, "No problem, I have a digital machine that can speed it up enough to keep the tail. You will never hear a difference."
So he used it. I said, "It doesn't sound right." It didn't. The texture was wrong to me. The timing seemed off. I know what we laid down, and knew exactly what to listen for.
He insisted that was because I knew what he'd done, and wanted to do an A/B blind test.
The clients thought that was a fun thing, to test my ability to hear certain details in a mix. So I left the room, they did an A/B. I got it right. Then they did it 9 more times. I didn't miss once.
Why? I don't have super-hearing. But I do know what to listen for, and I lived with my demo track for a couple of weeks before making the trip.
A guy who makes high end AC cables near me can show on an oscilloscope that his freakin' AC cables show a measurable difference. I tried a couple with my studio monitors and guitar amps, and hot damn, the difference was indeed audible. All of my studio audio gear is now powered with them, including my AC balanced isolation transformer, and the recording equipment, including the converters, the monitors, the monitor controllers, the preamps, the guitar amps.
Yes, I know. This is 'impossible'. But it isn't. I did a back and forth for my son, who has several gold records as an engineer/producer. He heard it right away.
I spent a lot on professional acoustical treatment, and upped the ante a couple of years ago with more. The room sounds great. So I hear stuff.
One of my engineer/studio owner (lots of high end gear and a great studio) friends came in and asked to hear the results. I played a few high resolution, 24 bit, 196 kHz tracks. He turned around from the mix position and said, "This sounds like a mastering suite."
I kinda like that!
I have been invited to guest lecture on music production at U of Michigan School of Music and at regional AES. So someone out there thinks I'm not 100% crazy!
"So you're 75% crazy, then."
"I'll take that and run with it!"
I honestly think that there are a lot of tests that are simply based on invalid assumptions, including the one that short A/B tests are always valid. They are not. There are problems that people often ignore.