I definitely hear a difference between fretboard woods.
As a synthesist, sound designer and pro studio rat, I tend to describe sounds in terms of ADSR envelopes, harmonic overtones, etc. The caveat is that most of the differences are obscured by amp or pedal gain, so unless you’re at least occasionally a clean player, you may not hear them.
In any case, the woods tend to have different ADSR envelopes; that is, their rates of attack, decay, sustain and release vary when a note is plucked.
The ear picks up on this, it’s one of the things that makes, say, a horn sound different from a piano. Both instruments generate sine waves that sound different at the same pitch, because their ADSR and harmonic overtones are different.
If you listen carefully to the envelope of the note through a clean amp, you’ll hear differences in ADSR between the woods used in guitars.
As an example, a Maple fretboard tends to have a fast attack, and faster decay, accentuating the pick and releasing quickly for the next note. This is partially why the ear perceives it as bright, and also why it’s well-suited to chicken pickin’. Notes pop off the fretboard. To a degree, ebony also has a fast attack, and is perceived as a brighter sounding material.
Rosewood tends to have a slower, more searing attack and decay, though I find BRW a little faster than IRW, and the notes ‘ring’ a little more. Madagascar RW is somewhere in the middle, nicely balanced IMHO.
Listen to the differences between identically constructed Maple-bodied acoustic guitars and rosewood, or mahogany. You hear differences, but of course because of the amount of wood used, they’re more obvious. Nonetheless, those differences in tone don’t vanish when these woods are used elsewhere on the guitar in smaller amounts. They’re simply less obvious.
It’s not a matter of thinking that one person can hear what another can’t. It’s more a matter of knowing what to listen for. Try listening to several guitars with various fretboard woods, listen for how they accentuate pick attack, and how a note decays and releases, and you may hear these differences. Eventually, if you play a bunch of them, concentrating on ADSR, you’ll notice the similarities and differences between these woods very easily.