Breaking it down, the speakers in the cabinet, and the cabinet design and materials, have two separate roles, and each affects the tone quite a bit.
The only way to know what will give you the sound you want is to play through each cabinet with your amp. I realize this seems like a total PITA, but it's the only advice that actually will guarantee you'll get a sound you like.
For example, when I got my Mesa Fillmore amp head, I got the head first, installed the NOS tubes I wanted, and then played through a bunch of cabinets. Instead of the Fillmore cabinet and speakers, I ordered a California Tweed cabinet and speakers from Mesa, because that's the sound I prefer.
I use PRS cabs with the original speakers for my HXDA and DG30 amps. I think they were well-chosen/spec'd, and they sound 'right' to me.
I'm not 100% into the speakers that came with my Mesa Lone Star combo, but I haven't changed them out yet out of sheer laziness. Hey, it's only been 6 years!!
And heck yeah you'd care if you play at home -- if you're into the tone and it matters to you.
I should add...
If you can't find stuff to play through locally, before taking
anyone's advice on what speakers/cab/amp to buy - even if they're a great player - listen to them play through their rig, or to their recordings. The great caveat here is that even with the very same rig, we all sound
different.
Even the best players have playing styles, tastes, favorite sounds, etc., that may or may not be what you want to hear with your own rig, and as per the above paragraph, you can still sound different.