I'm going to come at this from two different positions, as a musician and as an entertainment writer, because this really is a somewhat complicated question. I'm going to apologize if this is just too lengthy of BS'ey for some.
As a musician, I used to be somewhat critical of players. Before I started playing the instrument I looked up to players such as Billy Corgan (Smashing Pumpkins) Ed Roland and Ross Childress (Collective Soul), Chris Cornell and Kim Thayil (Soundgarden), and Jerry Cantrell (Alice In Chains). After maybe a year or so of being addicted to guitar I got fairly good and in learning some of my favorite artists songs I got kind of down on them because I couldn't believe how simple my favorite artists music really was. There was one instant where I figured out an album by ear in an afternoon. That didn't make me feel accomplished and proud, I felt like my favorite artists were kind of hacks.
Through the years when I would go on and learn more favorite albums by Creed, Sevendust, Puddle of Mudd, Nickelback, Finger Eleven, Godsmack, Our Lady Peace, Tool, I just got annoyed at the simplicity. Finally I graduated high school and went to college and didn't know what to major in, so I majored in music theory biased to jazz, which I still don't know why. My professor one day told me he was a big fan of some of the bands I wore shirts by and he gave me a different perspective that I never thought of. These guys, some great clean players (Ed Roland and Ross Childress, Jerry Cantrell), some sloppy (Billy Corgan, Kim Thayil) were actually kind of geniuses. Why? Because they could write a great song that stuck with you, musician or not, that appealed to more than just one type of person. Never thought of it that way. So I gained a new respect for all of the players whose music I once thought was retardedly simple.
So now when I'm learning some of my favorite bands songs, or can figure some out without an instrument in hand I kind of admire the genius of it and don't try to sit and analyze it. Why bother, obviously they're not just doing something right, they're doing something great.
As an entertainment writer I not only interview artists, but I review live performances and full albums. With interviews there are a few basic questions, but if they're a renowned player my inner guitar geek can't help but ask a few questions. I've interviewed Warren Haynes and Mark Tremonti - how could I not ask them about gear questions and tunings?
For live performances I think the only time I've ever analyzed the playing was with Collective Soul. Granted I've seen the band more with Joel Kosche than Ross Childress, but I still remember seeing and hearing Ross play those old songs flawlessly whereas Joel doesn't nail it the same way. However I'm there to review the show as a whole, not the people as players so I have to look at it non-objectively for that matter.
As for albums...I don't even want to think about how many I've reviewed in the past 8 years. What stinks though is you come to a point where it all starts to bleed together. Sometimes I'll listen to an album in my car a few months after I've reviewed it and I feel bad because 1, I wish I had more time to listen to it before I reviewed it because it may be better than I originally wrote, and 2, now I feel bad because I like the album that much. An example of this is Bush's latest Sea of Memories album. I only had two days with it. It's actually one of my top 3 favorite albums released last year
So in reviewing albums it's not about the playing, but if there are any key songs that grab you right away, continuity (they do pay people to order the tracks for song to song flow) and if the disc is worth it as a whole. There are times where I'll actually say, this song has two good tracks, go on iTunes or Amazon and get em there because the rest of this disc is filler. My rules that I give my writers are pretty much the same. If the album is great, great, why. If it's sucks, ok, detail why this is not the artist’s best work.
Again, sorry if that was a little long. My apologies for that. Just figured I would throw my hat in from a different perspective.