The first thing you notice when you pick up a guitar is how it feels in your hands. The guitar neck is paramount to that moment, and the feel and function of the frets plays a major role in this. If you boil it down, we're basically talking about fret material and shape. If the material is too soft, metal flakes will fall off of the guitar as it's played. If it's too hard, the frets contour to the fretboard. We use the hardest, functional material we can to avoid unnecessary re-frets. We also superglue the frets in their slots to avoid fret movement and to avoid introducing water back into the neck. As far as shape goes, a fret needs to be level and perfectly radiused to maintain the integrity and comfort of the neck, prevent string buzz, and give the instrument the ability to have low action.
Frets also contribute to an instrument's overall tone. Obviously, they create the scalable notes of the guitar, but it is also important to note that when a string is engaged against a fret, the fret is carrying the string's vibration into the wood. So, they can either add to or take away from a guitar's natural resonance and musicality. We use frets with a lot of nickel, which rings well. And between the fret radius, the superglue, and the tines of the fret, they fit so well in the fret slots that the sound transfer is unhindered.
In the end, the point is that every piece of the puzzle is important, including frets.