If you want to find the answer to "does it matter" how a man dresses, then unfortunately you will have to consider personal tastes and the context of the question - not to mention commonly accepted social norms. Not easy, then. You would think that to be considered for a top job in the city, you wouldn't be considered if you showed up for your interview in shorts and t-shirt. It matters too much; too much to the interviewing company, too much for commonly and socially accepted norms, and too much for the situation. Therefore yes, how a man dresses does and can matter.
When it comes to finding a man attractive, for instance, this question opens up a whole new set of arguments that depend upon personality traits, physical attributes and, of course, what a men wears - how much that will matter will depend entirely whichever the potential partner you question at the time and their tastes. After all, beauty is in the eye of the beholder, which may sound like a clich, but it is one that still holds true.
That said, men's fashion is becoming increasingly popular and with more men becoming more fashion conscious, it seems like - to men at least - how they dress does matter. The proliferation of men's fashion magazines, websites and awareness in men's fashion is general is increasing awareness of how men dress - both to men and women.
How much influence the media, famous celebrities and of course, the women in men's lives have, is a far reaching argument that spans all social economic circumstances, social scenes and of course down to how far any man can be influenced by another person (or combination of the above). Celebrity appeal is certainly influencing men's fashion, but how much "this matters" is a truly subjective question.
Many men consider the opinion of women to be gospel and would never consider fashion 'fads' in order to avoid running the risk of being ridiculed - something which, many men would argue, the women in their lives would be only too happy to oblige them for.