Does it matter what you wear to work? Heavens yes. Like it or not, many decisions are made on first impressions. You'll be quickly checked over for cleanliness, grooming, and what you have on. It's part of business etiquette.
Are there rules you can memorize? Not really. Like all social rules, you have to intuit what's going on because it can differ from place, to city, to state, to country.