Plenty of people manage to look scruffy in shirt and tie and plenty of others look professional in polo and shorts.
Overly tight or baggy clothing, and that which has not been cared for correctly (faded, creased, torn, stained etc) looks awful regardless of what the item of clothing is (although...