It sounds like he was within the company dress code so it doesn’t matter what the manager thought, this was discrimination. I’d probably leave too, though. After a while we get tired of fighting.