Riyad-us-Saliheen 810
Anas (May Allah be pleased with him) reported: Messenger of Allah (PBUH) permitted Zubair and 'Abdur-Rahman bin 'Auf (May Allah be pleased with them) to wear silk because they were suffering from an itch.
[Al-Bukhari and Muslim]
This Hadith tells us that, if needed, even men can wear clothes made from silk. For example, it may be suggested in case of a skin disease like itch. A man may also be allowed to wear a silk dress in order to escape extremely cold or hot weather provided he has no other clothes.