Is The trope that wizards can't use armor just an arbitrary limitation?

[mindstalk]

A sufficiently powerful magician should have invulnerable skin anyway and thus not need armor. :) Like Monkey in Journey to the West.

Gandalf the White implies that the weapons of the Three Walkers wouldn't avail them much against Saruman; he himself incinerates an arrow in mid-air and telekinetically disarms Gimli... which might the only TK in the legendarium.