On the morning after his arrival in the village, the Prisoner is given a new identity, along with a credit card, employment card, card of identity, and a health and welfare card. And upon the left lapel of his piped blazer he wears, but for a few seconds, his badge of identity, that of No.6 which he removes and tosses onto a back seat of the taxi. He nevers wears that No.6 badge again, save for one time during the episode of The Schizoid Man, when it suits No.6's purpose to do so.
But No.6 is not the only one to go about the village not wearing a numbered badge. The doctor who conducted No.6's medical in Arrival didn't wear a badge. The Butler, No.2's manservant doesn't wear a badge. What's more both the Professor and his wife in The General are allowed not to wear their numbered badges. Perhaps that's one of their certain privilages both the Professor and Madam Professor enjoy!