There are two things which are certain regarding the Prisoner-No.6. His individuality, and unquestioning loyalty. There is no-one else in the village like our friend No.6, well save for No.12-Curtis and the possibilty of No.1. As for the Prisoner's loyalty, look at the way he always goes running back to those ex-colleagues of his, even after his experience during The Chimes of Big Ben. I don't know how the Prisoner could have placed his trust in them again, after being betrayed by the Colonel and Fortheringay as he was. I wonder what would have happened if there had not been a change of Colonel for Many Happy Returns?
There is someone else in the village, someone as individual as No.6, and whose loyalty is unquestioning, that of the Butler. He like No.6 does not wear a numbered badge. And because we know less about the Butler than we do No.6, perhaps the Butler is the real man of mystery as far as the Prisoner is concerned.