In the episode Dance of the Dead, No.6 actually follows his observer No.240 to the Town Hall, and even tires to follow her inside. But to his shock, there is an electrical force field at the Town Hall steps preventing his entrance into the building.
The gardener working in a nearby flower border, and who sees this asks No.6 if he’s alright. "You tried to go in, by mistake? It’s fussy about who it lets in - this is the Town Hall."
So this begs the question "How does the Town Hall know who to let in and who not? Can the Town Hall building think for itself, or do those who work in the Town Hall, or someone such as the observer who would be passing through to the underground control room, who have some kind of recognition device upon their person, which a sensor picks up and lets them through. Something like the Electro-Pass, which synchronises with the alarm system and lets you through, this device was once given to No.6 by No.9 in Arrival, which he then used to gain access to the helicopter. In this way, it would seem to some people who never go in, that the Town Hall is actually fussy about whom it lets in!
But I wonder why No.6 needed to be told by the gardener that this is the Town Hall? I mean No.6 didn’t seem to know, judging by the expression on his face. After all, hadn’t No.6 been previously to the Town Hall during Free For All? This to witness the dissolution of the out-going Council. Unless of course the episode of Free For All was actually meant to follow Dance of the Dead at some point. Ah, but then No.6 had to consult the electronic "Free Information" board to find the location of the Town Hall in Free For All.
I put it down to too many script writers who worked on the Prisoner, together with the almost complete lack of continuity between the episodes! Less script writers would have meant more continuity between the episodes I feel.
Be seeing you.