So the Prisoner found out the location of the Village, much good it would do him. And thinking about it, he might come to realise who it is behind the Village, those of his own side - the British, or himself, depending on ones perspective. Eventually he went back to the only place he can ever go, to his cottage of '6 Private.'
I suspect that even if the Prisoner had left well alone, and not gone off to discover the location of the Village, "they" would have found him eventually, wherever he went, and would have ended up back in the Village eventually. So perhaps his return to the Village in 'Many Happy Returns' was inevitable, and why the Prisoner was so pragmatic about it, taking it in his stride the way he did.
Be seeing you