Speaking of unwelcome, rather than simply unexpected visitors, I remember seeing a website dealing with the history of Salford which had a former signalman's account of working at Brindle Heath Junction box.
In its heyday Brindle Heath Jn was in the middle of lots of railway infrastructure (multiple running lines, sidings, steam loco depot etc). By the end of the 1960s most of this had been dismantled, leaving the signalbox and a simple double track junction in the middle of an extensive abandoned wasteland.
Not far away was Pendleton, a rough and notorious area of inner-city blight, with its brutal post-war tenements & tower blocks and ample complement of disaffected "yoofs" and ne'er-do-wells. The two photos from Flickr illustrate the location.
The signalman's story was that at night Brindle Heath Jn would regularly be under siege from local vandals hurling stones (ballast) at the box windows from somewhere out in the surrounding darkness, with windows often being smashed until BR installed all-over protective grills. I can imagine it would be a very stressful and threatening place to be working on your own on a late shift - even if nothing was happening, you would never knew when something might kick off, or what creative methods of vandalism the scrotes might dream up next.
Apparently, after Brindle Heath closed following re-signaling, it was only a matter of weeks before the box was burned down by arsonists.
I suppose this wasn't the only mechanical signal box to suffer this kind of problem in Britain's rougher areas.
Brindle Heath Junction by
Ingy The Wingy, on Flickr
Brindle Heath Junction by
Ingy The Wingy, on Flickr