It would drive me nuts. I basically work in the saloon on stopping services working backwards and forwards and I am much happier being able to operate the doors from any coach.
I agree it would drive me nuts too. Generally I'm in the train checking tickets on every service I work regardless of time of day. Obviously certain trains I'll weigh up and decide not to bother and opt to just walk through/dispatch from various saloon doors occasionally and keep an eye on things (generally on very drunk trains).
There are journeys where I've been unable to do retail duties for over an hour because I have been busy dealing with other things. Most recently I had a guy onboard who kept moving seats, sitting on the floor, going into the toilet, had no ticket, claimed he was travelling to X, then when it came to X wouldn't get off and wouldn't tell me where he was actually going, generally acting odd. I spent an hour trying to arrange with BTP to meet the train and remove him, which was easier said than done and then had to figure out a suitable well staffed station to arrange for assistance with the matter. In-between operating doors, battling with poor phone signal, people not answering the phone etc it took my over an hour to get it sorted!
There are plenty of other things which take priority such as unit swops (confirming information with control/platforms over which unit is going where, which unit is going forward, where passengers need to go etc then communicating said information with the punters). If the train is splitting, ensuring passengers are in the right portion and that the gangway doors are all locked up ready to go in time.
If the service is significantly delayed I tend to avoid retail duties (going through when very very late and asking for money is just asking for a smack!) And rather go through advising of delay repay, and printing out journey itinerarys for onward connections before arriving at connecting stations for those who have missed their original connection.
That said I reckon I manage to check around 90% of all passengers tickets on my trains generally.
Some trains which are busy you can end up taking most of the journey to get through half the train and don't see the front half, in that instance those in the back will have seen you constantly whereas those in the front coaches I never got to will be wondering where the lazy guard is!
That said, there are definitely a few "relaxed" guards out there, it is particularly annoying when you're travelling on an "easy" train for a day out somewhere and don't see the guard come through.