I prefer forwards, but an aisle seat as I don't like being squashed against the side of the train, especially by a big bloke with a thick overcoat on! Also I have eyes that are sensitive to any degree of glare, and avoiding the window seat cuts down the chance of bright sunlight reaching my face.
I prefer going forwards as the "thrust" of the train tends to push me back against the seat back, which is more comfortable than the equivalent sensation when going backwards, which causes loss of back support and is generally less relaxing.
I must agree with the OP though, about how annoying it is when someone sits next to you when there are plenty of other seats free. Happened to me yesterday in Liverpool, and as I had picked one of the end-of-coach double seats that have no space between the seat and window, nor between seat and adjacent seat, when a large woman plonked herself next to me I got up and moved elsewhere. Whereupon she managed to occupy the entire double seat on her own!