If (and I'm not advocating this) you abolished Off Peak walk up tickets this problem would disappear.
Which is precisely what Scotland are trialling.
https://www.bbc.co.uk/news/uk-scotland-63998198
(Well the other way around, scrapping the peak premium).
Perhaps another topic, but seeing the loadings I would really challenge whether in timetabling and fares, there has been enough adaptation to what "peak" now is or isn't.
There is an odd mix of demand suppression, timing incentives, historical anomalies and administrative intransigence which has built the UK fares system.
I always think we should look at these not through the lens of the person who can understand the deepest depths of BR Fares, but someone who has landed here from a country with an efficient, integrated public transport system. They have sincerely gone to a vending machine and bought a ticket to a destination. Beyond choosing between one full rate ticket and one half rate ticket (and possibly 1st Class), it is unfathomable to most people in that person's shoes that they would be sitting on a train, to that destination, with a ticket with that destination printed on it, and for that ticket not to be valid.
Incentivisation and discounting should be done through advance ticketing "this train only" (and I think my idea for "Rail Miles" loyalty scheme wasn't an entirely stupid one). The price you pay on the train or walk up in the station should be a basic fare, and should not be punitively expensive, either.
Massive simplification is desperately needed.