Surely if they disagree with the online map, then they're going to have to sue the lawyers who acted on their behalf when they bought the property, since that map is part of the title deeds.
My suspicion- and it is just a suspicion- is that at some point in the past the cottage and railway platform/track were part of the same parcel of land and shared a common owner. This would make a lot of sense, being a railway cottage on a railway station. The new owners of the cottage seem to believe that because it was once a shared parcel of land then that means it still should be. This does appear to be a sincerely held belief.
I'm not a solicitor and I'm pretty clueless about Scottish land law. My opinion is based on my experience from conveyancing as how it works in England and Wales with mandatory registration. However from the online record available publicly I don't see how that belief can be correct. Being incorrect doesn't (necessarily) mean they are being malicious.
As for negligence, the first thing I always did was talk to my client and ensure that the title plan accurately reflected what they thought they were buying. It usually did but occasionally it did not; typically that happened when the property had a parking space under a separate register entry (common with flats for a while) and the vendor either forgot or "forgot" to include that parking space in the transaction.
If they don't own the platform etc, the only way for them to obtain ownership of it would be to agree to purchase it from whoever does own it. And they may not want to sell it.
I'm not sure if this applies in Scotland but in England and Wales you can acquire "possessory title", i.e. you never formally acquired the land but you've been in possession of it for a period of time (this is 12 years in England and Wales). The central registration of land now makes it a lot harder to obtain possessory title because the registered owner of the land is known, but it is easier with unregistered land.
I used to see it most often where houses that backed on to an old railway line moved their garden fences ten feet so as to bring the land of the old railway line into their property. Was always fun to unravel, and I was always happy to refer that upwards to my senior solicitor!