All Northern self-service ticket machines off line 13/7/2021

[py_megapixel]

A friend told me today that he was on our local station, one which has a ticket office manned all day,that the ticket machines were being loaded onto a van and taken away. Now that sounds serious to me? I would have thought that machines could be re-loaded , re-booted or whatever down the network or have their processor boards replaced in-situ?

Interesting. I'll see if my friend who lives next to another Northern station noticed anything similar.

Northern have been adamant so far that the attack was only on the backend and not the TVMs themselves, but maybe if the TVMs need to be re-imaged for some reason they might need to be taken away.

If the issue is determined to be serious enough that NT have dropped Flowbird as a supplier, then the hardware would have to be physically replaced. I doubt that's the case though.

 

[Killingworth]

A friend told me today that he was on our local station, one which has a ticket office manned all day,that the ticket machines were being loaded onto a van and taken away. Now that sounds serious to me? I would have thought that machines could be re-loaded , re-booted or whatever down the network or have their processor boards replaced in-situ?

I find that - interesting. It might be that one or two machines have other faults. Ours seems to be getting a burned image at the bottom of it's screen but powering off completely should stop that. This is what they look like inside.

 

[PR1Berske]

If the ransomware got their backups as well - they may have a hefty task in front of them.

There is an arms race at the moment between criminals looking to get paid their ransoms, and IT professionals seeking to defend against it. At the moment, I'd say that the criminals have the upper hand. One case I've seen recently targeted the IT staff desktops (rather than the servers directly), then waited until the IT staff had logged into both primary and backup systems, stole the passwords, and set about destroying both. It's not unheard of for targeted attacks to have a period of months where the attackers are "shadowing" the IT staff to try and map out the systems and work out how to defeat the protections put in place.

Where I work - we're investigating in moving back to tape-based backups on the grounds that a tape locked offline in a safe can't be tampered with. Even that doesn't close down all attack vectors - if someone managed to load malicious firmware onto the tape drives themselves, we could always be in a position where the tapes got scrubbed instead of loaded when it comes time to restore.

It doesn't surprise me that they're still down.

I remember the NHS attack a few years ago. That was a tad clumsy and fudged but it still brought down the NHS for a good number of days, and suddenly both the NHS and wider government must have seen how sophisticated these attacks had become.

The Northern attack sounds and feels like the most significant by an outside force for a while. One has to wonder just how much of an impact they could have had they brought down something national: electricity? internet/phone lines?

 

[Six Bells]

Before becoming a signaller, I worked in public sector IT; we were always told that to prevent attacks, it paid to be as up to date as possible (All operating systems and patches up to date) or as archaic as possible (Services still running on mainframes that could only be attacked by a malicious person sitting at a control terminal, so no remote access possible)

In my years on the railway I have seen nothing to contradict this.

The amount of "digital" railway infrastructure that runs on out of support, insecure kit would petrify the IT security bods in most industries.

 

[johntea]

I remember the NHS attack a few years ago. That was a tad clumsy and fudged but it still brought down the NHS for a good number of days, and suddenly both the NHS and wider government must have seen how sophisticated these attacks had become.

The Northern attack sounds and feels like the most significant by an outside force for a while. One has to wonder just how much of an impact they could have had they brought down something national: electricity? internet/phone lines?

I work in NHS IT, Wannacry (the attack, although not wholly specific to just the NHS) longer term has actually been a blessing in disguise for us as following that we rather quickly got as much money and support as we needed from NHS Digital in terms of security!

Our backups now comprise of primary online hard drive storage (this would be the most likely to be successfully hacked), daily / weekly / monthly tape backups which more recently we now even send off site to an external specialist company who store them for us and just as a cherry on top we got funding for some secondary ‘immutable’ hard drive storage (so the hackers can’t just wipe/encrypt/modify files)

Having said all that I do appreciate if we were unlucky enough to come under a full attack I suspect it would take several months to properly get absolutely everything back up and running again! We are only a small team of 6 and now have over 300 virtual servers to manage!

 

[Geeves]

I've not seen it posted and apologies if so but the Northern App was also off-line yesterday. I'm not sure if this is related but the ticket office is the last thing standing.

 

[skyhigh]

I've not seen it posted and apologies if so but the Northern App was also off-line yesterday. I'm not sure if this is related but the ticket office is the last thing standing.

Completely unrelated (also guards are selling tickets, along with revenue officers).

A friend told me today that he was on our local station, one which has a ticket office manned all day,that the ticket machines were being loaded onto a van and taken away. Now that sounds serious to me? I would have thought that machines could be re-loaded , re-booted or whatever down the network or have their processor boards replaced in-situ?

I'm not so sure about that - some machines are being replaced by a more durable type as a trial to find something suitable to replace the machines at stations which are constantly being vandalised, so perhaps that's what's happening there. But that's been planned for a while.

 

[Ken H]

Tape backups are essential. You need backups from the last 12 month ends and last 4 year ends. Whole system.
Pity everything has to be connected to everything else these days. Some benefits of stand alone kit...

 

[DB]

Before becoming a signaller, I worked in public sector IT; we were always told that to prevent attacks, it paid to be as up to date as possible (All operating systems and patches up to date) or as archaic as possible (Services still running on mainframes that could only be attacked by a malicious person sitting at a control terminal, so no remote access possible)

I'd agree with that to an extent, but not where it concerns Microsoft operating systems - the same holes tend to exist going back years, and once they are patched on the newer versions the malware writers will be looking to see if they affect older versions, and may well look to exploit this. Basically, anyone running out-of-support Microsoft operating systems is asking for trouble.

Really, patches these days need to go on immediately when they are released - yes, there's a small chance that they might be faulty and cause issues but that's now generally less of a risk than unpatched systems.

 

[LondonExile]

I'd agree with that to an extent, but not where it concerns Microsoft operating systems - the same holes tend to exist going back years, and once they are patched on the newer versions the malware writers will be looking to see if they affect older versions, and may well look to exploit this. Basically, anyone running out-of-support Microsoft operating systems is asking for trouble.

Really, patches these days need to go on immediately when they are released - yes, there's a small chance that they might be faulty and cause issues but that's now generally less of a risk than unpatched systems.

It's necessary, but not sufficient.

Relatively recently - a company called SolarWinds got compromised, and the patches their customers deployed were then hijacked to allow the attackers into the customers systems.

Those customers could have done everything right. Always updated, only allowed the systems to talk to the update server, verified digital signatures etc. All of this is then for nothing, as ultimately you can't know whether or not the code coming down from the supplier is malicious or not. Open Source software is not immune to these either - PHP accepted a malicious commit turning the platform into a webshell.

A case I know of recently had the controlling firmware on their storage arrays replaced with a malicious one that spun the disks out of tolerance leading to physical destruction.

This is why I firmly believe anyone who thinks they are 100% protected against ransomware either doesn't have any IT systems, or has deluded themselves.

There are steps to take to reduce your risk, and indeed in some cases it's clearly negligent to not take some steps - but there is always the risk you become the first victim to an even more sophisticated attack than you have defences against.

 

[jon0844]

The Northern attack sounds and feels like the most significant by an outside force for a while. One has to wonder just how much of an impact they could have had they brought down something national: electricity? internet/phone lines?

Sadly I think we'll find out one day. I think such attacks are going to become more commonplace.

 

[DB]

This is why I firmly believe anyone who thinks they are 100% protected against ransomware either doesn't have any IT systems, or has deluded themselves.

Yes, indeed - no system is 100% secure, especially if connected to the internet.

 

[Facing Back]

Yes, indeed - no system is 100% secure, especially if connected to the internet.

You don't even need to be connected to the internet - but it helps.

I hired a penetration testing company a few years back to test one of our secure "sand boxed" systems. When they found that the network security was pretty good (not perfect), they hired an actor, mocked up a security pass and charmed their way into the office building, where they found a desktop computer turned on and somebody had popped out to the loo... The rest is history.

The data security team at my place work very closely with our cloud provider - who themselves are pretty good - a well known, large company, and certainly not the cheapest, so we get to understand some of the attacks and some of the things that we and others are doing about it. As Ops here have posted, these attacks are becoming very sophisticated, and the people developing the tools can make millions. Microsoft and Apple both pay up to $200,000 to anyone who makes them aware of a vulnerability in their systems, but people develop these exploits can make more selling to criminal gangs.

We don't use tape backup - we considered it - but the sophistication of some of the malwear means that is lies dormant or more perniciously sltowly affects data and systems so you don't notice it is there for months, and by then the tapes themselves are corrupted. When you are dealing with peta-bytes of data, tapes are not useable in any case.

So as people here have said - from my perspective - it really is a war between the technical teams to stay ahead. I'm not going to go into details (because most of them are way too technical for me) but we use tools like AI (machine learning) which analyses data access patterns and looks for anomalies in both data and executable code - give aways include changes to historical or static data, access from unusual sources, patterns of data change etc. It is an industry in its own right - and as we are a data driven organisation - like many, it costs us a fortune, and we know we are far from immune.

The security advice I have been given by a company who shall remain nameless was that we don't need to always beat the hackers, we just have to be a hard enough target so that they prefer to attack someone else. Sounds quite mercenary doesn't it?

I know little about trains, other than I use them and I am a geek and enjoy reading these forums - but enterprise IT is something I do know a great deal about. Flowbird have my sympathies.

 

[Steddenm]

The machines today at Sunderland were constantly trying to boot windows without any success, repeatedly saying a keyboard was not connected.

The information display screens at the top of the machines were showing the windows blue screen of death with a QR code. The QR code directed to a webpage advising for the system to be reinstalled with a new copy of windows.

The LNER machines at Newcastle were all showing as offline also earlier (at 5am) but seems to be back online again now.

 

[Killingworth]

Seems the machines are reacting in different ways. I wonder if some of the machines may be suffering damage as an indirect result of the attack and won't be able to spark back into service as usual.

Separately, I've heard that tickets may not be being very actively sold on some trains - allegedly a physical shortage of tickets.

 

[Djgr]

Seems the machines are reacting in different ways. I wonder if some of the machines may be suffering damage as an indirect result of the attack and won't be able to spark back into service as usual.

Separately, I've heard that tickets may not be being very actively sold on some trains - allegedly a physical shortage of tickets.

The wrong type of paper?

 

[skyhigh]

Separately, I've heard that tickets may not be being very actively sold on some trains - allegedly a physical shortage of tickets.

There is 100% not a shortage of ticket rolls. We have plenty of them.

 

[py_megapixel]

The wrong type of paper?

Well given conductors mostly use PRT, while the TVMs are still using CCST, a rapid switch from one to the other could presumably mean there is a surplus of CCST stock but a shortage of PRT stock? It seems unlikely though, as I imagine they have plenty in reserve.

Most likely conductors are still hesitant to go through the train due to Covid, I'd have thought.

PRT = printed roll ticket, CCST = credit-card sized ticket, TVM = ticket vending machine

 

[Tracked]

Noticed the Northern machine, and the LNER machines, were showing out of order messages at Doncaster yesterday evening

 

[Djgr]

Well given conductors mostly use PRT, while the TVMs are still using CCST, a rapid switch from one to the other could presumably mean there is a surplus of CCST stock but a shortage of PRT stock? It seems unlikely though, as I imagine they have plenty in reserve.

Most likely conductors are still hesitant to go through the train due to Covid, I'd have thought.

PRT = printed roll ticket, CCST = credit-card sized ticket, TVM = ticket vending machine

Remind me again the purpose of the conductor

 

[Dai Corner]

Remind me again the purpose of the conductor

Well, the guard (to use the generic term) is primarily responsible for ensuring the train arrives safely and on time. Revenue protection is after those.

 

[deep south]

It's now 2 weeks and no sign yet of them working again...? That's not a very successful "new ticket machine roll out" is it...

 

[Dr Hoo]

I'm still not clear if the (relatively 'dumb') machines are essentially OK but the high-tech back office 'server' has been nobbled by a ransomware attack.

Was the server even 'new' or were the new machines essentially hooked up to an old and vulnerable server?

Is it the case that similar machines with other operators are still working, with different servers, (or are Northern's machines unique)?

Can anyone on the inside shed any light?

 

[deep south]

Can anyone on the inside shed any light?

Maybe there's no one left on the inside? I have a vision of two interns in a basement in Bangalore frantically reading manuals to work out what is going on!

 

[Bletchleyite]

I'm still not clear if the (relatively 'dumb') machines are essentially OK but the high-tech back office 'server' has been nobbled by a ransomware attack.

Was the server even 'new' or were the new machines essentially hooked up to an old and vulnerable server?

Is it the case that similar machines with other operators are still working, with different servers, (or are Northern's machines unique)?

Can anyone on the inside shed any light?

Chiltern's machines are of the same type, I have not heard of these being down.

Given a backup (they do have a backup, right?) how is wiping the server and reinstalling from the backup proving that hard?

 

[py_megapixel]

Chiltern's machines are of the same type, I have not heard of these being down.

LNER's too

Given a backup (they do have a backup, right?) how is wiping the server and reinstalling from the backup proving that hard?

If their backup takes the forum of just imaging the drive, then it also backs up whatever security vulnerability let in the ransomware in the first place!

 

[Mike395]

Chiltern's machines are of the same type, I have not heard of these being down.

Given a backup (they do have a backup, right?) how is wiping the server and reinstalling from the backup proving that hard?

I'm going to take an educated guess that if ransomware recovery is taking this long, the backup servers (and authentication needed to access them, either via the 'host' system or directly) were identifiable once the attackers were in the network, and these were also affected.

 

[py_megapixel]

I'm going to take an educated guess that if ransomware recovery is taking this long, the backup servers (and authentication needed to access them, either via the 'host' system or directly) were identifiable once the attackers were in the network, and these were also affected.

I can't imagine that anyone who cared about security would keep all of their backups on networked servers, for precisely this reason. Surely they must have an offline backup somewhere?

 

[Bletchleyite]

If their backup takes the forum of just imaging the drive, then it also backs up whatever security vulnerability let in the ransomware in the first place!

A professional business doesn't just have one backup. It'll have them going back a fair bit, ideally "exponentially" i.e. a day ago, a week ago, a month ago and so on.

 

[py_megapixel]

A professional business doesn't just have one backup. It'll have them going back a fair bit, ideally "exponentially" i.e. a day ago, a week ago, a month ago and so on.

I agree entirely. But if this was a 'professional business' then it would probably be up and running again by now, or at least would have some idea of roughly when it should be.