All Northern self-service ticket machines off line 13/7/2021

[Bletchleyite]

I'm going to take an educated guess that if ransomware recovery is taking this long, the backup servers (and authentication needed to access them, either via the 'host' system or directly) were identifiable once the attackers were in the network, and these were also affected.

My personal stuff is backed up in two places - MS OneDrive, but also (less frequently) a physical disk which is in my home safe, with several backup sets on it (I typically rotate three sets).

If any business isn't doing that, they deserve all they get. It's basics.

I agree entirely. But if this was a 'professional business' then it would probably be up and running again by now.

There is that.

Even if the backups were dead, surely running up an entirely new server would just be a matter of installing the base product and applying the separately backed up Northern-specific configuration without any of the transactional data?

 

[Mike395]

I can't imagine that anyone who cared about security would keep all of their backups on networked servers, for precisely this reason. Surely they must have an offline backup somewhere?

You would have thought so. But assuming there's not more to it than we're being told (which is entirely possible) and this was a pure ransomware (extortion) attack with no data compromised and a good level of logging/access control management in place to the server, I struggle to see how it would take a security team at the TVM manufacturer this long to at least put a temporary patch in place to close the loophole for the attack and get the core services back online, particularly as this manufacturer seems to also have unaffected TVMs used by different TOCs, so in theory there should be images available from these that could be taken and rebranded back to Northern as a worst-case scenario..

 

[ClagLover]

Ticket machines are still in situ at Leyland but the people in the ticket office have no idea what’s happened beyond “there was a big hoo ha (understatement) and some sort of cyber attack”

 

[Ken H]

A professional business doesn't just have one backup. It'll have them going back a fair bit, ideally "exponentially" i.e. a day ago, a week ago, a month ago and so on.

A colleague went to a business to install some software. He found they had one tape, and overwrote it every night. No-one had verified the backup routine was actually writing anything to the tape (It was) About 1996.

 

[Wolfie]

A professional business doesn't just have one backup. It'll have them going back a fair bit, ideally "exponentially" i.e. a day ago, a week ago, a month ago and so on.

Agreed. Just hope that they didn't, as they seem to have done for pretty much everything else that they have done, try to do it on the cheap and cut corners/remove safeguards.

 

[Skie]

Googling “flowbird problems” does reveal a number of previous balls ups.

 

[johnnychips]

Are they working yet?

 

[ClagLover]

Are they working yet?

No

 

[skyhigh]

No

Incorrect. They're being switched back on in phases, which started today.

 

[ClagLover]

Incorrect. They're being switched back on in phases, which started today.

Well apologies for being “incorrect” The ones at Chorley and Manchester were certainly not working this morning.

 

[skyhigh]

Well apologies for being “incorrect” The ones at Chorley and Manchester were certainly not working this morning.

You stated a blunt no, in reality it is much more nuanced than that.

6 stations came back online today. The rest will follow over the coming weeks. Despite what the supplier initially thought, investigation has shown that the malware has cause hardware damage to the machines, hence the delay in restoring the network.

 

[domcoop7]

You stated a blunt no, in reality it is much more nuanced than that.

6 stations came back online today. The rest will follow over the coming weeks. Despite what the supplier initially thought, investigation has shown that the malware has cause hardware damage to the machines, hence the delay in restoring the network.

That sounds a bit ... implausible. Not saying it's impossible, but malware causing hardware faults is exceedingly rare and really shouldn't be possible. If they got root / administrator access to each machine, and caused it to run kernel code that upped the clock frequency of a chip until it burnt out or repeatedly stored and deleted data to the SSD (assuming that's what they have) until they exceeded their fault tolerances I guess it could happen. But who or why would a malware attacker do that? It would take some effort and give no gain whatsoever.

Sounds more like a bit of backside covering by the supplier to justify why they haven't got a recovery plan in place.

 

[Dai Corner]

That sounds a bit ... implausible. Not saying it's impossible, but malware causing hardware faults is exceedingly rare and really shouldn't be possible. If they got root / administrator access to each machine, and caused it to run kernel code that upped the clock frequency of a chip until it burnt out or repeatedly stored and deleted data to the SSD (assuming that's what they have) until they exceeded their fault tolerances I guess it could happen. But who or why would a malware attacker do that? It would take some effort and give no gain whatsoever.

Sounds more like a bit of backside covering by the supplier to justify why they haven't got a recovery plan in place.

I won't quote the whole article due to the length, but for example

4.Make your hard drive permanently infected​

Here, I’m not talking about malware that corrupts some files you store on your hard drive, and that will go away as soon as you format it.

I’m talking about malware that stays forever in computer hardware, that doesn’t go away, not even after you’ve installed a new, clean OS version.

More details on whether a factory reset removes malware, you’ll find here.

For now, I’m going to detail the “achievements” of Equation Group, a sophisticated cyber-espionage team that the security researchers from Kaspersky Lab identified back in 2015. As it turns out, those hackers have tapped into hard drive firmware reprogramming.

How is hard drive firmware reprogramming possible?​

Whether it’s a traditional HDD or the newer SSD, any hard disk has several components. The most notable ones are the storage medium (magnetic disks for HDDs, flash memory chips for SSDs) and a microchip.

The microchip is this hardware part where software is embedded to include controls for reading and writing to the disk, along with multiple service procedures that help the hard drive detect and fix bugs.

You could call the hard drive’s microchip a small computer in itself because that’s what it is. It comes with its firmware, and this is the part that the hackers from Equation Group have managed to exploit.

By corrupting the firmware of the hard drive’s chip, they allow malware to read and write data on a particular subsection that can’t be touched even during disk formatting!

How does this make your hard drive useless?​

With the new firmware version, the hard drive’s malware can reinfect the HDD/SSD’s boot area. Consequently, the moment you have finished reinstalling the operating system, the malware is back up.

The remarkable thing about firmware is that the firmware itself is in charge of checking its status (the integrity of its code) and running the available updates. As you can imagine, once the firmware was compromised at this level, it can no longer be trusted to eliminate its infection.

This is why infected firmware is indestructible and why most specialists agree that it’s easier and cheaper to throw away a compromised hard drive than to try and get rid of infected firmware.

Before you freak out, you should note, though, that the process itself of corrupting the hard drive firmware is exceptionally complicated. Because each hard drive model is unique and developing alternative firmware for it is time, resource, and money consuming, hackers have a tough time pulling this off. And they’re not interested in doing it to the regular user, either.

9 Ways a Computer Virus Can Damage the Hardware | Antivirus Jar

A computer virus or malware is meant to steal your valuable data. Causing your hardware to break and making your PC unusable is not in a hacker's interest.

antivirusjar.com

 

[XAM2175]

Well apologies for being “incorrect” The ones at Chorley and Manchester were certainly not working this morning.

So you should have either 1) not posted at all, or 2) posted only that the ones that you looked at this morning in Chorley and Manchester were not working. You can roll your eyes at "incorrect" all you want, but your post was indisputably unhelpful.

 

[domcoop7]

I won't quote the whole article due to the length, but for example
[SNIP]
Before you freak out, you should note, though, that the process itself of corrupting the hard drive firmware is exceptionally complicated. Because each hard drive model is unique and developing alternative firmware for it is time, resource, and money consuming, hackers have a tough time pulling this off. And they’re not interested in doing it to the regular user, either.

Well quite. I didn't say it's impossible, but the gain to be had by flashing the firmware of an SDD controller or a motherboard (which in itself is not as easy as that article suggests, since the firmware would usually not permit a flash of an update that isn't digitally signed, often doesn't work remotely, and even if it does is so flaky as to mean a successful attack would have to be exceptionally lucky - hence why you never hear of it happening in the wild) is what? A ransomware hacker just wants the bitcoin. He or she (or the government they work for) will get that once they've managed to get access. Doing that is difficult enough. Trying to install an extra complicated malware system into the firmware that probably won't actually work is so much effort that they risk not even succeeding in their primary attack.

Like I say, not saying it couldn't happen, but it's a magnitude less likely than the supplier hasn't got a working backup system and needs to go round each machine and install it from scratch.

 

[Dai Corner]

Well quite. I didn't say it's impossible, but the gain to be had by flashing the firmware of an SDD controller or a motherboard (which in itself is not as easy as that article suggests, since the firmware would usually not permit a flash of an update that isn't digitally signed, often doesn't work remotely, and even if it does is so flaky as to mean a successful attack would have to be exceptionally lucky - hence why you never hear of it happening in the wild) is what? A ransomware hacker just wants the bitcoin. He or she (or the government they work for) will get that once they've managed to get access. Doing that is difficult enough. Trying to install an extra complicated malware system into the firmware that probably won't actually work is so much effort that they risk not even succeeding in their primary attack.

Like I say, not saying it couldn't happen, but it's a magnitude less likely than the supplier hasn't got a working backup system and needs to go round each machine and install it from scratch.

Hardware damage is apparently exactly what happened. I assume 'the machines' are the ticket machines at stations as opposed to the servers they connect to at the data centre.

Despite what the supplier initially thought, investigation has shown that the malware has cause hardware damage to the machines, hence the delay in restoring the network.

 

[flitwickbeds]

I guess it's possible that the machines were programmed to try so hard to connect to the server in the event of a failure that something, for want of a better phrase, "burnt out"?

 

[Djgr]

Currently at Manchester Victoria.

All ticket machines out of order.

Not a single ticket office window open.

Northern, the TOC that just keeps giving (grief)

 

[py_megapixel]

Currently at Manchester Victoria.

All ticket machines out of order.

Not a single ticket office window open.

Northern, the TOC that just keeps giving (grief)

Are the barriers closed?

 

[DB]

I guess it's possible that the machines were programmed to try so hard to connect to the server in the event of a failure that something, for want of a better phrase, "burnt out"?

Can't see how that could happen - repeated attempts to connect would just lead to repeated timeouts, which wouldn't cause any hardware damage.

 

[Djgr]

Are the barriers closed?

They are. Random people keep asking staff how of earth they buy a ticket

 

[DanNCL]

The Chiltern machine at Great Missenden, which is the same design and uses the same software as the Northern ones, is working and selling tickets this afternoon. Surely Northern could use a modified copy of Chiltern’s back end system to get their own machines running again?

 

[3rd rail land]

They are. Random people keep asking staff how of earth they buy a ticket

Presumably the answer they get is to buy an e-ticket online using their smartphone.

 

[yorksrob]

Fortunately Normy's seems to be back on now, so no need do descend into modern phone based dystopia just yet.

 

[30907]

Fortunately Normy's seems to be back on now, so no need do descend into modern phone based dystopia just yet.

So is Saltaire's, but not Keighley (which is staffed thouh).

 

[yorkie]

Presumably the answer they get is to buy an e-ticket online using their smartphone.

That may be an option for some but the staff are obliged to let passengers through to buy on the train (or at their interchange/destination if applicable)

 

[Mat17]

What happens if you don't have a mobile phone?

 

[Taunton]

A good number of computer users, having gone "into The Cloud", think that handles all their security issues, things like the computer disk crashing or the server being destroyed by fire.

Not so, if the information out there is lost, due to a range of problems, malicious or otherwise, you have to be able to get it back from somewhere else.

 

[AGH]

On Friday engineer had the one at Newton Le Willows (Manchester bound side) open tinkering with it. Seems the need to attend physically may be correct.

 

[paulmch]

Remind me again the purpose of the conductor

Safety
Punctuality
Customer Service
Revenue

...in that order.