All Northern self-service ticket machines off line 13/7/2021

[Killingworth]

A full week down and no apparent public news of restoration. I wonder how much revenue will be being lost? Or, perish the thought, may actually be being gained as more use apps and be encouraged to buy tickets from more vigilant ticket inspections.

 

[GLC]

This is now been ongoing for over a week; this is "somewhat amateur hour" at the very least... a clean rebuild from scratch and thorough testing shouldn't take this long.

There is significantly more work involved in restoring service after an incident like this than what you are suggesting. It shouldn’t be the case that it takes months to restore service, but a week+ is not unusual

 

[Andrew1395]

Perhaps GB Railways will decide to introduce its own self contained data network, with a mega data centre somewhere in a red wall area like Nottingham?

 

[Starmill]

Does it really make that much difference if the ticket machines aren't working? Most people will either:
- Already hold a ticket or booking reference as they've booked in advance or use a season ticket
- Buy a ticket on their phone anyway
- Be paying in cash, which means they pay on the train anyway
- Be trying to avoid paying, in which case they certainly won't be using the ticket machine!

 

[Steddenm]

The ticket machine today at Poulton le Fylde today was switched on, but the card reader had been deactivated as well as the cash insert bit. All it had on the screen was an error message though and kept trying to reboot Windows, but the screen orientation was the wrong way round.

And also, the CIS on the platforms were all showing the time as 00:00s and everytime it got to 59 seconds it went back to 00:00:00. The destinations were showing as

0013 Undefined
Calling at: undefined only. A service which has 0 coaches.

Seems like PLF has had more problems tonight too with the automated system being down and just counting up to 10 every fifteen minutes.

 

[Geeves]

Yes the CIS system was being upgraded yesterday however some issues occured, it was supposed to be 45 min job at 10am. They put Victoria back online at 1800 ish on the old server.

 

[yorksrob]

Does it really make that much difference if the ticket machines aren't working? Most people will either:
- Already hold a ticket or booking reference as they've booked in advance or use a season ticket
- Buy a ticket on their phone anyway
- Be paying in cash, which means they pay on the train anyway
- Be trying to avoid paying, in which case they certainly won't be using the ticket machine!

It does for me.

In the absense of a ticket office, I find them pretty good for purchasing split tickets and tickets to unusual destinations that guards might not have heard of.

 

[PR1Berske]

This is now been ongoing for over a week; this is "somewhat amateur hour" at the very least... a clean rebuild from scratch and thorough testing shouldn't take this long.

I wonder if we have a case here of "it's nearly fixed, just wait a couple of hours.... oh dear, still broken, let's try again" is being repeated. Of course the IT support team has probably been cut to the bone, or outsourced to the cheapest supplier (and has been said, Northern Rail were stuck with the incumbent...) so they may not have sufficient experienced staff to resolve in a timely manner.

if the contract doesn't have penalty clauses then the purchasing team - and senior exec responsible - need a stiff talking to. I wonder if his bonus can be clawed back - if he is still there of course.

It's unfair to say "amateur hour" if it's a serious attack by outside sources for which Northern and their IT team are simply unprepared. Do we know how big an attack it is? Because if it's this complex to repair, and the attackers are not amateurs, the fix might be more than anyone here anticipated.

 

[Djgr]

Does it really make that much difference if the ticket machines aren't working? Most people will either:
- Already hold a ticket or booking reference as they've booked in advance or use a season ticket
- Buy a ticket on their phone anyway
- Be paying in cash, which means they pay on the train anyway
- Be trying to avoid paying, in which case they certainly won't be using the ticket machine!

In which case why bother with ticket machines at all?

 

[PR1Berske]

In which case why bother with ticket machines at all?

If you can imagine the voice of a passenger going "But there wasn't even a ticket machine for me to BUY before boarding," then you can imagine why there are ticket machines.

 

[Dai Corner]

If you can imagine the voice of a passenger going "But there wasn't even a ticket machine for me to BUY before boarding," then you can imagine why there are ticket machines.

I wonder how many passengers there are who would use a debit card in a ticket machine but wouldn't buy a ticket on their phone?

 

[roversfan2001]

I wonder how many passengers there are who would use a debit card in a ticket machine but wouldn't buy a ticket on their phone?

Well it's at least 1. Assuming 'buy a ticket on their phone' refers to e-tickets (as opposed to ToD).

 

[Robertj21a]

This is now been ongoing for over a week; this is "somewhat amateur hour" at the very least... a clean rebuild from scratch and thorough testing shouldn't take this long.

I wonder if we have a case here of "it's nearly fixed, just wait a couple of hours.... oh dear, still broken, let's try again" is being repeated. Of course the IT support team has probably been cut to the bone, or outsourced to the cheapest supplier (and has been said, Northern Rail were stuck with the incumbent...) so they may not have sufficient experienced staff to resolve in a timely manner.

if the contract doesn't have penalty clauses then the purchasing team - and senior exec responsible - need a stiff talking to. I wonder if his bonus can be clawed back - if he is still there of course.

On the assumption that you must be an IT expert (?), do you actually know the true extent of the problem ?
You do sound a bit like someone having a rant without having all the facts - I'm sure that can't be the case.

 

[SargeNpton]

Perhaps GB Railways will decide to introduce its own self contained data network, with a mega data centre somewhere in a red wall area like Nottingham?

The term "iron ring" comes to mind.

 

[Killingworth]

I wonder how many passengers there are who would use a debit card in a ticket machine but wouldn't buy a ticket on their phone?

I'm now quite happy to buy online, be it on an app or online, and have my ticket on my phone.

However, when travelling with my wife I print out our tickets at home as she is still not convinced. Observation on the platforms and in trains suggests increasing numbers are not using ticket machines. Hopefully for Northern's income more ticket checking is being arranged.

 

[Starmill]

In which case why bother with ticket machines at all?

Indeed. Was it a reasonable use of public money to install almost 650 of them?

 

[py_megapixel]

Indeed. Was it a reasonable use of public money to install almost 650 of them?

Well at least the public can be reassured by the fact that as little as possible of their money was seemingly spent...

 

[deep south]

On the assumption that you must be an IT expert (?), do you actually know the true extent of the problem ?
You do sound a bit like someone having a rant without having all the facts - I'm sure that can't be the case.

I have been in IT for many years, and no-one outside knows the true extent. It looks like they haven't informed the ICO so it may well be that no customer data is at risk. I wasn't trying to have a rant, but there are precautions against intrusions that should have been in place. I do have personal experience where organisations have specifically chosen to implement "less effective" safeguards to save money / reduce costs, and the overall cost of "not doing it right" can turn out to be much higher.

There is a truism in IT that has been around for many years that seems to have been forgotten nowadays; there are three potential attributes to balance - good / fast / cheap. Pick any two from three....

 

[Djgr]

Think I might have found a no fuss solution

 

[Bletchleyite]

Think I might have found a no fuss solution

On-board revenue collection worked well enough in the days when Northern/RRNW/whatever was full of half-empty 2-car 150s and Pacers, but the high loadings and longer trains we get today make collecting fares impossible, so a TVM-and-penalty-fare approach is the only one really workable.

OK, you'd get away with having a few routes as Paytrains - Ormskirk-Preston, Kirkby-Wigan etc - but not most of Northern like it used to be.

That the TVM system is cheap rubbish doesn't make using TVMs a problem in itself, it makes using those TVMs a problem. But that said anything can be caught out with this sort of attack these days unless you want to have to pay someone to drive round with a USB stick and update them all.

 

[flitwickbeds]

...there are three potential attributes to balance - good / fast / cheap. Pick any two from three...

Indeed. My side business is web design and coding. I enforce this in all my new clients before I take on their job.

You can have it cheap and good, but it won't be fast.
You can have it good and fast, but it won't be cheap.
You can have it fast and cheap, but it won't be good.

As it's only a side businesses which I fit around my full time job, I don't even offer the "fast" option - but I use it to say "I will make it good, it will be cheaper than other quotes you might get... But other companies may be able to offer a quicker launch. Your call how you proceed."

99% of my clients get it straight away!

 

[Bletchleyite]

Indeed. My side business is web design and coding. I enforce this in all my new clients before I take on their job.

You can have it cheap and good, but it won't be fast.
You can have it good and fast, but it won't be cheap.
You can have it fast and cheap, but it won't be good.

99% of my clients get it straight away!

Most of those are fine, of course, but I do wish more IT firms would simply refuse to do the third one, as it is to nobody's benefit, really. Clients insisting on the latter one damage the industry's reputation terribly.

 

[flitwickbeds]

Most of those are fine, of course, but I do wish more IT firms would simply refuse to do the third one, as it is to nobody's benefit, really. Clients insisting on the latter one damage the industry's reputation terribly.

Yep. I've lost count of the amount of projects I've rescued from fast-and-cheap developers where the client hasn't got the results they've wanted, realised their mistake, upped the budget and got their website back to "good" (but not fast).

 

[py_megapixel]

Indeed. My side business is web design and coding. I enforce this in all my new clients before I take on their job.

You can have it cheap and good, but it won't be fast.
You can have it good and fast, but it won't be cheap.
You can have it fast and cheap, but it won't be good.

As it's only a side businesses which I fit around my full time job, I don't even offer the "fast" option - but I use it to say "I will make it good, it will be cheaper than other quotes you might get... But other companies may be able to offer a quicker launch. Your call how you proceed."

99% of my clients get it straight away!

Most of those are fine, of course, but I do wish more IT firms would simply refuse to do the third one, as it is to nobody's benefit, really. Clients insisting on the latter one damage the industry's reputation terribly.

I sometimes wonder if I just have strange tastes in software design. For example, I can't stand the kind of software (be it websites or anything else) where almost every element has some kind of of flashy animation on it. Is there really any need for every paragraph of text to fade up as I scroll, and do the buttons really need to gradually change width on hover, and does the navigation menu with all of about 3 options really need to slide in and take up the entire viewport? Absolutely not, and I actually find it quite irritating, but to your average small business owner I imagine it looks quite attractive.

The point of this is that the "cheap/good/fast, pick any 3" thing doesn't always hold true. Sometimes in the case of a basic site for, let's say, a restaurant, I'd rather have something which has been put together in a short time, is simple (and therefore cheap) but robust, than something which tries to do everything, is covered in 'trendy' things (like superfluous animation) which serve only to make the site less accessible and more irritating to use.

Of course this isn't really applicable to ticket booking interfaces, which are inherently complex, but still need to be good to use.

 

[XAM2175]

The point of this is that the "cheap/good/fast, pick any 3" thing doesn't always hold true. Sometimes in the case of a basic site for, let's say, a restaurant, I'd rather have something which has been put together in a short time, is simple (and therefore cheap) but robust, than something which tries to do everything, is covered in 'trendy' things (like superfluous animation) which serve only to make the site less accessible and more irritating to use.

What part of that isn't covered by "good"?

In reality the extended maxim is that you should never expect all three attributes for a project but you may occasionally get lucky - or you might end up with only one of them.

 

[py_megapixel]

What part of that isn't covered by "good"?

It depends how you define "good", is the point. I would argue that it is covered, but if everyone thought the same there would be many overly flashy rubbish small business websites than there are currently!

 

[Killingworth]

Extract from Northern's latest update to Stakeholders

"TVMs: Earlier this week we experienced technical difficulties with our ticket vending machines (TVMs), which meant all have had to be taken off-line. We continue to investigate the issue, alongside our supplier, and it appears the ticket machine server has been subject to a ransomware cyber-attack. Working with the supplier, swift action was taken; the incident has only affected the servers which operate the ticket machines. Moreover, customer and payment data has not been compromised. We are working to restore normal operation to our TVMs as soon as possible. In the meantime, we are advising customers to either use Northern’s mobile app or website to purchase tickets in advance and, where necessary, to collect those from one of our ticket offices. Clearly, those offices can also be used to buy tickets. As promise to pay and pre-paid tickets cannot be obtained, we are advising customers to board their scheduled service. Penalty fares are suspended while we are in this position, which may continue for some time."

 

[LondonExile]

If the ransomware got their backups as well - they may have a hefty task in front of them.

There is an arms race at the moment between criminals looking to get paid their ransoms, and IT professionals seeking to defend against it. At the moment, I'd say that the criminals have the upper hand. One case I've seen recently targeted the IT staff desktops (rather than the servers directly), then waited until the IT staff had logged into both primary and backup systems, stole the passwords, and set about destroying both. It's not unheard of for targeted attacks to have a period of months where the attackers are "shadowing" the IT staff to try and map out the systems and work out how to defeat the protections put in place.

Where I work - we're investigating in moving back to tape-based backups on the grounds that a tape locked offline in a safe can't be tampered with. Even that doesn't close down all attack vectors - if someone managed to load malicious firmware onto the tape drives themselves, we could always be in a position where the tapes got scrubbed instead of loaded when it comes time to restore.

It doesn't surprise me that they're still down.

 

[Skie]

A backup and disaster recovery strategy is useless if you never actually test it. So many IT departments religiously perform backups but only when disaster strikes do they find out that it wasn’t backing up some critical item or the restoration process isn’t well documented or is flat out broken.

There are also lots of old hat IT security people who just do things that worked 20 years ago and refuse to change. If you have to change your password every 60 days, aren’t encouraged to use multiple word passphrases instead of Password123 and password managers are disabled then your IT security is probably worthless.

 

[billh]

A friend told me today that he was on our local station, one which has a ticket office manned all day,that the ticket machines were being loaded onto a van and taken away. Now that sounds serious to me? I would have thought that machines could be re-loaded , re-booted or whatever down the network or have their processor boards replaced in-situ?