[NL] Sunday 3. April 2022 Train services in the Netherlands almost completely stopped

[Austriantrain]

Due to an IT fault, train operations in the Netherlands almost completely broke down today and will not be reestablished until tomorrow.

See NS‘ press release

Sunday 3 April no more NS trains | NS

Sunday 3 April no more NS trains

www.ns.nl

 

[MisterT]

This was an NS Reizigers only issue. Other operators (including NS International for cross-border services) were able to run as usual.

The issue has been resolved, but the root cause is still not known. It is also not known why the backup system failed.
The regular night trains will be the first trains to start running again.

 

[Bletchleyite]

I suspect a country sharing the same flag colours as the Netherlands likely is involved. No, not France or the UK, one more usually associated with red. Bit poor that they are just saying "nothing at all, come back tomorrow" - what are stranded people supposed to do, walk? You would think they could manage to put something basic together.

 

[biko]

The problem was in the IT for the passenger information and control. So it was impossible to keep track of where trains went as I understand it.

 

[MisterT]

I suspect a country sharing the same flag colours as the Netherlands likely is involved

No, not at all. There was no hack or anything else related to cybercrime, just a good old bug/system failure. That much is already known.

So it was impossible to keep track of where trains went as I understand it.

Correct. And not just the trains, all drivers and guards as well. With everything being centralised these days, that makes a very difficult situation.

 

[rvdborgt]

This was an NS Reizigers only issue. Other operators (including NS International for cross-border services) were able to run as usual.

Except that NS was lying about that because quite a number of their international services were cancelled in the first hours after the failure.

 

[DanielB]

.Bit poor that they are just saying "nothing at all, come back tomorrow" - what are stranded people supposed to do, walk? You would think they could manage to put something basic together.

The problem is in the centralization, so there's hardly any local staff to start a simple shuttle service. Signals, points and trains were working, staff was willing to offer something, but there simply was no coordination.

Even poorer is that NS webcare was making statements that alternative transport costs would not be refunded, which is a violation of EU passenger rights. And even in it's franchise, NS is obliged to bring you home in case of delays where they're at fault: one should even get a taxi when a delayed train causes missing the last bus home.

In the mean time regional bus operators suffered from severe overcrowding on services parallel to railways, resulting in skipping local stops and thus impacting even more people.

 

[MisterT]

Except that NS was lying about that because quite a number of their international services were cancelled in the first hours after the failure.

Lying is a pretty serious accusation. It would mean that NS International had the same system failure, but didn't communicate it with malicious intent. Did you even consider that they weren't lying at all? After all, there might be another reason why those trains were cancelled in the first few hours.
For example, there were NS Reizigers trains stranded in all kinds of places. The trains in the large stations piled up pretty quickly and other trains were stopped at some random stations somewhere along the lines.
Both things could lead to some international trains being cancelled because those other trains were blocking up stations and lines, and international trains couldn't run any further as a result.
Same with staff that was already on its way to start their shifts by means of the train. They wouldn't be able to make it in time.
Once that was sorted out, international trains ran without real issues. They even provided some additional stops to help stranded passengers along their routes, and extended some of the services to pick up other passengers as well.

Obviously the whole thing was a disaster for NS Reizigers and I'm ashamed that my employer failed to bring all those people home, failed to provide alternative means of transport and failed to inform passengers of their rights, but I have no bad words for the colleagues at NS International. They did what they could do to help.

 

[Gerald Fiennes]

National system failures do seem to keep on happening in NL, though.... (GSM-R: looking at you here!!!)

 

[Bletchleyite]

National system failures do seem to keep on happening in NL, though.... (GSM-R: looking at you here!!!)

Failures happen. The point is more that there is seemingly no contingency plan, e.g. a set of printed paper diagrams to institute a simple hourly or half hourly shuttle service on main routes.

 

[biko]

GSM-R: looking at you here!!!

This wasn’t a failure of GSM-R as communication with signallers wasn’t affected. It solely had to do with the systems within control (Bijsturing in Dutch) meaning control didn’t know where trains and staff were. As I understand it, systems of NS international are separate so they didn’t have problems.

 

[rvdborgt]

Lying is a pretty serious accusation. It would mean that NS International had the same system failure, but didn't communicate it with malicious intent. Did you even consider that they weren't lying at all? After all, there might be another reason why those trains were cancelled in the first few hours.
For example, there were NS Reizigers trains stranded in all kinds of places. The trains in the large stations piled up pretty quickly and other trains were stopped at some random stations somewhere along the lines.

NSI wasn't lying, they didn't say anything, not on their website in any case.
On ns.nl, however, NS was claiming all day long that international trains were not affected by the failure. This was not true, as they could establish themselves easily and since they chose to say something else, that classifies as lying in my book. DB on the other hand, first said that ICE and IC to Amsterdam would end at the last scheduled stop in Germany, which they later changed to "running with delays".

 

[MisterT]

Again, straight up lying implies some malicious intent by NS. I don't see that anywhere.
Bad communication? Yes, definitely. Bad communication with international partners? Probably. Straight up lying? No.
Especially in the early stages of these kind of enormous disruptions, reliable information is hard to come by.

 

[rvdborgt]

Again, straight up lying implies some malicious intent by NS. I don't see that anywhere.

Lying does not necessarily mean there's malicious intent. To say or write something that you know is not true is already lying.

 

[MisterT]

So you were there and know for sure that the person publishing the message had absolute knowledge about some of the international trains being cancelled and decided to change the message?
If not, I would kindly ask you to refrain from that kind of speculation. We both agree that things went horrendously wrong, and that communication was a failure, but there isn't any reason to suggest lying about it as well.

 

[LNW-GW Joint]

This is the Railway Gazette report:

Scheduling software failure sidelines NS services

NETHERLANDS: National passenger operator NS suspended all services on April 3 following problems with its resource allocation and operations planning software.

www.railwaygazette.com

According to NS, the IT failure at the end of the morning had affected ‘the system that generates up-to-date schedules for trains and staff’, which it insisted was ‘important for safe and scheduled operations’. The software normally adjusts train diagramming and crew rostering automatically in the event of any incident on the network, but this was ‘not possible due to the failure’.

As I understand it, Network Rail is very keen on these Train Management System applications, used to replan services at times of disruption, and is experimenting with them in certain ROC installations.
Looks like some lessons could be learned from NS.

 

[rvdborgt]

So you were there and know for sure that the person publishing the message had absolute knowledge about some of the international trains being cancelled and decided to change the message?

Of course, you can never be 100% sure. Maybe the one person publishing the message didn't know or didn't check himself. But the information on cancellations was available for anyone to see (it's public information) and as an organisation, NS must have known but chose not to tell. Or chose not to check (which would be gross negligence) and publish something regardless.

 

[DanielB]

This is the Railway Gazette report:

Scheduling software failure sidelines NS services

NETHERLANDS: National passenger operator NS suspended all services on April 3 following problems with its resource allocation and operations planning software.

www.railwaygazette.com

In contrast to what this article states, I understood that the software actually already failed in the morning. The problems were initially limited to frozen information displays at stations.
The crashed software only became a problem when a disruption occured in Hoofddorp (near Schiphol Airport) which messed up the complicated crew rosters. Being unable to reschedule staff then quickly resulted in chaos and the decision to cancel all trains.

 

[Falcon1200]

The crashed software only became a problem when a disruption occured in Hoofddorp (near Schiphol Airport) which messed up the complicated crew rosters. Being unable to reschedule staff then quickly resulted in chaos and the decision to cancel all trains.

That is simply astonishing; Do NS not have some form of Operational Control organisation with staff able to manipulate staff and set diagrams during disruption ?

 

[MisterT]

Eh yes, they do. But that is exactly the software that failed

 

[Falcon1200]

Eh yes, they do. But that is exactly the software that failed

Thanks, but a couple more questions arise;
If the service had been running normally with no disruption, could that not have continued ? And do they not have paper copies of diagrams as a basic back-up, I know that is old fashioned stuff nowadays but that is how the railways ran for decades !

 

[Dr Day]

Sorry lots more questions to understand the potential relevance elsewhere.

What is the equivalent software used in GB? From some of the description posted above feels like a TOC rather than Network Rail piece of software (ie rostering/vehicle allocation rather than signalling) so potentially different TOCs have different suppliers so if the same thing happened it wouldn't bring down the network nationally, just in those TOCs that used it? Yet other posts talk about passenger information displays which indicates something to do with the on-the-day timetable itself which here would be a Network Rail issue. Or was it related to the attributes of the train (eg is the 0700 from X to Y a 3-car or a 2-car today) which are presumably only known once the specific unit has been allocated to the planned diagram?

Did freight trains run?

 

[MisterT]

If the service had been running normally with no disruption, could that not have continued

They did that, but then multiple disruptions happened at the same time, and that was the point where everything went wrong.

And do they not have paper copies of diagrams as a basic back-up

I don't think so, but I don't think that it would have been possible to keep up with the number of trains we run on a Sunday. It's not just a simple A to B and back to A diagram, because that's not efficient enough. We'd need a whole lot more staff if everything was diagrammed like that.

Sorry lots more questions to understand the potential relevance elsewhere.

It was an NS Reizigers issue. So local trains by Arriva, Keolis and Connexxion ran without issue, as did freight trains and international trains.

The passenger information system failure had the same cause (broken data connection to the real-time system), which meant that it only showed trains as planned. Which also meant that any cancellation, late running, or platform change wasn't visible to passengers.

 

[rvdborgt]

The passenger information system failure had the same cause (broken data connection to the real-time system), which meant that it only showed trains as planned. Which also meant that any cancellation, late running, or platform change wasn't visible to passengers.

Wasn't there any backup for that broken data connection?

 

[MisterT]

I have no idea how those systems are set up exactly and what the redundancies are. This is just my own interpretation of the words of one of the developers. I guess he could have been talking about the connection with the same management system that had failed.

 

[Gerald Fiennes]

This, and the various GSM-R issues I referred to above, do perhaps suggest that centralising everything in a 'single national system', whilst superficially appealing (particularly in a smaller country such as NL), may well have some drawbacks. The systems architecture needs a review!

 

[MisterT]

In Dutch news today: from a letter from the state secretary to the parlement, it appears that there actually was a backup system, but it failed. There will be an external review about the events and the decisions that were made.

Edit: and also in an internal news message, the same was said by the CEO of NS Reizigers. There was a backup system, but it failed. The root cause is still unknown, but a hack or other kind of cyber attack is already ruled out as a possibility. Both internal and external reviews are ongoing, there will also be a review looking into possibilities to keep the trains running in case something like this happens again.

 

[paul_munich]

they didn't run one single replacement bus, with the amazing explanation "we dont have enought buses, so we dont run any buses at all". Some buses between Amsterdam, Utrecht, the Hague and Eindhofen would have been better then nothing at all, and there should have been plenty of buses available on a Saturday...

 

[DanielB]

The problem usually isn't the availability of buses on a Sunday, but the availability of drivers for them. But still, you'll need an enormous amount of buses to serve for example Amsterdam - Utrecht - Eindhoven by replacement bus. A four car VIRM carries ~400 seated passengers, a six car unit has ~600 seats. So you'd need 5-7 buses to replace just one train and there are four trains an hour on this route currently, taking at least 2 hours for a single trip by bus.

So a simple calculation: you'd need at least 80 to 112 buses to serve this one corridor, even ignoring trains with coupled VIRM sets. Imagine how many you'd need to replace trains countrywide?
If anything had been done, the only thing really making sense would have been shuttle trains as those would have the required capacity.

 

[rvdborgt]

If anything had been done, the only thing really making sense would have been shuttle trains as those would have the required capacity.

Yup. And I understand that would have been possible and staff/line managers (or whatever they're called today) wanted to organise that but management didn't want it.