Elizabeth Line - total loss of signalling 26/11

[TFN]

AFAIK this wasn’t a planned upgrade that went wrong. This is just software that went wrong.

 

[Sunil_P]

National Rail OJP showing trains running Paddington <-> Shenfield some OK, some "with delays", some "cancelled".

 

[AndrewE]

A lot of trains displayed on RTT are still cancelled, see: https://www.nationalrail.co.uk/live-trains/departures/tottenham-court-road-elizabeth-line/

Oh dear, that's another illusion shattered! I thought RTT was the best that outsiders (i.e. the public) could get to seeing the actual current plan...

 

[Joe Paxton]

I increasingly seems like something untoward has happened here

The thought that crossed my mind yesterday was whether it could be the result of nefarious actions. But that's probably a pretty far-fetched scenario.

 

[Sunil_P]

Lines reopened between Abbey Wood and London Paddington: disruption expected until 15:00​

Incident: Abbey Wood | National Rail

Lines reopened between Abbey Wood and London Paddington: disruption expected until 15:00. Find details of all current incidents on National Rail Enquiries.

www.nationalrail.co.uk

 

[AndrewE]

The thought that crossed my mind yesterday was whether it could be the result of nefarious actions. But that's probably a pretty far-fetched scenario.

not far-fetched at all. I know people say that generally a cock-up is more likely than a conspiracy, but it wouldn't be the first time, and https://www.bbc.com/news/articles/ceqxezer7nqo says

the latest in a series of warnings about the cyber-warfare capabilities of Russia, which McFadden is to call a "hidden war" being waged against Ukraine.
He is also expected to single-out Russia's Unit 29155, which the government says has carried out a number of attacks in the UK and Europe.
In a speech to the Nato Cyber Defence Conference at Lancaster House in London, the cabinet minister will warn "cyber war can be destabilising and debilitating" and describe the Kremlin as "exceptionally aggressive and reckless" in this area.
McFadden will focus on the threat of Russia's capacity to shut down power grids and "turn the lights off for millions of people", as well as its willingness to target British businesses "in pursuit of its malign goals".
"Given the scale of that hostility, my message to members today is clear: no-one should underestimate the Russian cyber threat to Nato. The threat is real," he will tell the gathered Nato allies.
He will claim that hacking groups aligned with the Russian state have been responsible for at least nine separate cyber attacks against NATO states, including unprovoked assaults against critical national infrastructure.

 

[D6130]

The thought that crossed my mind yesterday was whether it could be the result of nefarious actions. But that's probably a pretty far-fetched scenario.

Yes....that thought had also crossed my mind. A Russian cyber attack in retaliation for Ukraine having been given the go-ahead to fire British-supplied long-range missiles into Russian territory?

 

[Taunton]

No info such as "part suspended" on the status page?

I wonder what page this is?

An Internet page I have to search for every time I go to do a journey, or even look at as I go along. That doesn't particularly work at all when down in the Underground.

The platform train describers? They don't have any room any longer, given they are full of incessant See It Say It Sorted messages.

 

[apinnard]

They’re not. Siemens had the interim signalling contract for the Jubilee line extension, via their acquisitions of Westinghouse/Invensys. The Victoria line uses a Westinghouse-derived signalling system.

Thales is now part of Hitachi.

Also, Hitachi absorbed Ansaldo STS. And we all know how good their stuff is (I’m looking at you, Manchester South.)

 

[Crossover]

Just restarted. 9B82 is the first (and only) train in the COS as of 19:20.

Test trains will run in the COS shortly.


ETCS plus ATO allows 24tph in the thameslink core. I was told that ETCS wasn’t chosen for Crossrail as CBTC worked with Platform Doors while ETCS didn’t have that capability yet.

What does COS stand for in this scenario please (also seen CCOS, not sure if a typo) - apologies if I have missed it explained somewhere else

 

[liam456]

Crossrail Central Operating Section (the shiny new tunnels)

 

[NickBucks]

Any chance of Mayor Khan changing the name of this line to the Liz Truss line to avoid our late lamented monarch from being mired in this?

 

[Sunil_P]

Any chance of Mayor Khan changing the name of this line to the Liz Truss line to avoid our late lamented monarch from being mired in this?

T R U S S

 

[xotGD]

T R U S S

Wrong forum, Dr P!

 

[43096]

Thales is now part of Hitachi.

Although to be clear, it’s not the whole of Thales that Hitachi have acquired: it’s the Ground Transportation Systems business.

 

[Taunton]

It's difficult to understand how there can have been a complete failure throughout Tuesday morning, which was fixed by the afternoon and service resumed, but then another complete failure on Wednesday all morning as well. From what I can see the very first trains got out on Tuesday, but failed along the way, whereas on Wednesday nothing seems to have started at all.

 

[Goldfish62]

It's difficult to understand how there can have been a complete failure throughout Tuesday morning, which was fixed by the afternoon and service resumed, but then another complete failure on Wednesday all morning as well. From what I can see the very first trains got out on Tuesday, but failed along the way, whereas on Wednesday nothing seems to have started at all.

Everything's back running normally now. No need to panic and rip out CBTC and replace it with ETCS.

 

[Horizon22]

It's difficult to understand how there can have been a complete failure throughout Tuesday morning, which was fixed by the afternoon and service resumed, but then another complete failure on Wednesday all morning as well. From what I can see the very first trains got out on Tuesday, but failed along the way, whereas on Wednesday nothing seems to have started at all.

It appears that when the system uploads the timetable overnight / reboots there is a failure. Both happened at around 0400.

 

[Taunton]

Everything's back running normally now. No need to panic and rip out CBTC and replace it with ETCS.

Well we wonder about this, because in the two years of operation it never seems to have been satisfactorily cracked what the transition problem is. Time after time we are just blandly told "signal failure", but it seems the issue is regularly that eastbound trains do not make the signal system transition at Paddington platform, and everything fails from there. As there does not seem to be any comparable issue at Stratford westbound, one presumes an analysis has been made of why one is less reliable than the other, but there has been no meaningful word.

For the information of the "fixed now, don't ask any more" team, this has caused substantial disruption to the business offices, not only in Canary Wharf but also in The City, for the last two days, with staff either grossly late or actually turning round back home to work. It's been of such a scale it should really come to the attention of the government, but it seems the DfT is incapable of understanding the issue beyond two words "signal failure".

 

[Horizon22]

Well we wonder about this, because in the two years of operation it never seems to have been satisfactorily cracked what the transition problem is. Time after time we are just blandly told "signal failure", but it seems the issue is regularly that eastbound trains do not make the signal system transition at Paddington platform, and everything fails from there. As there does not seem to be any comparable issue at Stratford westbound, one presumes an analysis has been made of why one is less reliable than the other, but there has been no meaningful word.

Transition issues are now few and far between. The majority of the issues have been train software and comms issues, and subsequent firmware updates on trains have significantly reduced the number of incidents.

For the information of the "fixed now, don't ask any more" team, this has caused substantial disruption to the business offices, not only in Canary Wharf but also in The City, for the last two days, with staff either grossly late or actually turning round back home to work. It's been of such a scale it should really come to the attention of the government, but it seems the DfT is incapable of understanding the issue beyond two words "signal failure".

I don’t know how you think it hasn’t come to the government or more appropriately the London Assembly. Naturally Siemens will investigate the nature of the fault, but naturally they may not wish to broadcast the specifics of it. I imagine it’s a sensitive nature of some description.

It’s only just been resolved; I’m sure a report will be forthcoming in time.

Fundamentally it was a “signal failure” (TfL used “signalling systems fault”). Just a very critical one resulting in temporary complete failure.

 

[Purple Train]

Transition issues are now few and far between. The majority of the issues have been train software and comms issues, and subsequent firmware updates on trains have significantly reduced the number of incidents.



I don’t know how you think it hasn’t come to the government or more appropriately the London Assembly. Naturally Siemens will investigate the nature of the fault, but naturally they may not wish to broadcast the specifics of it. I imagine it’s a sensitive nature of some description.

It’s only just been resolved; I’m sure a report will be forthcoming in time.

Fundamentally it was a “signal failure” (TfL used “signalling systems fault”). Just a very critical one resulting in temporary complete failure.

It was a "temporary fault with the signalling equipment" according to the auto-announcer on the train this morning. It did raise a hollow laugh, as I sure hope it isn't permanent!

 

[talldave]

I wonder what page this is?

An Internet page I have to search for every time I go to do a journey, or even look at as I go along. That doesn't particularly work at all when down in the Underground.

The platform train describers? They don't have any room any longer, given they are full of incessant See It Say It Sorted messages.

"If you see anything suspicious" - you mean like a running train???

 

[notverydeep]

Everything's back running normally now. No need to panic and rip out CBTC and replace it with ETCS.

Isn't ETCS just a specific form of / set of standards for CBTC (Communications Based Train Control)?

And more generally, isn't ETCS a way to apply long standing Metro-like signalling principles to main line railways across Europe? But then I plan metro services, so I would say that!

 

[riceuten]

Didn't take long for someone to personally blame Sadiq for the issue. I am sure he personally ordered the failure.

Can we cease with this petty point-scoring, unless you can prove his personal involvement in the issue?

 

[SECR263]

I know what you mean, but EL is still operating west of Paddington and east of Liverpool St/Stratford.
So not as obst

Travelled in on Thameslink and Southern caught a Jubilee in morning, now on DLR home. No onboard announcement on trains or Station Liz line is

Yes....that thought had also crossed my mind. A Russian cyber attack in retaliation for Ukraine having been given the go-ahead to fire British-supplied long-range missiles into Russian territory?

Why would Russia bother, Liz line regularly shoots it self in foot on frequent occasions.

 

[Tio Terry]

I retired over seven years ago now so haven't had any dealings with Crossrail in all that time.

Back then there were ongoings discussions revolving around the fact that Crossrail Limited had bought Siemens CBTC fixed equipment but that Boris had insisted that TFL purchase the trains and bought Bombardier trains because they would be British made in Derby.

Trouble was their onboard signalling system, from Alstom if I remember correctly, wasn't compatible with the Siemens product. Nobody had a contract to interface the two systems and neither company really wanted to share their intellectual property rights. Net result was a long delay and a massive amount of money to write bespoke safety critical software in order to interface the two systems.

I wonder if the recent problems revolve around this bespoke software?

 

[jon0844]

Didn't take long for someone to personally blame Sadiq for the issue. I am sure he personally ordered the failure.

The people who want the railway to become totally driverless are presumably the same people who want to return to the glorious days of rail, with steam trains managed by semaphore signals...

 

[PudseyBearHST]

Could someone please explain the difference between ETCS and CBTC and how both systems work? Thanks

 

[Class15]

Noticed that only one signal-route is set at a time in the core. That would mean yellow signals everywhere but I assume it’s a completely different system.

 

[Winthorpe]

This must have been a truly awful thing to deal with on the software side.

It is very complex software (am I right four signal systems?) that is safely critical. If it goes wrong, there’s no such thing as a quick fix with degraded working.

Find the fault, be properly sure the fault is understood, then be sure the solution works.

Testing alone must be a nightmare.