Heathrow have put out an announcement, and since are giving briefings and have the transport secretary doing similarly, which seem highly dubious and very much going on the offensive in response to criticism of their exposure to this not unforeseeable event. I find myself questioning the veracity of their claims.
Black swan incidents do happen from time to time, fallback systems fail, the perfect storm of issues coincides, and perhaps this is such an event – but I am also very interested in whether decision making arising from misaligned incentives (per Willie Walsh), "enterprise risk frameworks" and the impact of everything else that tries to squeeze the life out of hard engineering decisions so as to make them just another line item around a boardroom table.
I know as much as the next person on the decisions that led to this event, but the fact of the matter is that a single substation failure seems to have had cascading effects on the ability for a piece of "critical national infrastructure" to operate at all – and they didn't have any means to recover quickly. I don't see any other way to skin it.
It is rather frustrating when this quickly becomes a PR game of cover, deflect and pacify rather than accept, apologise and commit to better. This seems so prevalent in our services and institutions today (it is rare to ever receive an apology) and is a good reminder of how centralisation and standardisation reduces the resilience of a system in handling anomalous events without complete collapse.
Quoting from the piece in the press, with my commentary:
It's trivially possible for this to be true without it actually meaning much – are those supplies intended to be resilient and interchangeable, or separate supplies for separate purposes? The rail network could still operate through the site with traction feeders from elsewhere – satisfying the definition of "multiple sources".
Great – so this sort of confirms the above in that it's not intended as a redundant system ("a source is interrupted" => run on diesel). Safety of flight wasn't impacted, which everyone keeps reminding us about as if that should be a surprise, but the airport couldn't function effectively for a day.
Do they have multiple HV feeders, from separate sources, maybe connected in a ring, and N+1 or N+2 redundancy so they can lose one and still run the airport on full load? Apparently not, or at least they weren't working correctly.
Meaningless drivel about consuming as much energy as a small city, which seems to be intended to keep criticism at bay by some appeal to handwaving "it's complicated". Plenty of other energy demanding facilities (data centres, heck even actual urban areas) are typically capable of operating with loss of a single feeder...
It rather surprises me that, had this sort of eventuality been foreseen, this wasn't a tried, tested, practised and documented procedure such that it could have been implemented with far less time.
I will stand corrected if there are indeed legitimate reasons that, despite the above, things broke in unforeseen ways – but I hold my suspicions that some people may have egg on face today as a result of botched resilience arrangements or failure to adequately plan – with huge consequences for many travellers and employees.
I'm currently in Dallas. We were quite a few hours in before getting redirected back to Dallas last night.I wonder how many residents of Windsor, Hounslow and the Richmond area have overslept today…
A quick look suggests most Virgin and BA flights that were on the way to Heathrow have done a “Uey” and heading back to origin. A few have diverted - eg flights from Cape Town and Rio landing in Madrid.
Well summed up. As minimum the DfT in conjunction with DENZ should have an independent investigation into this but no sign of that currently.Heathrow have put out an announcement, and since are giving briefings and have the transport secretary doing similarly, which seem highly dubious and very much going on the offensive in response to criticism of their exposure to this not unforeseeable event. I find myself questioning the veracity of their claims.
Black swan incidents do happen from time to time, fallback systems fail, the perfect storm of issues coincides, and perhaps this is such an event – but I am also very interested in whether decision making arising from misaligned incentives (per Willie Walsh), "enterprise risk frameworks" and the impact of everything else that tries to squeeze the life out of hard engineering decisions so as to make them just another line item around a boardroom table.
I know as much as the next person on the decisions that led to this event, but the fact of the matter is that a single substation failure seems to have had cascading effects on the ability for a piece of "critical national infrastructure" to operate at all – and they didn't have any means to recover quickly. I don't see any other way to skin it.
It is rather frustrating when this quickly becomes a PR game of cover, deflect and pacify rather than accept, apologise and commit to better. This seems so prevalent in our services and institutions today (it is rare to ever receive an apology) and is a good reminder of how centralisation and standardisation reduces the resilience of a system in handling anomalous events without complete collapse.
Quoting from the piece in the press, with my commentary:
It's trivially possible for this to be true without it actually meaning much – are those supplies intended to be resilient and interchangeable, or separate supplies for separate purposes? The rail network could still operate through the site with traction feeders from elsewhere – satisfying the definition of "multiple sources".
Great – so this sort of confirms the above in that it's not intended as a redundant system ("a source is interrupted" => run on diesel). Safety of flight wasn't impacted, which everyone keeps reminding us about as if that should be a surprise, but the airport couldn't function effectively for a day.
Do they have multiple HV feeders, from separate sources, maybe connected in a ring, and N+1 or N+2 redundancy so they can lose one and still run the airport on full load? Apparently not, or at least they weren't working correctly.
Meaningless drivel about consuming as much energy as a small city, which seems to be intended to keep criticism at bay by some appeal to handwaving "it's complicated". Plenty of other energy demanding facilities (data centres, heck even actual urban areas) are typically capable of operating with loss of a single feeder...
It rather surprises me that, had this sort of eventuality been foreseen, this wasn't a tried, tested, practised and documented procedure such that it could have been implemented with far less time.
I will stand corrected if there are indeed legitimate reasons that, despite the above, things broke in unforeseen ways – but I hold my suspicions that some people may have egg on face today as a result of botched resilience arrangements or failure to adequately plan – with huge consequences for many travellers and employees.
I'm currently in Dallas. We were quite a few hours in before getting redirected back to Dallas last night.
It is not possible to insure or mitigate for every single event. Cost benefit will always play a role. When did this last happen?Heathrow have put out an announcement, and since are giving briefings and have the transport secretary doing similarly, which seem highly dubious and very much going on the offensive in response to criticism of their exposure to this not unforeseeable event. I find myself questioning the veracity of their claims.
Black swan incidents do happen from time to time, fallback systems fail, the perfect storm of issues coincides, and perhaps this is such an event – but I am also very interested in whether decision making arising from misaligned incentives (per Willie Walsh), "enterprise risk frameworks" and the impact of everything else that tries to squeeze the life out of hard engineering decisions so as to make them just another line item around a boardroom table.
I know as much as the next person on the decisions that led to this event, but the fact of the matter is that a single substation failure seems to have had cascading effects on the ability for a piece of "critical national infrastructure" to operate at all – and they didn't have any means to recover quickly. I don't see any other way to skin it.
It is rather frustrating when this quickly becomes a PR game of cover, deflect and pacify rather than accept, apologise and commit to better. This seems so prevalent in our services and institutions today (it is rare to ever receive an apology) and is a good reminder of how centralisation and standardisation reduces the resilience of a system in handling anomalous events without complete collapse.
Quoting from the piece in the press, with my commentary:
It's trivially possible for this to be true without it actually meaning much – are those supplies intended to be resilient and interchangeable, or separate supplies for separate purposes? The rail network could still operate through the site with traction feeders from elsewhere – satisfying the definition of "multiple sources".
Great – so this sort of confirms the above in that it's not intended as a redundant system ("a source is interrupted" => run on diesel). Safety of flight wasn't impacted, which everyone keeps reminding us about as if that should be a surprise, but the airport couldn't function effectively for a day.
Do they have multiple HV feeders, from separate sources, maybe connected in a ring, and N+1 or N+2 redundancy so they can lose one and still run the airport on full load? Apparently not, or at least they weren't working correctly.
Meaningless drivel about consuming as much energy as a small city, which seems to be intended to keep criticism at bay by some appeal to handwaving "it's complicated". Plenty of other energy demanding facilities (data centres, heck even actual urban areas) are typically capable of operating with loss of a single feeder...
It rather surprises me that, had this sort of eventuality been foreseen, this wasn't a tried, tested, practised and documented procedure such that it could have been implemented with far less time.
I will stand corrected if there are indeed legitimate reasons that, despite the above, things broke in unforeseen ways – but I hold my suspicions that some people may have egg on face today as a result of botched resilience arrangements or failure to adequately plan – with huge consequences for many travellers and employees.
Do they have multiple HV feeders, from separate sources, maybe connected in a ring, and N+1 or N+2 redundancy so they can lose one and still run the airport on full load? Apparently not, or at least they weren't working correctly.
A failure of an electrical infeed is a perfectly plausible scenario that should be protected against. A big signalling centre has at least three independent supplies plus would have a back up generator for such mission critical infrastructure. Heathrow should be similarly equipped. DfT and DENZ should be ordering an independent review of why Heathrow found itself in this position,It is not possible to insure or mitigate for every single event. Cost benefit will always play a role. When did this last happen?
In reality there was some inconvenience for passengers which can be insured against and a contingency plan which looks to have worked well.
Heathrow response could be just two words. 'Nobody died'.
I expect not all the airport lost power; and the whole-airport closure was more about maintaining control of the situation on the ground than an inability to operate everything everywhere.
They do have mitigation.I know nothing about how a site like Heathrow is powered. However I find it baffling that Heathrow wouldn’t have some sort of power failure mitigation and it wasn’t regularly tested. It’s so bizarre that it makes me feel there’s something else going on other than a routine power failure. I realise that makes me sound like a tinfoil hatter…
There was also a report saying that the substation that failed was running over capacity due to a lot of new load in the area.
A data centre might have a UPS but it could be feeding critical infrastructure only or only be specified to run full load for a very short time.
This might be the report you require, originally issued in 2022 with updates in 2023 & 2024. Basically says lack of electricity supply was holding back connections to 12,579 homes across 3 west London boroughs (Hillingdon, Hounslow, Ealing)There was also a report saying that the substation that failed was running over capacity due to a lot of new load in the area.
3.2.2.
Heathrow Airport
Heathrow Airport is a strategic transport hub that is currently supplied by multiple GSPs in the area. As shown in Figure 3, North Hyde GSP supplies the area to the Northeast of the airport. There are also dedicated supplies to other parts of the Heathrow Airport site. SSEN work closely with Heathrow Airport to develop a co-ordinated strategic plan for their future needs, including decarbonisation. The impact of EVs traveling to Heathrow airport is explored in section 5.2.
Providing 24/7/365 resilient power to a global transport hub requires continuous innovation and dedication. This means around-the-clock maintenance, repairs, and renewals to thousands of high voltage electrical substations, transformers, relays, and more than 600km of cables.
30 Years in Aviation: Heathrow
Celebrating 30 Years in Aviation with Heathrow Airport. Over three decades, our long-term partnership with Heathrow has remained a testament to collaboration and shared innovation when delivering airport excellence. From delivering Terminal 5 to attaining Carbon Trust Level 3, we have supported...www.ukpowernetworksservices.co.uk
What they did was take a choice that losing a major part of the incomming natiional grid infrastructure at no notice for a significant time was a very low risk.
I know nothing about how a site like Heathrow is powered. However I find it baffling that Heathrow wouldn’t have some sort of power failure mitigation and it wasn’t regularly tested. It’s so bizarre that it makes me feel there’s something else going on other than a routine power failure. I realise that makes me sound like a tinfoil hatter…
There was also a report saying that the substation that failed was running over capacity due to a lot of new load in the area.
So do oil filled transformers have a bund around them like you need for fuel storage tanks? Probably not.
Usually, yes, large transformers are bunded. The arrangements are a little more complicated, too - the coolers are usually an external radiator-type array and there are often pumps and fans to help keep everything cool on newer transformers.So do oil filled transformers have a bund around them like you need for fuel storage tanks? Probably not.
By matching quotes I've located that report here - it was done for the extended northern runway proposal. But it's not exactly detailed; the relevant sections is (as OCRed; it can't be copied):Heathrow airport and the Government were warned a decade ago that the airport was overly reliant on very limited sources of power from the National Grid to keep it functioning, it has emerged.
A report by a global planning and infrastructure consultancy identified “the key weakness” to the airport’s ability to operate as being its dependence on just three electricity substations.
These cannot replace each other without lengthy reconfiguration
When the North Hyde substation in Hayes exploded and went up in flames at 11.20pm on Thursday it plunged the entire airport into chaos, forcing the diversion of thousands of flights and leaving tens of thousands of passengers stranded.
It was precisely the nightmare scenario predicted in a report by Jacobs consultants in 2014, which assessed the operational risks faced by Europe’s busiest international hub.
As Heathrow’s operations began to return to normal following its reopening on Friday night, its chief executive placed the blame on the external power failure and said similar disruption could happen at other airports.
Thomas Woldbye said he is “personally quite proud” of how the airport responded to the power outage that brought all flights to a halt.
He added: “This situation was not created at Heathrow airport. It was created outside the airport and we had to deal with the consequences.”
The Jacobs report, published by the Government, warned: “Beyond the management of supply and grid services, which lie outside the airport’s control, the responsibility for managing electricity supply risk lies with the airport and businesses operating from the airport.”
It warned: “While some services can be temporarily supported with generator or battery backups, the key weakness is the main transmission line connections to the airport.
“Outages could cause disruption to passengers, baggage and aircraft handling functions and could require closure of areas of affected terminals and potentially the entire airport.
“Even a brief interruption to electricity supplies could have a long-lasting impact as systems can take time to recover.”
2.4 Utility Outages
2.4.1 Electricity
Electricity is a critical utility for the airport. Outages could cause disruption to passenger, baggage and aircraft handling functions and could require closure of areas of affected terminals or potentially the entire airport. Even a brief interruption to electricity supplies could have a long-lasting impact as systems can take time to
recover.
Beyond the management of supply and grid services, which lie outside the airport's control, the responsibility for managing electricity supply risk lies with the airport and businesses operating from the airport. While some services can be temporarily supported with generator or battery backups, the key weakness is the main transmission line connections to the airport.
Heathrow is equipped with on-site generation and appears to have resilient electricity supplies that are compliant with regulations and standards. Its new energy centre is a biomass heat and power plant that enables both the T5 and T2 terminal complexes to be heated and powered from the plant, and also to feed the local power grid to support demand from other users.
It is not unreasonable to assume that the expansion of the airport would be accompanied with additional provision of resilient supplies and back-up generation or uninterruptible power supplies to serve the additional facilities. These additions could potentially enhance the resilience of existing facilities.
While the expansion of Heathrow would increase the number of passengers and aircraft affected by any power outage, its current provision of on-site generation and other measures to ensure resilient supply appear to be adequate to enable the expanded airport to withstand and recover from interruptions to supply.
Was absolutely fine; but on a packed Elizabeth line trainI'm travelling back from HK Saturday 21st. I'll let you know how I get on....
NESO’s investigation will support efforts to build a clear picture of the circumstances surrounding this incident and the UK’s energy resilience more broadly so that it’s prevented from ever happening again.
The UK Government Resilience Framework: 2023 Implementation Update (HTML)
"We are living through dangerous and volatile times. The risks we face are more complex - and they are evolving faster - than ever. Over the last year, the ongoing Russian aggression in Ukraine, increased cyber-attacks, early examples of artificial intelligence being misused, and extreme weather conditions have shown us the wide-ranging and long-lasting impact such risks have across our society."...
One of the documents referenced above is a National Risk Register with this goal in the introduction...
"By focusing on our collective resilience, we can
help the nation be more safe, more secure – and
in turn, more prosperous. This National Risk
Register plays a vital role in that process, allowing
us to build towards an even brighter future."
I work for an NHS ALB, pharmaceuticals production. In case of power failure we have a backup generator that runs the whole site, all critical equipment is on UPS that covers switchover. The generator is tested monthly, powering the site for 3 hours. We have enough diesel on site for 7 days.
I know nothing about how a site like Heathrow is powered. However I find it baffling that Heathrow wouldn’t have some sort of power failure mitigation and it wasn’t regularly tested. It’s so bizarre that it makes me feel there’s something else going on other than a routine power failure. I realise that makes me sound like a tinfoil hatter…
All the newer feeds to Heathrow from the bulk supply point to the south and west are 33kV (e.g. covering T5, cargo area, maintenance hangers, jet fuel storage, fire station) , the two oldest including the one that failed was 22kV and the other 22kV had to be turned off to fight the fire.Heathrow has grown over the years so perhaps a result of piecemeal growth, shoe horning into a site and not looking at the big overall Heathrow picture in terms of what happens if power fails?
If a plane has declared an emergency, for example due to them beginning to eat into their 30 minute final reserve fuel, any airfield will be made available to them regardless of whether it's civil, military etc.Can Farnborough not handle any of these planes or Northolt?
Can Farnborough not handle any of these planes or Northolt?