NRE broken?

[Adam Williams]

Is it me or are the static pages that display restriction codes just completely broken at the moment?



It should literally be a page with some static text on it describing a ticket's restrictions.

How do you screw up something so basic that pax will be relying on en-route? You could literally host this as a bunch of static HTML pages from a CDN.

NRE is really a joke at this point - the site isn't even responsive (in 2021!), you have issues like this, it's full of ads, the apps have been left broken and unmaintained... I could go on!

 

[Joe Paxton]

[...]
NRE is really a joke at this point - the site isn't even responsive (in 2021!), you have issues like this, it's full of ads, the apps have been left broken and unmaintained... I could go on!

I suspect the current unsatisfactory situation could drag on until GBR comes into being. (I can also well imagine the early stages of GBR's existence being quite messy too...)

 

[Adam Williams]

This does now appear to be resolved, so it was broken seemingly continuously for ~ two days? Apparently this also affected the "ticket and validity finder" tool.

I think I'd be shot if a service I maintained was completely inaccessible for two days with no customer communication!

As far as I can tell there's no NRE statuspage either and I didn't see any mention of this on Twitter so the outage was completely undocumented.

 

[Watershed]

There appears to be no appreciation of the fact that these pages are effectively a contractual appendix. NRE is indeed wholly inadequate but it's one of those behemoths that no-one wants to change until they're forced to.

 

[merry]

There has been a very large core DNS service outage affecting websites (and part sites) globally, so it's not surprising that some part of NRE (or other rail operator sites) might have been out for a while. DNS issues can, by their nature, take up to 72 hours to resolve globally (as updates propagate). Annoying, but an artefact of how the internet actually works.

 

[Adam Williams]

There has been a very large core DNS service outage affecting websites (and part sites) globally, so it's not surprising that some part of NRE (or other rail operator sites) might have been out for a while. DNS issues can, by their nature, take up to 72 hours to resolve globally (as updates propagate). Annoying, but an artefact of how the internet actually works.

This was not a DNS issue - at least not on the face of it, it would have had to have resolved to something that was serving up the server error page I screenshotted above otherwise I'd have got a browser error.


In fact we can prove that NRE doesn't use Akami's Edge DNS service quite easily:

$ dig nationalrail.co.uk ns

; <<>> DiG 9.16.1-Ubuntu <<>> nationalrail.co.uk ns
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7392
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;nationalrail.co.uk.        IN    NS

;; ANSWER SECTION:
nationalrail.co.uk.    21599    IN    NS    ns1.netnames.net.
nationalrail.co.uk.    21599    IN    NS    ns2.netnames.net.
nationalrail.co.uk.    21599    IN    NS    ns5.netnames.net.
nationalrail.co.uk.    21599    IN    NS    ns6.netnames.net.
nationalrail.co.uk.    21599    IN    NS    ns7.netnames.net.
nationalrail.co.uk.    21599    IN    NS    ns8.netnames.net.

Looks like DNS is handled by https://www.cscdbs.com/en/domain-security/dns-services/

Ignoring the above and thinking about it logically though: Obviously I don't know how it's architectured after it hits whatever served up my error page, but if something was accessible on port 80/443 then it should've been technically possible to sort this out without a further DNS change. There was a web server there, and it worked enough to serve me something.

 

[gallafent]

In fact we can prove that NRE doesn't use Akami's Edge DNS service quite easily

Perhaps some backend service which the NRE webserver uses to fetch the data it needs to compose those ticket validity pages is served by an affected server, though … all that you showed was that the public facing servers below nationalrail.co.uk are not.

 

[Adam Williams]

Perhaps some backend service which the NRE webserver uses to fetch the data it needs to compose those ticket validity pages is served by an affected server, though … all that you showed was that the public facing servers below nationalrail.co.uk are not.

Entirely possible!

But if, as you acknowledge, there was a working frontend NRE webserver then it could've been rigged up to at least return some static content that didn't involve hitting that backend for the duration of the outage. We have also concluded that the DNS server for the domain was functional (and the A record has a reasonably short TTL), so worst case it could've been pointed at a static site host/CDN to provide continuity of service for e.g. the short nre.co.uk/ URLs printed on the ticket.

As Watershed pointed out, this effectively forms part of the passenger's contract to travel. It's really poor for it to be down for this duration of time with zero comms, and there are multiple things that could've been done as a stop-gap to make the information available that were not.

No status page, no workarounds attempted, limited customer comms, long downtime window, no post-mortem write-up => poor engineering culture.

 

[gallafent]

But if, as you acknowledge, there was a working frontend NRE webserver then it could've been rigged up to at least return some static content that didn't involve hitting that backend for the duration of the outage.

Oh yes, totally agree, something (and ideally something meaningful, even if it's just a single static page with a table containing all the validity codes, for example, which would be really easy to shoehorn onto the frontend, even if the backend had fallen apart completely, you'd hope) should have been put in place while the existing backend was down, clearly! — was just noting that the DNS for the frontend being OK didn't preclude something breaking further down the stack because of the big Akamai outage!

 

[WelshBluebird]

The original post was last Wedneaday, Akamai issues were on Thursday. I know those issues affected a lot (we had a nervous 15 mins at work before it became obvious it was Akamai causing the issues) but it wouldn't have affected anything the day beforehand.

 

[Adam Williams]

was just noting that the DNS for the frontend being OK didn't preclude something breaking further down the stack because of the big Akamai outage!

No, it was a good point!

Can't make assumptions about any networky stuff beyond the frontend without seeing the code