The objective of service specification is to maximise social benefit, normally by minimising customer generalised journey time (within the network's capability and available resources of course). Forgive me if you are already familiar with the concept of generalised journey time, but this is where the journey time includes a valuation of the 'weightings' that customers attach to different elements of journey time, as well as the value of non-time elements. An example of weightings would be that an average customer prefers a minute spent on a moving train compared to a minute waiting at the platform, or a minute in a seat compared to a minute standing on a crowded train. Non time elements includes things like how comfortable the seats are, the fare paid and how secure they feel on board.
The DfT does set the framework by which these things are evaluated through its Transport Analysis Guidance (
https://www.gov.uk/guidance/transport-analysis-guidance-tag). TfL then uses a document called the Business Case Development Manual to implement this framework on its own projects (linked here:
https://tfl.gov.uk/corporate/transp.../foi-request-detail?referenceId=FOI-4306-1718). This expands on the DfT guidance includes parameters specific to metro operation and TfL that are set by TfL using its own research or research it has commissioned. The specification of new train types would follow this framework of evaluation and TfL's normal governance processes. The National Rail network has its own version of the framework called the Passenger Demand Forecasting Handbook (PDFH), some details here:
https://www.raildeliverygroup.com/pdfc/about-the-pdfh.html.
Completely agree with you on double-ended routes. These make much better use of Central London infrastructure and train fleets, hence the Elizabeth line and proposed Bakerloo line extension and Crossrail 2. The weakness of Thameslink in this context, is that it is trying to serve a selection of more regional markets (almost Inter-City in the case of Brighton to Cambridge), where customers have a higher expectation of comfortable seating, toilets etc. that are less important for metro trips. As you say - almost all services end up being a compromise between the competing needs of different passenger groups...
[apologies for the essay!]