I'm probably one of the few on here who has actually travelled through the site on a train (412xx tank, 2 carriages, 2 passengers) - the alignment of the S&D branch from Evercreech to Highbridge is the main track through the site. Ivo Peters' films show how it used to be, including the train stopping at the cottage at Cockmill Crossing to deliver drinking water, middle of nowhere, which is now in the centre of things.
Castle Cary for long had no main line trains stopping at all - only Weymouth line trains. Incidentally, the station is nowhere near the little town itself, the railway being down in the valley, the town up on a hill. Then just one evening train from Paddington started serving it, and things slowly built up from there. The closest station with an all-day service to London was actually Templecombe, on the Waterloo line, but that's even further from the festival site.
In the beginning Glastonbury Festival was decidedly for oddballs, Hippies and Druids, and became slowly mainstream from there. It started with the Druids who went to Stonehenge for the summer solstice, and were in the area the following week, many with offbeat lifestyles living in groups of old dilapidated buses that they roamed the country in, which would normally come to the immediate attention of the police were they not just keen on moving them on and getting them out of the area. I've never been to the Festival itself (shame, it's still on my mythical bucket list
) but am familiar with the various traffic and attendee issues. Most locals give the roads around the location a very wide berth for the week.