The rules are not as clear cut as they should be.
Firstly I believe you could claim a partial refund, based on abandoning your journey and returning to your origin, the rules state:
Part F: Your Refund and Compensation Rights
29. Refunds on Tickets Which You Have Chosen Not to Use or Have Been Part-used
29.1 If you purchase any Ticket(s) and then choose not to travel, you may apply for a refund
from the original retailer, unless the terms and conditions of your Ticket(s) state
otherwise. You must do this no more than 28 days after the expiry of the Ticket(s)
unless the terms and conditions of your Ticket state otherwise.
29.2 In such cases, a deduction from your refund in the case of part-used Tickets, will
normally be calculated on the cost of the journey(s) actually made. An administration
charge may also be made, which will not exceed £10 per Ticket; however, if the
administrative charge and/or other deductions exceed the refund amount no refund
will be payable. If a delay or cancellation on any leg of your journey is the reason you
could not complete your journey, you are entitled to a full refund on any Tickets held
for that journey under condition 30.1
29.3 Your refund application will be processed as soon as reasonably practicable, and the
refund paid within one month of receipt of your application.
As you did get the use out of your ticket - in part - to visit a location that was en-route, I believe you could argue that you should be entitled to the difference between the cost of just a ticket to the place you did visit, and the amount you paid.
There is also a strong argument to say that you would further be entitled to delay compensation of the non- refunded amount, assuming 15 mins is a qualifying period for such a delay.
Some people would argue that you could alternatively claim each section of the journey as Delay Repay, although if you didn't actually make use of the ticket, due to the delay, beyond the first destination, it's generally accepted that's not what should happen.
Some people have claimed the full fare can be refunded but I don't think that is the case where the ticket has bee part used to visit a destination short of the final destination of the ticket (if that was correct people could claim full refunds very easily on many occasions!)
Was it all one TOC involved? I would be tempted to put in an email to customer services explaining what happened and hopefully they will provide a reasonable level of compensation.