If the South West is included in your "down south" from Birmingham remit, then how about Totnes?
Slightly quirky "alternative" town with attractive buildings and motte & bailey castle. Market day Friday, lots of independent shops, it gets better as you climb the High Street towards the castle...