It is the site of one of England's most dramatic 12th-century sieges.  Rougemont House was built in 1769 by John Patch, a surgeon at the Devon and Exeter Hospital, on a site that had formerly been part of the moat and defences for the Norman castle.  The landscaping and the planting belong to the late 18th century, first by Thomas Patch of Rougemont House who died in 1787, developed by his successor, Edmund Granger, on the advice of William Jackson, a local architect, painter and musician. The grounds were bought by the city in 1912. Northernhay House was later demolished and the gardens of Rougemont House were linked to Northernhay Gardens.

Victorian writers called it 'a perfect sylvan retreat', 'a gem set in the heart of Exeter'.

Steps have recently been taken to protect and nurture some of the large mature trees, and as with the adjacent Northernhay Gardens the Council continues to focus on improving biodiversity at the site.  The garden is close to the city centre and provides an ideal location for lunches and a place for peace and quiet.  Access is via Castle Street and the gardens are located behind the Library.  The gardens are open from dawn till dusk.