In the early 17th century, work began on transforming some of Lord Seymour's mansion into a grand stately house comparable with the great contemporary houses being built elsewhere in England. This work was of Lord Seymour's son Edward Seymour II from 1563 to 1613 and he inherited Berry Pomroy in 1593. Immense scale ambitious project replacing old hose north range with grand staircase, galleries which lead to luxurious new rooms in the east range and raising the north wing to three storeys high. It is unclear why he wanted to undertake such extravagant building work or how he intended to fund them but in any case building came to an abrupt halt. In 1611 they had serious financial trouble and last contribution to the fragile status of his family he purchased a baronetcy. So when he died in 1613 he was Sir Edward Seymour, first baronet of Berry Pomeroy with his home unfinished never to be completed.
Neither Sir Edward Seymour III, second baronet from 1572 to 1659, nor his son Sir Edward IV from 1610 to 1688, generally known as 'Colonel Edward', made any attempt to complete their predecessor's ambitious works. Indeed an inventory of 1688 shows that the family retreated into an updated version of the old Elizabethan mansion. Though both Sir Edwards were enthusiastic Royalists during the Civil War, the castle itself was never garrisoned or attacked. Only the help of the Colonel's beloved and formidable wife, Anne, that Berry Pomeroy was kept out of government hands during the Commonwealth, whereafter Sir Edward was loyal and highly honoured by Charles II at the Restoration. The old cavalier dispensed famous hospitality at Berry to all comers, and continued in government service well into his seventies.
In 1697 John Prince stated that Berry Pomeroy Castle was now demolished, and buried in its own ruins. This demolition was deliberately carried out by the new owner who was Sir Edward IV's son Sir Edward Seymour V from 1633 to 1708. Known as Speaker Seymour from his holding Parliamentary office in 1673 and he had quarrelled harshfully with both his parents as a young man about his political concerns in London.
The whole timber beams were removed, flagstone floors lifted up, decorated stonework was removed, lead and tiles stripped from roofs, glass torn from windows where the leading was then melted down. The castle remained property of the Seymour baronets who by then had permanently settled in Maiden Bradley in Wiltshire. In 1750 at last succeeded to the the family title of Duke of Somerset. It is still owned by the present duke, a direct descendant of the Berry Pomeroy Seymours.
In the late 18th century, the supremely picturesque ruins of Berry Pomeroy became a magnet for artists and sightseers. The development of nearby Torquay and later Paignton as holiday resorts brought many visitors to admire the wonderful ruins and speculate about their supposedly sinister history. This is mainly due to the exaggerated novels and gothic tale of jealousy, multiple murder and ghostly revenge that was published in 1806 and re issued with a succession of late Victorian romantic novel sets. Even today it is unfortunately referred to as the most haunted castle in Devon
The castle is suppose to have various spirits that reside there. Alot of ghostly phenomena and appariations have been reported in the castle. The White Lady is the restless soul of Margaret Pomeroy and she haunts the dungeons of St Margaret's Tower as she waves to visitors. She was held captive by her own sister, Eleanor because of jealousy and starved to death. The Blue Lady lures people into various parts of the castle to get them lost. People believe that she is the ghost of the daughter of one of the Norman castle lords. She was raped by her father and became pregnant with his baby. Her father strangled the child in one of the rooms of the castle. Other tales say that it was she who strangled the child. She has become the death warning of the Seymours. There have been sightings of a Cavalier and a lady wearing a gray dress as well as ghostly shadows. It has been witnessed that strange lights appear to be on in the castle, voices are heard and some cold spots.