The Danvers family of Swithland, Leicestershire was a well‑established line of landed gentry by the first half of the 18th century. Sir Joseph Danvers, 1st Baronet (1686-1753) married Frances Babington (1694-1759) at St Paul’s Cathedral in 1721. Sir Joseph was an active Whig politician, serving as Member of Parliament for several constituencies between 1722 and 1747, and was created a baronet in 1746. The family seat, Swithland Hall, lay at the heart of the Leicestershire estates long associated with the Danvers name and remained a visible symbol of their position within the political and social elite.

Following the gradual extinction of the direct male Danvers line, the family’s estates, associations and heirlooms passed through the female line into allied noble families. Of particular significance is the marriage of Mary Danvers, granddaughter of Sir Joseph, to Augustus Richard Butler, later known as Butler‑Danvers, a younger son of Brinsley Butler, 2nd Earl of Lanesborough. Their son George (1794-1866) succeeded as 5th Earl of Lanesborough, uniting the Danvers inheritance with that of the Butler family. It was he who commissioned the building of a new mansion to replace an earlier residence. Through this descent, the two portrait groups of the Danvers family and the single portrait of one of the daughters entered the collections of the Earls of Lanesborough, where they remained until the late 1970s when they were purchased by Francis Anthony Brinsley ('Jimmy') Valentine, nephew of Denis, the 9th Earl.