It was built by the architect earl Henry Herbert in 1723–24 (under Colen Campbell and latterly his assistant Roger Morris), on ground leased by the earl in 1717 and 1729 amidst the ruins of the parts of Whitehall Palace that burned down in 1698 (and still covered in its rubble). Its design may have inspired the 9th earl's designs for Marble Hill House. The 9th earl died here in 1733, as did his great-grandson the 11th Earl, in 1827.
It was the subject of a major rebuild by the 10th Earl in 1756–59, and in 1762 Lady Hervey wrote that it was "taken for the Duc de Nivernois, the French Ambassador". Gardens were created in 1818 by demolishing the house's riding-house and stables, and the main floor-level terrace (including the portion over the water-gate) was retained. The lease was repeatedly renewed (passing to the Earl of Harrington) until in or around 1853, when the land and house became crown freehold (housing the Ministry of Transport c.1930, and later parts of what would become the Ministry of Defence). It was demolished to build the Ministry of Defence main building in 1938.
- Constable visited the terrace of Pembroke House in 1826 and painted the views from its riverside terrace in preparation for this painting, and it may be the house at extreme left, to the left of the tree (although this may also Michael Angelo Taylor's House).
- "Marble Hill House". English Heritage. Retrieved 26 January 2019.
- "MOD London Offices". Ministry of Defence. Archived from the original on 18 October 2012. Retrieved 11 September 2017.
- Steven Brindle, 'Pembroke House, Whitehall', in The Georgian Group Journal, vol. VIII, 1998, pp.88-113.
- 'Pembroke House', Survey of London: volume 13: St Margaret, Westminster, part II: Whitehall I (1930), pp. 167-179.
- 'Whitehall: Precinct and gardens', Old and New London: Volume 3 (1878), pp. 376-382.
- Pembroke House in "The Opening of Westminster Bridge" by Constable
- THE FIRST PEMBROKE HOUSE. ELEVATION AND PLAN