The earliest known house on the site was Berkshire House, built in about 1626-27 for Thomas Howard, second son of the Earl of Suffolk and Master of the Horse to Charles I of England when he was Prince of Wales. Howard was later created Earl of Berkshire.
After being occupied by Parliamentarian troops in the English Civil War, used for the Portuguese Embassy, and lived in by Edward Hyde, 1st Earl of Clarendon, the house was lived in by Charles II's mistress Barbara Villiers, who was made Duchess of Cleveland in 1670, following which the house was known as Cleveland House. She refaced the old house and added new wings. After being owned for some years by a speculator, the house was sold in 1700 to John Egerton, 3rd Earl of Bridgwater, after which it passed by inheritance until 1948.
Cleveland House was re-designed in the Palazzo style by Sir Charles Barry in 1840. The rebuilding was completed and renamed in 1854 for Lord Ellesmere, heir of the 3rd Duke of Bridgewater. It is built in Bath stone with a slate roof in three storeys with a basement.
The building was damaged in the Second World War by a bomb in the street and was subsequently adapted for office use. It was sold by John Egerton, 6th Duke of Sutherland. By 1979, it was the corporate head office of Tube Investments, the British engineering and metals conglomerate.
The Marquess of Stafford had Cleveland House extended in 1803-1806 by architect Charles Heathcote Tatham to accommodate the Stafford Gallery (renamed the Bridgwater Gallery in Bridgwater House), where the collections of paintings of the Duke of Bridgwater and his nephew and heir George Leveson-Gower, 1st Duke of Sutherland (whose second son Ellesmere was) were on at least semi-public display.
The collection included about 70 paintings from the famous Orleans Collection, some of which are now in the Sutherland Loan to the National Gallery of Scotland. The collection was opened in 1803, and it could be visited on Wednesday afternoons over four, later three, months in the summer by "acquaintances" of a member of the family, or artists recommended by a member of the Royal Academy.
The painting Charles I Insulted by Cromwell's Soldiers, thought lost in a World War II air raid, was rediscovered in 2009.
- "Images of England: Bridgewater House, Westminster". Historic England. Retrieved 22 August 2008.
- Survey of London which has a full history.
- "John Latsis". The Independent. London. 18 April 2003.
- Nellis Richter, Anne. "Changing Subjects: The Gallery at Cleveland House and the Highland Clearances". British Art Studies.
- Penny, Nicholas, National Gallery Catalogues (new series): The Sixteenth Century Italian Paintings, Volume II, Venice 1540-1600, p. 468, 2008, National Gallery Publications Ltd, ISBN 1-85709-913-3
- Stourton, James (2012). Great Houses of London (Hardback). London: Frances Lincoln. ISBN 978-0-7112-3366-9.