The house is built of ashlar in three storeys to a square plan (9-bay x 8-bay) on a raised basement with advanced corner towers.
The house and the associated Fife gates, walled garden, Collie lodge, mausoleum, ice house, Bridge Gates House and the Eagles Gate lodge are designated as Category A listed group of buildings.
Duff House was designed by William Adam and built between 1735 and 1740 for William Duff of Braco. It was originally intended to have flanking pavilions linked by colonnaded quadrants but these were never completed. David Bryce Junior was later commissioned to provide a 3-storey pavilion and corridor block, but this was damaged by a bomb in 1941 and subsequently demolished.
The Earls of Fife moved out of Duff House in 1903, gifting the property to Banff Burgh in 1906. Since then Duff House has been in turn a palm court hotel, a sanatorium, a prisoner of war camp and a barracks but still sits in a designed landscape, albeit with addition of a golf course. Since 1995 it has been part of the National Galleries of Scotland.