It is said that this family came to Britain with William the Conqueror in the person of Walter, Count de Longueville. However, Barrow states the family were dependents of de Varenne (or de Warenne) and simply came from Longueville-la-Gifart in Seine Maritime. The East Lothian village of Gifford and a nearby stream, Gifford Water, both take their names from this family.
Two of this family appeared in Scotland in the train of Ada de Warenne, daughter of William de Warenne, 2nd Earl of Surrey (she married in 1139  Prince Henry (d.1152) son of David I of Scotland (d.1153): Hugh (1) and William, a cleric, who became "ecclesiastical advisor" to King David I. Hugh (whom Martine calls "an Englishman") obtained lands in East Lothian, where he settled. William perambulated with King David in Perthshire and was a witness on many important charters, notably the foundation charter of Jedburgh Abbey and others for Countess Ada. He and Hugh witnessed a charter of King David granting lands at Crail. From Malcolm IV of Scotland he obtained lands at Yester (Jhestrith) in the parish of St. Bothans, East Lothian., Hugh appears in further royal charters till after 1189, and appears to have witnessed many of the charters of King William the Lion (ruled 1165–1214), under whom he rose to distinction.
His son and heir was William de Giffard of Yester, who was sent on a mission to England in 1200 and who also witnessed several charters of William, 'The Lion'. In 1244 he was one of the guarantees of a treaty with England, when he must have been a good age.
His son, Hugh de Giffard of Yester (2), was one of the Guardians of Alexander III of Scotland and his Queen; and one of the Regents of the Kingdom appointed by the Treaty of Roxburgh dated 20 September 1255. According to the practice of those feudal times, he had his own sheriff. This Hugh de Gifford built a castle, or tower house, at Yester (half a mile south-east of the present-day Yester House) on a promontory between the Hopes Water and a little tributary, the Gamuelston Burn. To the north is one side of a high curtain wall with an offset base. Beneath it a stair descends to the cavern called Bohall or Hobgoblin Hall, (Goblin Ha'), featured in Sir Walter Scott's Marmion, which Gifford was said to have constructed by magic.
Walter Bower thus speaks of him in noting his death in 1267: "Hugo Giffard dominus de Zester moritur, cujus castrum, vel saltem caveam et dongionem, arte demoniacula antiquae relationes fuerunt fabricatas," (Scotichronicon, Liber X, cap. 21). Anderson states he left three sons:
- William de Giffard of Yester
- Hugh (3), ancestor of the Giffards of Sheriffhall.
- James, who, with Hugh (4), swore fealty to King Edward I of England in 1296.
Burke also gives four daughters, (although it is possible they belong to a later generation) of whom:
- Euphemia, who married Sir Archibald Macdowall, Knt., of Makerstoun, Roxburghshire.
- Johanna, married Sir Robert Maitland of Thirlestane, to whom she conveyed Lethington. Ancestors of the Earls of Lauderdale
- Jonat, who married Adam de Seton, Master-Clerk.
- Anderson, Alan O., Scottish Annals from English Chroniclers AD500 - 1286, London, 1908: 262.
- Anderson, Alan O., Scottish Annals from English Chroniclers AD500 - 1286, London, 1908: 215.
- Harvey, Charles C.H., & MacLeod, John, editors,Calendar of Writs preserved at Yester House 1166–1625, Scottish Record Society, Edinburgh, 1930, p.1, no.1, see charter dated between 1166–1171 by William The Lion who states that Hugh held these lands "by grant from my brother King Malcolm and Ada the Countess, my mother".
- Anderson, Alan O., Scottish Annals from English Chroniclers AD500 - 1286, London, 1908: 355.
- History of the Commoners of Great Britain and Ireland etc., by John Burke, London, 1836, vol. 3, pps: 434-5.
- The Scottish Nation by William Anderson, Edinburgh, 1867, vol.V, pps: 297-8.
- Fourteen Parishes of the County of Haddington, by John Martine, Edinburgh, 1890, pps: 292-5.
- The Normans in Scotland, by R.L.Graeme Ritchie, Edinburgh University Press, 1954, p. 276.
- The Buildings of Scotland - Lothian (except Edinburgh), by Colin McWilliam, Penguin Books Ltd., London, 1978, pps:208-215. ISBN 0-14-071066-3
- The Anglo-Norman Era in Scottish History, by Professor G.W.S.Barrow, Clarendon Press, Oxford, 1980, p. 43. ISBN 0-19-822473-7
- The Origin of the Gifford Family in Scotland by G.Charles-Edwards, in The Scottish Genealogist, vol.XXVII No.4, Edinburgh, December 1980, pps: 152-161. ISSN 0300-337X