He was born at Tickhill Castle, the son and heir of Frederick Lumley-Saville (1788–1837), the only son of the fifth son of the fourth Earl of Scarbrough. His mother, Charlotte Mary (née de la Poer Beresford), was the daughter of Rev. George Beresford, grandson of Marcus Beresford, 1st Earl of Tyrone. His father was 
Educated at Eton College, he entered the army as a Cornet in the Hussars, and retired in 1837 as a Lieutenant-Colonel in the West Yorkshire Yeoman Cavalry. After retirement he entered politics as an elected representative peer and sat in the House of Lords until he inherited the family titles in October 1856.
Lumley's predecessor, John Lumley-Savile, 8th Earl of Scarbrough, died childless. Lumley inherited the earldom of Scarbrough and barony of Lumley although he was only the first cousin once removed from the eighth earl. He also inherited the Viscountcy of Waterford in the Peerage of Ireland.
Scarbrough added considerably to the family seat of Sandbeck Park in Yorkshire, which dated to the 17th century. In 1857, he hired William Burn to remodel and improve the house. In 1869, Benjamin Ferrey built a private chapel for the earl.
They had three sons and four daughters. All four of his daughters married peers:
- Lady Algitha Frederica Mary (23 November 1847 – 22 June 1919), married William Orde-Powlett, 4th Baron Bolton
- Lady Ida Frances Annabella (born 28 November 1848 – 22 August 1936), married George Bridgeman, 4th Earl of Bradford
- Lyulph Richard Grandby William, Viscount Lumley (7 June 1850 – 23 August 1868)
- Lady Lilian Selina Elizabeth (20 October 1851 – 24 December 1943) married Lawrence Dundas, 1st Marquess of Zetland
- Lady Sibell Mary (25 March 1855 – 4 February 1929); married: firstly, Victor Grosvenor, Earl Grosvenor (28 April 1853 – 22 January 1884), with whom she had Hugh Grosvenor, 2nd Duke of Westminster; secondly George Wyndham MP, Chief Secretary for Ireland
- The Hon. Aldred Frederick George Beresford; later Viscount Lumley (16 November 1857 – 4 March 1945); married Cecilia Gardner
- Col. The Hon. Osbert Victor George Atheling (16 November 1857 – 14 December 1923), married Constance Ellinor Wilson-Patten, granddaughter of John Wilson-Patten, 1st Baron Winmarleigh; father of the 11th Earl
Scarbrough died at Sandbeck Park at age 71 and was succeeded in the earldom by his eldest surviving son.
- "Obituary". The Times. The Times Digital Archive. 6 December 1884. p. 10.
- Cokayne, George (1895). The Complete Peerage (Edition 1, Volume 7). archive.org. p. 77.
- Innes, Anne; Innes, Eliza; Innes, Maria. The Peerage and Baronetage of the British Empire as at Present Existing. Hurst and Blackett, Publishers. p. 526. Retrieved 21 September 2016.
- Cockayne, G.E., The Complete Peerage or a History of the House of Lords and All its Members From the Earliest Times, Vol. XI (London, 1888), 514-5.
- Journals of the House of Lords. H.M. Stationery Office. 1885. p. 92.
- Historic England. "Sandbeck Park (House) (1314665)". National Heritage List for England. Retrieved 21 September 2016.
- Lincolnshire Archives Office : Copy accounts of the Earl of Scarbrough with James Whitton re Skegness development 1878-1880.MISC DON 304/2
- Fox-Davies, Arthur Charles. Armorial Families: A Complete Peerage, Baronetage, and Knightage. Jack. p. 880. Retrieved 21 September 2016.
|Peerage of England|
| Earl of Scarbrough