Invergarry Castle was the seat of the Chiefs of the MacDonells of Glengarry,a powerful branch of the Clan MacDonald. It was burned down in 1654 byGeneral Monk, then re-built c.1660-1665. After the 1745 uprising Invergarry Castle was sacked and partially destroyed by troops under "Butcher" Cumberland as part of the systematic suppression of the Highlands.

Edward "Bear" Ellice (1781-1863) was a Director of the Hudson Bay company, which traded throughout the Americas. His son Edward Ellice (1810-1880) later became deputy governor of the company. Edward "Bear" Ellice was also to become the Member of Parliament  epresenting Coventry in the House of Commons, 1818 to 1826 and 1830-1863. He added the Glengarry portion of the estate, including Invergarry and lands, to his Glenquoich holding in 1860.

Invergarry House, later re-named Glengarry Castle Hotel, was built in
1866-1869 by celebrated architect David Bryce for Edward Ellice Jnr. David Bryce built over 100 Baronial Mansions and his other works include Fettes College, The Royal Infirmary and The Bank of Scotland all of which are in Edinburgh.

Each David Bryce house or project was unique, but used elements drawn from a familiar repertoire including canted bay windows and turelles - small round towers. An unusual feature in Invergarry House is the Canadian pitch-pine panelling which dominates the entrance hall.

Inscriptions can also be found on stone plaques set into the outer walls of the house:

• Mur tog an tighearna an tigh gu diomhain saoithrichidh a luchd togail.
  (Psalm 127: v1) (Except the Lord build the house, they labour in vain that build it.)
• With thy blessing let the house of thy servant be blessed forever. (II Samuel Chp 7: v29)
• A merciful man will be merciful to his beast (Old Proverb)
• An cuir am muchadh agus cha bhi thu'n dith (Do not smoother him and there will not be, or Do not extinguish or suffocate and you will not be in need)

From its completion in 1869, Invergarry House was occupied by the Ellice family until 1923, subsequently by Sir Fredrick Wills, Sir Mortimer Singer and W.U. Goodbody. It then passed into the possession of the Hydro-Electric Board until purchased by the three Cameron sisters.