Also known as the Morse-Libby House, the Mansion was built between 1858 and 1860 as a summer home for Ruggles Sylvester Morse, a Maine native who made his fortune in New Orleans as the proprietor of luxury hotels. The quality and preservation of its interiors make Victoria Mansion an unparalleled example of pre-Civil War grandeur.

"Residence of R.S. Morse" from Guide Book for Portland and Vicinity by William Willis, 1859. This is the earliest known image of Victoria Mansion.

Morse died in 1893 and the following year his widow sold the house with most of its furnishings to J.R. Libby, a prominent dry goods merchant. The Libby family preserved the original decor and made few changes to the property. They occupied the Mansion until about 1929. After a hurricane damaged the Mansion in 1938, its future was in peril. In 1940 it was scheduled to be demolished and replaced by a gas station.

J. R Libby and family photographed against the south wall of the Parlor, Thanksgiving 1916.

At this crucial moment, retired educator Dr. William H. Holmes purchased the property to save it. Aided by his sister, Clara Holmes, he opened the building to the public in 1941 as a museum in honor of Queen Victoria. In 1943 the Society of Maine Women of Achievement was incorporated to assume control of the site. Now known as Victoria Mansion, Inc. this non-profit organization owns and operates the building as a historic house museum. Victoria Mansion was designated a National Historic Landmark in 1970.