The mansion that stood at the southwest corner of Porter Avenue and Niagara Street was built by Hiram Pratt in 1835. Its architect was John H. Selkirk. Pratt never lived in it because he was a major figure in the financial disaster created partially by and around Benjamin Rathbun.

Hiram Pratt, born in 1800 in Vermont, came to Buffalo with his family in 1804. His father created and lost through misfortune the family trading business and Hiram was left to make his own way which he did, first as a grocer and then founding a forwarding business. When he married Maria Fowle of Northampton, Massachusetts, he found someone as ambitious as he.

Pratt saw that Benjamin Rathbun was a skillful and successful entrepreneur and began working on projects with Rathbun whenever the opportunity arose. Because of this, Rathbun saw that Pratt was named to the board of directors for the new Bank of Buffalo in 1831; he soon became cashier. In 1835, he became president of the bank and lent money to his friends and business associates in speculative projects.

At the time he built the great sandstone mansion on top of Prospect Hill and donated the two five-acre squares that would become Prospect and Columbus Park, Pratt's career was seemingly approaching its apex. He was just 35 years old when elected mayor of Buffalo in 1835 by the Common Council. (He would be elected again in 1839.)

When Benjamin Rathbun was arrested for forgery in 1836 and his empire collapsed, Hiram Pratt also suffered financially, though he had numerous enemies in Buffalo who pointed out that he had benefitted most from Rathbun's collapse. This war over his reputation continued until 1840, when Pratt died at age 40 on his way to Saratoga for a rest cure. His widow continued to live with their three daughters at the corner of Swan and Center Street.

Bela D. Coe was a businessman who made his success in Canandaigua in the hotel business that served stage travelers crossing the state from Albany to Buffalo and parts in between. Sameul Welch described him as "an educated gentleman of quiet, unostentatious, undemonstrative manner, but of rare tact and ability for business, and the organization and controlling of men." He was a careful planner, conservative investor and never exceeded his financial reach. That made him the polar opposite of Benjamin Rathbun, a successful Buffalo hotel man with his widely known Eagle House who engaged in numerous ambitious business and construction projects in the 1820s and early 1830s. Eventually, Coe invested with Rathbun in the stage business and moved to Buffalo in 1825, opening his own hotel, the Mansion House, near the Eagle House. By 1835, Coe and most others realized that the canal and the newly encroaching railroad would spell the end of profits from stage lines and he sold his interest in the Old Line Mail company to Rathbun.

Coe avoided the real estate speculation that brought so many to financial ruin, and so was in a position to purchase the large Prospect Hill estate from the Hiram Pratt family in the early 1840s.

Coe and his wife, Elizabeth, who had no children, had first lived in a stately brick colonial home with extensive grounds on Main Street, bounded by Eagle and extending to Washigton Street. They were active in forming the young city's high society, giving garden parties and entertaining at home. Buffalo's promising young authors and poets could be found in their salon. Known for her roses, Mrs. Coe expanded her gardens once they moved to the Porter Avenue mansion. The home would be completely surrounded by lawns and gardens. Niagara Frontier author Olga Lindberg quotes an unnamed historic source as describing the garden as "a pleasaunce with knottes and mazes," referring to herb gardens planted in the shape of bow knots and hedges planted in intricate designs. "The grounds are tastefully laid out in geometric order, with labyrinthian walks. The borders are stocked with a large and valuable collection of shrubbery, including a rare collection of roses, chiefly from France."

Porter Thompson, as he was called, was the only son of Sheldon Thompson, early Buffalo lake merchant and shipbuilder who, like Bela Coe, avoided the financial losses visited on so many Buffalo entrepreneurs in the financial crash of 1836 and the economic depression that followed. One of Sheldon Thompson's investments was in a white lead factory which produced the lead powder used in paint. Porter Thompson clerked for his father in family waterfront offices, learning business practices and then in the Thompson & Co. lead works on Georgia Street. Briefly, he left to contribute to the founding of a blast furnace which eventually became the Buffalo Union Iron Works. He returned to the lead business in 1866 and purchased an interest in the S.G. Cornell & Son, eventually becoming president of the Cornell Lead Works on Delaware Avenue at the corner of Virginia. He was said to have "a modest and retiring disposition."

He and his wife, Matilda, had eleven children. One, Laetitia, had her debut party in the mansion: "She was arrayed in a simple dress of white relieved with bands of pink. Though her dress was simple, her party was not...The old mansion perhaps had never seen more gaiety than on the January evening in 1888 when MIss Laetitia made her bow to the fashionable world. The drawing room pillars were wound with ropes of pine and holly; from the capitals strands of pine and holly were festooned to the chandelier in the center of the ceiling, where they were woven with copious clusters of mistletoe. Growing plants blossomed along the walls and somewhere from behind the greenery floated the music of an orchestra. In the supper room a pyramid of fruits and flowers rose in the center of the principal table and from the pyramid sprang a tiny fountain to shed its fine and silvery spray on the close-surrounding fragrant bank of massed blossoms. Laetitia Thompson's coming-out party seems to have been an all-night affair; for when it broke up it did so only to reassemble for breakfast at the Cary castle at Delaware and Huron." Commercial Advertiser, January 7, 1888 (History of North Western New York)