With no more than a fourth grade education, Dorsey, a black man, came to Miami-Dade County in 1896 to work on the railroad. He settled in Miami’s "Colored Town" now known as Overtown. A carpenter by trade, Dorsey helped build the railroad that brought transportation to Miami and soon responded to the need for housing for black workers. He became a Notary Public and according to the 1914 City Directory, was "the only colored dealer." He purchased lots and built rental workforce houses.

The value of the land he purchased increased quickly as Miami expanded, and Dorsey's fortune grew as well. Eventually Dorsey amassed the largest real estate empire owned to date by a black man in Miami-Dade County. His holdings included Fisher Island as well as other properties throughout Dade and Broward counties. He also owned properties in Cuba and the Bahamas. He became Miami’s first black millionaire.

D.A. Dorsey provide land for a park and donated land for a library and school with the stipulation that ech be used for the "colored race." To honor his memory the park, school, and library are named for this benefactor. Well respected as a businessman and civic leader, Dorsey was Chairman of the Colored Trustee Board for the county school system.

The Dorsey House was acquired in 1990 by The Black Archives, History and Research Foundation of South Florida, Inc. It is listed on the U.S. Secretary of Interior’s National Register of Historic Places. Following federal guidelines the reconstruction of the original house was completed in 1995.