Born in Leesburg, Virginia on November 9, 1815, Grimes was fortunate to grow up a free man. He witnessed the horrors of slavery in the South, and he devoted his life to assisting fugitive slaves and advocating abolitionism.
After moving to Washington, D.C., Grimes began a career as a hackney driver, providing transportation for people in and around Washington, D.C. Owning his own coach enabled him to serve as a conductor of the underground railroad for years without suspicion. He transported fugitive slaves from Virginia to Washington, D.C. and then assisted in moving them North. In 1839, Grimes was caught attempting to rescue a family of slaves from Virginia, and he was sentenced to two years in jail in Richmond, Virginia.
In jail he found religion and after his release in 1840, Grimes was baptized in the Baptist faith and was licensed to preach by a panel chaired by the president of Columbian College, a Baptist institution in the District of Columbia (now George Washington University), In 1846 he moved to Massachusetts and associated himself with the American Baptist Missionary convention in New Bedford, Massachusetts and in Boston. In November 1848 he was ordained as pastor of the Twelfth Baptist Church. He was pastor of the Twelfth Baptist Church for 27 years. Rev. Grimes actively opposed the Fugitive Slave Act, and his church became known as "The Fugitives Church." He became an important figure in national church organizations and at the American Baptist Missionary Society Convention at Philadelphia. In 1858 he was among those who pushed the organization to oppose slavery. The convention voted to have no fellowship with slaveholding ministers. He was president of the American Baptist Missionary Convention and the Consolidated Baptist Conventions for several years.
Anthony Burns was an escaped slave from Virginia who came to Boston and became a member of Rev. Grimes's church in 1854. When Burns’ former slaveholder discovered where Burns was living, he ordered his arrest. Rev. Grimes led a fierce effort to free Burns from jail, but the trial commenced, and the judge, in accordance with the Fugitive Slave Act, ruled that Burns was still property of his slaveholder. Rev. Grimes was able to raise enough funds to purchase Burns’ freedom. The Burns case was the last time that a fugitive slave was prosecuted under the Fugitive Slave Act in Massachusetts.
Many members of Twelfth Baptist church wanted to fight for the Union, and Rev. Grimes lobbied for the establishment of an African-American regiment. When their efforts prevailed with the formation of the Massachusetts 54th regiment, Rev. Grimes recruited men to serve in the infantry.
Rev. Grimes took ill just after a meeting of the Home Mission Society and died of apoplexy March 14, 1874 at his home in East Somerville, Massachusetts. His burial place is not known.