I went to church at Fishkill village and returned home through the rain and got very wet but heard a very good sermon preached. Mr. J D L Verplanck, Mr. W S Verplanck and Miss A L Verplanck attempted twice to cross the river to Newburgh to take passage in the steam boat for NY. But was disappointed and was obliged to return home.
So begins the diary of James F. Brown, recording the day’s events for January 1, 1829. For the next 37 years — until March 1866, when he suddenly stopped — Brown would continue adding to this account from his home in Fishkill Landing (today’s City of Beacon) on a regular basis.
What makes it especially remarkable is who wrote it. Brown was an award-winning gardener, widely respected Beaconite, friend to the Hudson Valley’s most prominent residents, and among New York State’s very first Black voters — and he achieved it all after securing his personal freedom. As Brown biographer and Bard College professor Myra Armstead writes in her book, Freedom’s Gardener, it was “a virtually unheard-of undertaking for an African American man,” especially one who had escaped enslavement.
Penned in a clear hand over 10 account books, Brown’s diary may not be a page-turner: “He’s so regular and dispassionate about his writing that it’s almost boring,” Armstead admits.
Yet thanks to its author’s encyclopedic interests — Brown records daily chores and weather, local and national events, travel on trains and steamboats, and civic amusements — the diary offers a detailed, historically valuable narrative of everyday life in 19th-century America and specifically, along the Hudson River.
“What can a man do who has his hands bound and feet fettered?”
How Brown came to be in a position to write the diary and enjoy a fulfilling career as head gardener and caretaker at Mount Gulian, the Hudson River estate of the wealthy Verplanck family, is definitely compelling.
Born enslaved in Maryland in 1793, Brown worked on a couple of farms there (along the way learning to read and write) before becoming a so-called “wage-slave” in Baltimore. While still subject to restrictive racial laws, he was able to make an independent living — perhaps as a servant, coachman, or skilled carpenter — while remitting a portion of his monthly income to a slaveholder.
By March 1828, Brown was working as a servant for the Verplancks at their Manhattan home. How he secured this position is unknown, but “it was a game-changer for him,” says Elaine Hayes, executive director of the Mount Gulian Historic Site.
As late as 1800, the Verplancks had been slaveholders, but by the time of Brown’s hiring they “had evolved in their thinking,” notes Hayes, and were ardent abolitionists — so ardent that when a dinner guest recognized Brown as a fugitive, patriarch Daniel Crommelin Verplanck paid the slaveholder $300 (nearly $9,000 in today’s dollars) to secure his freedom.
Brown earned enough money to secure the freedom of his enslaved wife, Julia, shortly after their marriage in 1826, and had been promised his own freedom as well. When this failed to occur, Brown decided to escape. Prior to heading north in August 1827, he wrote a letter to his slaveholder containing these powerful words: “What can a man do who has his hands bound and his feet fettered? It will certainly get them loose in any way that he may think the most advisable.” In the letter, Brown also pledged to pay for his freedom.
Brown would repay his employer in a series of installments. “I think this says something about the drive and ambition of a slave to become free,” notes Armstead. “It also says something about the mindset of enslaved Blacks, where they could explore opportunities for freedom.”
Enduring concern for lives of fellow African Americans
Sometime before his first diary entry, Brown had moved to Mount Gulian, located just north of the present Newburgh-Beacon Bridge, where he was first employed as a coachman and general laborer before demonstrating his skill with plants and eventually overseeing not only the grounds but the accounts related to them. “Clearly, he had a special relationship with [the Verplancks],” Armstead says. “They entrusted him with a lot of family responsibility.”
During much of his career, Brown worked on the gardens in tandem with Mary Anna Verplanck, Daniel’s daughter, who managed Mount Gulian. Together, they made the grounds a horticultural showplace, attracting visitors from near and far. Produce and flowers grown there won awards at major fairs, which Brown, as a Black man, could attend only because of the Verplanck family’s prominence, while Mary Anna was excluded for being a woman.
Brown’s diary clearly illustrates the joy he took in spending time outdoors, recording seeds acquired and plants grown with minute detail. His expertise earned him respect and friendship with gardeners on nearby estates as well as with Andrew Jackson Downing, dubbed the father of American landscape architecture, who lived in Newburgh. Brown frequently rowed across the river to meet with Downing.
The diary also records Brown’s interest and involvement in the lives of his fellow African Americans, through his attendance at civic events, the abolitionist-leaning periodicals he read regularly, his leadership in establishing the first Black burial ground in Beacon, and associations with figures involved in the Underground Railroad.
In fact, Armstead suspects that Brown intentionally downplayed his personal involvement in that movement. “I was very careful not to over-speculate or go beyond what I could prove, but I think he was probably far more active in the Underground Railroad,” she says.
Achieving the American Dream
In the diary entry for Nov. 8, 1837 — and with his characteristic straightforwardness — Brown recorded one of the most momentous events of his life: “The election at Fishkill took place this day at which place James F Brown voted for the first time.”
It not only marked his participation in a fundamental right of citizenship — one granted to fewer than 30 Black men in New York at the time — but indicated he had achieved another part of the American dream. A requirement for Black people to vote was ownership of property. Brown had fulfilled it by purchasing a home not far from Mount Gulian. Every November until the diary ends, Brown diligently noted his participation in the electoral process. “It shows he wanted to be a citizen 100%,” says Hayes.
The lowest tide that has been for many years in the Hudson River was this day — the flats was bare from the Long Dock down to Denning’s Point — So that persons could walk down to get eels and fish with one hand.
Brown’s diary ends abruptly with this entry on March 26, 1866. No one knows why he decided to put down his pen, leaving blank any record of the final two years of his life as a self-described “old colored man.” He passed away on January 7, 1868, hailed by the local Fishkill Standard as “a prominent man…whose death will be regretted by many.”
The Verplancks, who inherited Brown’s diary, eventually donated it to the New-York Historical Society. It’s available for reading online. Another family member had the foresight to uproot and save a variety of plants from the gardens at Mount Gulian before they fell into decay in the 20th century. When the historic site later began restoring the grounds, roses, peonies, and yucca “with the same DNA of those that James and Mary Anna planted” were graciously returned, says Hayes. They take pride of place in Mount Gulian’s Heritage Garden.
Both the diary and the garden help to educate people about the example Brown set, one that still resonates strongly today, about striving to be part of what Armstead calls the American “narrative” — “the perception of opportunity, the wisdom of risk taking, the profitability of new ideas, and the soundness of democracy.”
“What makes James F. Brown’s story compelling,” Armstead writes in Freedom’s Gardener, “is that although he was born a slave in Maryland and therefore outside the intended or expected reach of that narrative, this [B]lack man nonetheless embraced it. Its message somehow reached him while he was still in bondage and propelled him to escape slavery, to challenge the exclusions it contained, and to forge a new and successful life for himself in upstate New York.”
“What inspires me is his persistence. He was planning his whole life on being a free man. What he had to do to arrive at that end is just remarkable,” says Hayes. “To me, he’s a hero.”
