Most of what can be learned about Madison’s racial attitudes is found in letters of his that remain largely unexplored. In one astonishing document, written in 1783, he explained to his father that he faced a quandary with respect to an enslaved man named Billey, whom Madison had brought with him to Philadelphia as his manservant while he was in Congress. Madison wrote that he could not bring Billey back to the family plantation, because after Billey had experienced the relative freedom of Pennsylvania, “his mind is too thoroughly tainted to be a fit companion for fellow slaves in Virginia.”
Madison could not “think of punishing him,” he wrote, “merely for coveting that liberty for which we have paid the price of so much blood, and have proclaimed so often to be the right, and worthy pursuit, of every human being.” In other words, Billey possessed the same “right” of liberty as every other human being — the right they had fought for in the Revolutionary War.
But it was a time of economic crisis. Madison felt he could not free Billey because he could not afford to. His compromise was to sell Billey into a term of indentured servitude, as allowed by Pennsylvania law, at the end of which he would become free. Madison therefore got some value for his slave, while satisfying his conscience. At the end of his term of service, Billey was indeed freed, taking the name William Gardener, marrying and staying in Philadelphia, where Madison continued to have contact with him for years.
Madison understood perfectly well that slavery was morally wrong — and why. But he also could not overcome the economic reality on which his entire livelihood and that of the family plantation was based.
Thanks to Madison’s influence, his characteristic treatment of race — principled ideals followed by compromise — made its way into the Constitution itself. The story begins in 1783, four years before the Constitutional Convention in Philadelphia. The Continental Congress had asked the states for annual contributions of money, determined proportionally based on population. Southerners wanted to count slaves as little as possible to reduce their share; Northerners wanted to count slaves fully so that the South would pay more. Madison proposed counting three-fifths of slaves as a compromise, believing it was “very near the true ratio” with respect to the taxable wealth produced by them. All but two states approved of the proposal, but because unanimity was required, it did not pass.
At the Constitutional Convention in 1787, the tables were turned. Southerners wanted to count slaves fully to increase their legislative representation, while Northerners argued that slaves, who couldn’t vote, should not count at all. The three-fifths compromise was adopted precisely because everyone knew that the number had been used when both sides saw things the other way. Madison was happy to see it adopted.
There was also the issue of the constitutional provision that preserved the slave trade for 20 years after ratification. Madison objected strongly to it. Like other slaveholders, he considered trade in slaves more odious than ownership. Yet when it became clear that South Carolina and Georgia, backed by Connecticut, insisted on protecting the slave trade as a condition for agreeing to the Constitution, Madison went along. Reaching consensus was paramount. Without the guarantee of slavery and the slave trade, there would have been no Constitution — and that was Madison’s highest priority.
Further demonstrating the tension between his beliefs and his actions, Madison called for abolition — but only in private. In 1790, just after drafting the Bill of Rights, Madison argued in a letter for creating a colony in Africa to resettle freed slaves. That would, he said, “afford the best hope yet presented of putting an end to the slavery in which not less than 600,000 unhappy Negroes are now involved.”
His reason for sending former slaves to Africa was “the prejudices of the whites,” which would make integrating freed blacks “impossible.” He thought that because prejudice derived “principally from the difference of color,” it must be “considered as permanent and insuperable.” Years later, Madison proposed that Congress sell the public the Western lands included in the Louisiana Purchase to raise $600 million to purchase and free all 1.5 million slaves.
But ultimately Madison could not afford even to free his own slaves in his will. According to family lore, he left separate instructions to Dolley, his wife, to emancipate his slaves upon her own death. If this message in fact existed, she was unable or unwilling to fulfill it.
Madison’s contradictions resonate with our own vexed racial situation. From the 14th Amendment to the Civil Rights Act of 1964 and affirmative action, America, at its best, has repeatedly aspired to establish racial equality in the face of our past of slavery and segregation. Yet the need to accommodate the realities of economics, politics and the prejudice of others — which is to say, the difficulty that even well-intentioned people can have in accepting the true cost of meaningful change — has continued to hold back equality in practice.
Madison had many remarkable qualities. He was a constitutional genius, an accomplished statesman and a calm, rational leader, even in wartime. When it came to race, however, he had feet of clay, as do most Americans still. Our founders can serve as important role models for us — provided we try to learn from their mistakes and flaws as well as their successes and virtues.