In 1869, President Ulysses S. Grant appointed George Boutwell as Secretary of the Treasury and Richardson as his Assistant Secretary, and the Treasury Department hired John Sanborn, a wealthy businessman and lobbyist.  Congressman Benjamin Butler, their fellow Massachusetts colleague, encouraged the selection of all three men for their respective positions.  In early 1873, Boutwell resigned after the Massachusetts legislature elected him to the U.S. Senate, and Grant promoted Richardson to the post of Treasury Secretary.  Sanborn had already been named as one of the Treasury Department's special agents authorized by Congressional statute (1872) to collect delinquent taxes.

The tax agents' contracts were sent to Secretary Richardson for approval, which he signed without reading.  The agents usually earned a 10% commission on the tax monies they recovered, but Sanborn's contracts stipulated that he receive a 50% commission.  Given his role in exposing a Whiskey Ring early in the Grant administration, Sanborn's first assignment was to collect back-taxes from 39 distillers.  In his next contracts, the number of tardy taxpayers expanded exponentially to 760 in October 1872, 2000 in the spring of 1873, and 592 in July 1873.  

Furthermore, Sanborn neglected the legal mandate to identify the specific claim and the laws broken in each instance.  In one case, he simply wrote down company names listed in a railroad directory, and in other cases, he collected his commission when he had done little or no work.  Sanborn netted at least $200,000 in less than two years.  His scam was facilitated by other protégés of Congressman Butler, including the Treasury Department's Solicitor, New England's Collector of Internal Revenue, a Treasury Department Special Agent, and others.

In the spring of 1874, the Sanborn Contracts story broke in the press, and the House Ways and Means Committee instigated an investigation into the matter.  The actual process for collecting the delinquent taxes flabbergasted committee members, but they had difficulty finding incriminating evidence against those allegedly involved.  Secretary Richardson denied all knowledge of the scheme, and insisted that he had considered the contacts and related documents to be routine office materials that he could safely ignore.  He admitted that he should have paid closer attention, but the Secretary and other Treasury officials argued that Sanborn had done nothing illegal or unethical.  In Nast's cartoon, Richardson sits upon books labeled "Law," "Contracts," and "Duty," while a paper reading "Internal Revenue Law" is in the wastebasket.