In 1924 McCloy joined the prestigious New York corporate law firm of Cravath, de Gersdorff, Swaine, and Moore, rising to partner in 1929. He spent several years representing Bethlehem Steel in the Black Tom case, in which that firm sought and in 1939 eventually obtained $20 million in damages from the German government for sabotaging an American munitions plant during World War I.
In 1940 McCloy joined the War Department as a consultant to Secretary Henry Lewis Stimson, a lifelong hero and role model. Appointed assistant secretary the following year, McCloy was involved in virtually every major political and military wartime decision until he left that position in November 1945. He staunchly advocated the wartime internment of Japanese Americans and throughout his life would claim that national security reasons had amply justified the consequent infractions of civil liberties. He supported War Department proposals envisaging Germany's economic reintegration into Europe, helping to thwart Treasury Department plans to partition the country and destroy its industry. In 1945 he unsuccessfully opposed employing atomic weapons against Japanese cities, something he always contended had been unnecessary.
McCloy left the War Department in 1945, briefly heading the World Bank from 1946 to 1948. From 1949 to 1952 he was American high commissioner in the Federal Republic of Germany (FRG, West Germany) and was responsible for implementing that country's return to independent statehood. Controversially, he decided to pardon various German industrialists including Alfred Krupp, whom the Nuremberg tribunals had convicted of war crimes, a decision that many ascribed to Cold War expediency. McCloy staunchly backed his longtime French friend Jean Monnet's efforts to bring about West European political and economic integration, considering this essential to heal long-standing Franco-German antagonisms and strengthen Europe economically.
McCloy was president of the Chase Manhattan Bank from 1953 to 1960, after which he returned to law. He remained one of the Wise Men, recognized foreign policy experts whom successive presidents consulted on a wide range of international issues. The journalist Richard H. Rovere even termed him the "chairman" of the American Establishment. From 1961 to 1974 McCloy was a presidential advisor on arms control. Although he did not attend the meeting of senior advisors that counseled President Lyndon B. Johnson in March 1968 to withdraw U.S. forces from Vietnam, by that juncture McCloy was already known to be disillusioned with the American commitment to that country. He died in Stamford, Connecticut, on 11 March 1989.
Isaacson, Walter, and Evan Thomas. The Wise Men: Six Friends and the World They Made; Acheson, Bohlen, Harriman, Kennan, Lovett, McCloy. New York: Simon and Schuster, 1986.; Kraske, Jochen, with William H. Becker, William Diamond, and Louis Galambos. Bankers with a Mission: The Presidents of the World Bank, 1946–91. New York: Oxford University Press for the World Bank, 1996.; Kraske, Jochen, with William H. Becker, William Diamond, and Thomas A. Schwartz. America's Germany: John J. McCloy and the Federal Republic of Germany. Cambridge: Harvard University Press, 1991.