Brown was born on January 5, 1959, in Urbana, Ohio, and had an older sister, Beth (c.1957–1964). His mother, Joyce Helen (Eldridge), was a conductor, composer and concert pianist. His father, Clarence J. "Bud" Brown, Jr., was a newspaper publisher who helped manage the Brown Publishing Company, the family-owned newspaper business started by Clancy's grandfather, Congressman Clarence J. Brown. From 1965 to 1983, Bud Brown also served as a congressman, and later as Chairman of the Board of Brown Publishing. The family operated the business until 2010.
Brown graduated from St. Albans School in Washington, DC and Northwestern University.