Frazier - also known as Smokin' Joe - had been receiving hospice care in Philadelphia after being diagnosed with cancer several weeks ago.
The 67-year-old was the first man to beat Muhammad Ali in 1971, but lost his next two bouts with Ali.
He held the world title between 1970 and 1973.
On Sunday, Frazier's manager Leslie Wolf said the boxer's condition was very serious but that doctors and Frazier's team were "doing everything we can".
Frazier won an Olympic gold medal in 1964 after going to the Games as a replacement for Buster Mathis, who had beaten him in the trials but could not attend the Games due to an injury.
He won the heavyweight title in 1970 by defeating Jimmy Ellis in New York.
Three years later he lost the title to George Foreman.