The bus, which at the time was a busy highway, was buried under a rubble in Gwangju, about 270 kilometers (168 kilometers) southwest of the capital, Seoul. .
Others were shot, the minister added.
The 16 demolition workers on the site were safe and did not support any damage as the building collapsed.
About 190 firefighters were called in for search and rescue, and police joined in the rescue operation, the minister said.
The cause of the crash is unclear, Reuters said, citing officials at the Gwangju Fire and Safety headquarters. The Ministry of Lands, Infrastructure and Transport will investigate the cause of the failure.
South Korea has been embroiled in a bloody security crisis – trade deficit killed more than 500 people in 1995, a landslide in 1994 killed 49 people. In recent years, the country has tried to improve its security.
This report was updated to reflect the extent of the injury.