This is because the stadium with views of the sparkling blue Indian Ocean could be demolished because its pavilion stand violates heritage laws protecting a 17th century Dutch fort, according to the government.
Sri Lanka's Cultural Affairs Minister Wijeyadasa Rajapakshe told parliament that the fort risks losing UNESCO World Heritage status because of unauthorised construction, including Galle's 500-seat pavilion.
"We have to decide if we want to remain in the World Heritage list or keep the pavilion," Rajapakshe said.
The Galle stadium in the south of the island is hallowed among cricket fans because of its stunning setting and it is here that the Sri Lankan team is often at its most formidable.
The hosts have won a majority of matches played there since 1998, with the strong winds favouring their wily spinners.
The world's highest Test wicket-taker, Sri Lanka's spin wizard Muttiah Muralitharan, took his 800th scalp against India at Galle in 2010. Six years earlier, Australia's legendary Shane Warne took his 500th Test wicket there.
The first Test at the venue was in June 1998 and Sri Lanka won that match against New Zealand by an innings and 16 runs.
Last week, Sri Lanka won their first Test against South Africa in Galle by 278 runs with two days to spare, with the Proteas all out for a record low 73 in the second innings.
The Galle pavilion was built in 2008, four years after the grounds were devastated by the Indian Ocean tsunami which killed at least 31,000 people in the country.
The British captured Galle in 1796, but did not make any significant alterations to the structures in the walled city. It is now a key tourist attraction. Sri Lanka's Sports Minister Faiszer Musthapha said there will be no immediate demolition.
"We want to maintain the World Heritage status for the fort. We will work out an alternative for the cricket stadium," he added.
Southern Development Minister Sagala Ratnayaka noted that UNESCO did not object to the cricket ground, but wanted unauthorised structures around it removed, including the two-storey pavilion.
The building, named after former President Mahinda Rajapakse, obstructs the view of the fort from the main Galle road.
Sri Lanka's 1996 World Cup winning skipper Arjuna Ranatunga had suggested that the authorities could dismantle the offending pavilion and erect temporary stands when Tests were being played.
"We need to keep both the heritage status for the fort and at the same time play international matches at Galle,' Ranatunga reasoned.
He said a proposal 10 years ago to play day and night matches at Galle, floated while he was the interim head of Sri Lanka Cricket was shelved because setting up lights would have violated building codes in the area.
His cabinet colleague Rajapakse noted that the government plans to build another stadium in Galle, two hours' drive south of Colombo.
"We could have another cricket ground in Galle soon," he added.