Prime Minister Scott Morrison has come under fire for calling for the scrapping of Queensland’s vaccine mandate when the state reaches 80 per cent jab coverage.

Defence Minister Peter Dutton stresses the proportion of people declining to get the jab is small and they need to be allowed to participate in society.

“You cannot segregate a part of the community even if you disagree with the decision they’ve made,” he told the Nine Network on Friday.

“At some stage … you’ve got to allow people to come back into society.

“You want them to be vaccinated, but there’s a small portion who will make a decision not to be.”

Deputy Labor leader Richard Marles accused the coalition of picking a fight with Labor states.

“This undermines the vaccine rollout that is going on in WA and Queensland,” he said.

Mr Marles pointed out NSW also had restrictions on what unvaccinated people could do.

“The prime minister wasn’t saying this in respect of NSW,” he said.

Victorian Premier Daniel Andrews lashed Mr Morrison for “double speaking to extremists”, while Queensland Deputy Premier Steven Miles said the prime minister was trying to claw together “a coalition of anti-vaxxers”.

Finance Minister Simon Birmingham said Labor premiers were co-ordinating to pick “phoney fights” with the government.

“We’ve got to be really careful in terms of the approach that’s taken to maintain maximum confidence in Australians to actually get vaccinated,” he told Sky News.

“And that’s not by exacerbating political fights, it’s not by exacerbating partisan differences.”

The proportion of Australians worried about being infected is at its highest point since the start of the pandemic.

An Australian National University survey of nearly 3500 adults found 40 per cent thought they would likely contract the virus in the next six months.

In April this year, just over 10 per cent of people were worried they would become infected.

While the majority thought the worst of the pandemic was behind them, 45 per cent believed the worst was yet to come.

People were also increasingly experiencing severe psychological distress, with the rate jumping from 10.6 to 12.5 per cent between April 2020 and October this year.

Nationally, about 84 per cent of the 16-plus population is double-dosed and 91 per cent are partially protected.

Victoria has lifted almost all restrictions for the fully vaccinated as the state verges on a 90 per cent double-dose rate for people aged 12 and older.

It reported 1273 new cases and eight more deaths on Friday. NSW recorded 216 new infections and three additional deaths.

There were 25 more cases reported in the ACT on Thursday.

Northern Territory officials confirmed a woman who illegally entered the Top End last month sparked an outbreak infecting 19 Indigenous people so far.