Malcolm Turnbull says his decision to delay the return of federal MPs is merely good housekeeping designed to pass the same-sex marriage laws.
The prime minister was standing firm during the rounds of breakfast television on Wednesday, despite the opposition, several crossbench MPs and Tony Abbott criticising his move.
"All we are doing is moving the two sitting weeks that were scheduled for to begin on November 27 forward a week to begin on December 4 so the house will be there when the Senate delivers the same-sex marriage bill," Mr Turnbull told the Seven Network.
"This is just good housekeeping. It is all about honouring and respecting the commitment of the Australian people."
But his predecessor, Mr Abbott isn't convinced.
"You might not always want to go back to parliament but you always have to go back to parliament because that's your job," he told Sky News.
'The great historical forces do play a part, but let's not underestimate the role of individuals as well,' he said.
'Perhaps on both sides of parliament, it's been the era of toxic egos and I think this has been part of our problem too many people have put themselves first and not their country'.
Labor and four crossbench MPs have written to the prime minister asking that the House of Representatives convene next week as originally planned after the sitting week was scrapped.
The entreaty was sent by Bill Shorten and MPs Bob Katter, Adam Bandt, Andrew Wilkie and Rebekha Sharkie on Tuesday. Victorian independent Cathy McGowan declined.
The crossbench support means 73 MPs support the original parliamentary program - the same number the government has on the floor of the house with Barnaby Joyce and John Alexander off fighting by-elections.
"We have been elected to represent the people of Australia and we stand ready to do so," the MPs wrote.
Opposition Leader Bill Shorten said it was a "joke" the prime minister had delayed parliament because he was scared of a royal commission into banks.
Among the bills which needed debating were the government's response to the child sexual abuse royal commission, new regulations affecting bank executives and anti-money laundering and counter-terrorism financing laws.
But Mr Wilkie, who said his constituents were outraged parliament was not sitting as expected, will not show up if Malcolm Turnbull refuses to change his mind.
The MPs also noted the parliament should consider setting up a commission of inquiry into the banks and debate a bill to restore penalty rates.
Shadow cabinet will meet in Canberra on Monday and it's possible caucus will also convene.
