ACTU secretary Sally McManus has urged rogue union boss John Setka to resign, saying the Morrison government would use the controversy around him "for their own ends".
But Mr Setka, the Victorian secretary of the CFMMEU, insists only his construction union members will determine his future.
Mr Setka met with Ms McManus in Melbourne on Thursday, where she told him he should resign.
However, he stands firm against claims he criticised anti-domestic violence campaigner Rosie Batty.
He says he is the victim of false allegations being made for political gain.
Ms McManus, who has spoken with Ms Batty, told reporters it was in the interests of the union movement and working people that Mr Setka stand down from his role.
"Where an individual's actions cause damage to the whole movement, the interests of union members and the whole movement needs to come first," she said.
"The current federal government will use any opportunity to impose further anti-democratic, anti-worker laws on all union officials and all unions.
"It is clear they (the government) will try and exploit this situation for their own ends."
Mr Setka told reporters as he left the meeting it had been a "nice open and frank discussion".
Asked about his future he told reporters: "I'm going back to the office. My members are the ones that ask me ...They're the ones that pay my wages.
"That's who I will be taking advice off."
Earlier on Thursday, a monthly meeting of Victorian construction union shop stewards endorsed Mr Setka's leadership and called for the national CFMMEU office to follow suit, The Age reported.
The government is considering having another go at passing legislation to deregister law-breaking unions and ban officials for misconduct.
Employment Minister Michaelia Cash said it was a matter for the attorney-general to progress but, at present, the government would not be able to muster the numbers in the Senate for the bill to pass.
Mr Setka has been suspended from the Labor Party and faces likely expulsion after reportedly telling colleagues at a meeting that Ms Batty's advocacy had led to men having fewer rights.
He says a union member at the meeting had leaked the "false allegations" for political gain.
CFMMEU Victoria Secretary John Setka at the 2018 Labor Party National Conference Source: AAP
Labor leader Anthony Albanese says he spoke with other people at the meeting who insist that Mr Setka's comments were clear, whether they were misinterpreted or not.
"There is a whole history of comments Mr Setka has made, not just the one at that meeting but ones that I've been critical of in the past," Mr Albanese told ABC Radio National.
Former Labor prime minister Kevin Rudd earlier said the CFMMEU and Mr Setka should consider how their actions are hurting the Labor brand.
"I'd call upon the union to have a strong, hard look at itself and what damage in the eyes of the Australian public this delivers," he told Seven's Sunrise program.
Maritime Union of Australia's WA secretary Christy Cain was at the meeting from which reports of Mr Setka's comments emerged, and supports his account.
