The men in footage they uploaded to the ‘Patriot Blue’ Facebook page.
Speaking with the Nine Network, Prime Minister Malcolm Turnbull said there should be “zero tolerance” for racial abuse.
“There is no place for racial vilification in Australia,” Mr Turnbull said.
Patriot Blue describe themselves as a "group of concerned patriots" whose "mission" is to stop Melbourne councils changing the date of Australia Day from January 26.
Labor leader Bill Shorten said he phoned Senator Dastyari after the incident, who he said was in “good spirits”.
“I was disgusted by the idiots calling Sam shocking names not worth repeating,” Mr Shorten told Channel Seven.
“What is this country coming to, when you can’t go out and have a meal without being abused by idiots, and by racist idiots?”
Senator Dastyari said he would speak with his family before deciding whether to pursue any potential legal response to the incident, which may have breached racial vilification laws.
He told the Nine’s Today show he was becoming accustomed to being “followed by white nationalists everywhere I go”.
“I worry about all the people out there that have to put up with all this kind of abuse who don't have the structures that someone like I'm lucky enough to have," Senator Dastyari said.
But the senator was also able to joke about the incident.
“They went to a bar and didn't even stay for a drink,” he said. “What kind of patriots are they?”
Footage of the incident was posted on the Facebook page of the group, Patriot Blue, on Wednesday night. Senator Dastyari was at the pub ahead of an event promoting his book in Footscray.
He was accompanied by Labor MP Tim Watts, who is seen trying to intervene in the incident.
Senator Dastyari tries to order a drink at the bar as one of the men pushes the camera closer to his face.
“Have another drink mate, is it halal certified?” the man continues.
“Racist rednecks, you follow me everywhere I go,” Senator Dastyari replies.
“What race is Islam,” one of the men respond.
“Are you a Muslim, Sam? Are you a Muslim or what?” another says.
The men in the video say they are “working class” Australians who value free speech. They also heckle Senator Dastyari over links to Chinese donors which cost the Labor senator his frontbench job last year.
“Do you know what’s funny, we’re the real Australians,” one says.
“Look at this terrorist, why don’t you go back to Iran?’ the heckling continues.
Senator Dastyari tries to handle the situation with humour, laughing the comments off and walking away.
“I’m not going to bother with you guys, you’re embarrassing yourselves,” he responds.
“That was really impressive. You must be really proud of yourselves,” he adds jokingly in another retort later on.
Neil Erikson, the man wearing a high-vis Toll shirt in the footage, said the Labor MP gave as good as he gets.
"Sam's a strong bloke he's got a thick skin, he's been in politics for a long time, " Mr Erikson told ABC radio on Thursday.
"He called us rednecks, which is a racist term in fact, so look, he gives as good as he gets.
"I think he's playing the victim a little bit."
A Toll spokesperson told AAP the actions in the video "in no way reflect our beliefs and values as a company".
"We are currently investigating the matter and whether any of the individuals involved are currently employed by Toll.
"At Toll, we celebrate our diverse workforce and we expect all of our people to behave in a manner that is consistent with our values and stated policies at all times."
- With AAP
