The compound was largely empty apart from tourists and Jewish visitors, with Muslims again praying and protesting outside the site instead of entering through the metal detectors.
The Haram al-Sharif compound, known to Jews as the Temple Mount, includes the Dome of the Rock and the Al-Aqsa mosque.
Several hundred people could be seen praying outside two different entrances to the site on Monday.
There were protests after the prayer, with crowds shouting: "Aqsa mosque, we sacrifice our souls and our blood."
Police later sought to move them back.
"We will not break the solidarity of the people," said Jamal Abdallah, a Palestinian who now lives in the US state of Arizona and was planning to visit Al-Aqsa but changed his mind when he was made aware of the situation.
In the evening, several dozen Palestinians blocked a road near the Old City. Israeli officers dispersed the protesters, who hurled stones and other objects, police read.
Palestinian medical sources said 11 people were treated for rubber bullet injuries and dozens of others for tear gas inhalation.
Israel installed the metal detectors after Friday's attack near the holy site that saw three Arab Israelis open fire on Israeli police.
They then fled to the compound, where they were shot dead by security forces.
It was among the most serious incidents in Jerusalem in recent years and heightened Israeli-Palestinian tensions.
Israel took the highly unusual decision of closing the compound for the weekly Friday prayers, triggering anger from Muslims and Jordan, the holy site's custodian.
The site remained closed on Saturday, and parts of Jerusalem's Old City were also under lockdown.
Israeli authorities said the closure was necessary to carry out security checks, adding that the assailants had come from within the holy site to commit the attack.
They began reopening it on Sunday, but with metal detectors in place, while security cameras were also being installed in the area.
Al-Aqsa officials have refused to enter and have called on worshippers to do the same.
Palestinians view the new measures as Israel asserting further control over the site.
Crowds chanted "Allahu Akbar" (God is Greatest) as they gathered near the Lions Gate entrance to Jerusalem's Old City on Sunday.
On Sunday night, skirmishes broke out between Israeli police and worshippers outside the entrance, with the Red Crescent reporting 17 people wounded.
With tensions high, two mosques in the northern Israeli Arab town of Maghar were targeted on Sunday night, one with a stun grenade and another by gunshots. No serious damage was reported.
One of the two policemen killed in Friday's attack lived in Maghar. Both of the officers were from the Druze minority, Arabs who belong to an offshoot of Shiite Islam.
Netanyahu order
Prime Minister Benjamin Netanyahu took the decision to install the metal detectors and cameras following a meeting with security officials on Saturday.
He also spoke by phone with Jordan's King Abdullah II on Saturday night before leaving on a trip to France and Hungary.
Abdullah condemned the attack, but also called on Netanyahu to reopen the Al-Aqsa compound and stressed the need to "avoid any escalation at the site".
Palestinian president Mahmud Abbas conveyed a similar message to Netanyahu when the two spoke by phone on Friday in the wake of the attack.
Hamas, the Islamist movement that runs the Gaza Strip and Abbas's rival, has welcomed the attack, calling it "a natural response to the Zionist terrorism and the desecration of the Al-Aqsa mosque".
On Monday in a joint statement, Hamas and Islamic Jihad called for demonstrations over Al-Aqsa.
"We call for an end to all the Zionist measures and for the extremist government to take its hands off the blessed Al-Aqsa mosque," it said.
Proposals to change security measures at the compound have sparked controversy in the past.
A plan developed in 2015 between Israel and Jordan to install cameras at the site itself fell apart amid disagreement over how they would be operated.
The Haram al-Sharif/Temple Mount is central to the Israeli-Palestinian conflict.
It stands in east Jerusalem, seized by Israel in the 1967 Middle East and later annexed in a move never recognised by the international community.
It is considered the third holiest site in Islam and the most sacred for Jews.
Jews are allowed to visit but not pray there to avoid provoking tensions.
