Anies Baswedan won with 58 percent of the votes versus 42 percent for Basuki Tjahaja Purnama, known by his Chinese nickname as "Ahok", based on 100 percent of the votes in an unofficial "quick count" by Indikator Politik. Other pollsters showed similar results.
The national elections commission will announce official results in early May.
The turbulent campaign featured mass rallies led by a hardline Islamist movement, which has strengthened in recent years in a country long dominated by a moderate form of Islam. More than 80 percent of Indonesia's population professes Islam.
"Going forward, the politics of religion is going to be a potent force," said Keith Loveard, an analyst at Jakarta-based Concord Consulting and an author of books about Indonesian politics.
Baswedan's huge margin of victory was surprising since opinion polls in the run-up to the election had pointed to a dead-heat. Purnama won the first round of voting for governor in February in a three-way race.
Indonesian social media users likened the election outcome to the shock results of the US presidential vote and the Brexit vote of last year.
One Twitter user, @fuadhn, said Indonesians "can feel what US and British citizens feel now. Welcome populism..."
The election came on the eve of a visit by U.S. Vice President Mike Pence, as the Trump administration seeks to engage the world's fourth-largest nation and largest Muslim-majority country as an emerging regional power.
Pence is scheduled on Thursday to visit the biggest mosque in Southeast Asia, Jakarta's Istiqlal Mosque.
The Jakarta election will be seen as a barometer for the 2019 presidential election, given the city's outsized importance as both the nation's capital and commercial centre.
Purnama is backed by President Joko Widodo's ruling party. Baswedan is supported by a retired general, Prabowo Subianto, who narrowly lost to Widodo in a 2014 presidential vote and is expected to challenge him again.
Police said 15 people were detained following reports of disturbances at several polling stations in the city of 10 million people, after what the Jakarta Post this week dubbed "the dirtiest, most polarising and most divisive" election campaign the nation had ever seen.
Security appeared light at several polling stations, though police said 66,000 personnel were deployed across the city.
Religious tensions have been an undercurrent in the campaign, with Purnama on trial for blasphemy over comments he made last year that many took to be insulting to Islam.
Hundreds of thousands of Muslims took to the streets late last year to call for his sacking and to urge voters not to elect a non-Muslim leader. One person died and more than 100 were injured after one protest turned violent.
Some voters may have been reluctant to vote for Purnama because of worries about “five more years of protests on the streets by Muslim hardliners,” Loveard said in a telephone interview.
Ismail Yusanto, spokesman for one of the groups, Hizbut Tahrir Indonesia, said the election showed that Jakarta voters didn't want a non-Muslim leader.
"It is forbidden under Islamic law, to have an infidel leader," he told Reuters.
Members of hardline groups, including the Islamic Defenders Front (FPI), celebrated the election result at Istiqlal mosque in central Jakarta on Wednesday evening, praying and cheering for the governor-elect.
Baswedan and Prabowo were also expected at grand mosque to join the prayers, according to media reports.
Reconciliatory tone
Baswedan, a respected scholar who many viewed as moderate, drew widespread criticism during the campaign when he aggressively courted the conservative Islamic vote, appearing publicly with hardline Islamic leaders during anti-Purnama rallies.
Baswedan, surrounded by his political patrons including Prabowo, struck a conciliatory tone at a news conference after unofficial results came in, pledging to "safeguard diversity and unity".
His platform has focused on improving public education, providing no-deposit home loans for low income groups and opposing a giant seawall in Jakarta Bay that Purnama has advocated. Baswedan has denied he plans to implement Islamic sharia law in Jakarta if elected.
Baswedan will officially take over as governor in October.
Purnama congratulated his rival in a news conference.
“We still have six months (in office) until the new governor is inaugurated and we will finish up our homework," Purnama said. "We hope that in the future everyone can forget the campaign period.”
Purnama's blasphemy trial resumes on Thursday and he faces up to five years in jail if convicted.
Investors in Indonesian markets are likely to return to fundamentals such as corporate earnings, now that the political uncertainties surrounding the divisive Jakarta election have diminished, analysts said.
"As long as there are no security issues, the election outcome should not significantly stall the reform programme of the national government, in our view," Citigroup said in a note.
It maintained its year-end target of 6,150 points for the Jakarta stock exchange .JKSE, which represents an 8 percent upside.
