Ganna lived up to his billing as the world time trial champion with a time of 35'32" on the 31.9-kilometre course between Montbrison and La Bâtie d'Urfé.
Perhaps peeved at his Stage 3 shortcomings, van Aert burst out of the gate, catching his four-minute man Victor Lafay (Cofidis) and almost passing his two-minute man David Gaudu (Groupama-FDJ).
The Jumbo-Visma rider was clocked 10 seconds faster than Ganna after 11.6 kilometres, but soon found himself 10 seconds slower with less than 10.9 kilometres remaining.
Ganna, riding for Ineos Grenadiers, was able to retain his lead on the meandering course, though only by 2.3 seconds as van Aert reached the line.
British time trial champion Ethan Hayter (Ineos Grenadiers) was next to cross the line, 17 seconds slower than Ganna, before Mattia Cattaneo (Quick-Step Alpha Vinyl) finished in fourth.
Though the final mountain stages will decide the general classification, Ganna's time trial victory did its part in identifying the real contenders.
"It was a long day in the hot seat but in the end it's important not to stand up from this seat," Ganna said after the race.
"Wout arrived really close but I was lucky that I won.
"When I saw it was raining this morning, I didn't think it was a super day but then we were lucky with the sun and the other course was dry for everyone.
"That meant it was the same conditions for everyone. That's important to compare us."
Van Aert's fast finish behind Ganna means he now holds a 53-second lead over second-placed Cattaneo, with fellow team leader Primoz Roglic in third at 56 seconds.
The Critérium du Dauphiné continues tonight on SBS and SBS On Demand with coverage of Stage 5 commencing at 11:10pm (AEST).
