A clinical trial on two groups of elderly people found those who followed an exercise program had a small improvement in overall thinking skills compared with those who did not exercise.
But the results suggested the benefit might last only as long as people continued with an exercise plan.
The findings, published in the journal Neurology, examined people with early signs of vascular dementia.
This is the second most common type of dementia, after Alzheimer's disease, and affects about 150,000 people in the UK.
It causes memory loss and difficulties with thinking, problem-solving and language, and occurs when there are problems with the supply of blood to the brain.
The study involved 70 people with an average age of 74.
Half of them took part in a one-hour exercise class three times a week for a six-month period.
The other half were given information on their condition and followed a healthy diet, but were not told to exercise.
Both groups were tested before the study, after it finished and six months later for overall thinking skills, executive function skills - such as planning and organising - and how well they could carry out their daily routine.
The results showed those who exercised had a small improvement on the test of overall thinking skills - improving by 1.7 points - compared with those who did not exercise.
Study author Teresa Liu-Ambrose, of the University of British Columbia in Vancouver, Canada, said: "This result, while modest, was similar to that seen in previous studies testing the use of drugs for people with vascular cognitive impairment.
"However, the difference was less than what is considered to be the minimal clinically important difference of three points."
Six months after the exercise group stopped taking exercise, their scores were no different than those who did not exercise.
There was no improvement on executive function skills or daily activities.
However, the exercise group had better blood pressure control and there were improvements on how far they could walk in six minutes.
Researchers said this was important because high blood pressure increased the risk of dementia.
