The former Morecambe boss had been installed as the new Rovers chief after the departure of Gary Owers, with Harvey's experience of the Conference thought to be just the thing to turn things around at the New Lawn.
Rovers had led their managerless visitors since the 17th minute thanks to Allan Russell's second goal of the season, his first at the New Lawn.
It was the U's who had the first clear sight of goal on ten minutes, as Rovers fell asleep from Trevor Robinson's long throw and Marcus Richardson somehow headed wide from six yards out.
The opening goal in the 17th minute also stemmed from a long throw, Rovers' Kevin Nicholson hurling the ball into the area from the left to Russell. The Scottish striker's first touch was excellent and afforded him the space to fire a low shot across keeper Paul Crichton and into the bottom corner.
Russell had two difficult opportunities to double his tally in quick succession. First he reacted superbly to Les Afful's slide rule pass and took the ball around Crichton, but saw his angled shot headed clear before it reached the goal line by Michael Morrison.
Then, on 25 minutes, he ran on to Paul Stonehouse's lofted ball but could only poke the ball past the advancing Crichton and a foot wide of the post.
The U's had sights of goal from long range through Richardson and Jon Brady before the break, while the home side ought to have had a penalty when Afful was tripped inside the box by Robinson.
There were scares for home fans in the 52nd minute, when Rob Gier's mis-hit cross landed on the top of the net with Steve Williams frantically back-pedalling, and four minutes later when Robbie Simpson's scissor-kick just missed the target.
U's skipper Rob Wolleaston came close to scoring an own goal on 69 minutes when he headed Simon Clist's free-kick powerfully towards Crichton, who reacted well to tip over from close range.
Mark Peters was denied an equaliser eight minutes later when his header from Brady's corner was clutched on the line by Williams.
But any thoughts that Rovers would hang on to all three points were scuppered when Robinson's long ball was headed back across the area by Richardson to Carey-Bertram, who pounced in a crowded six-yard box.
The striker, who joined the U's on a free transfer from Hereford United in the summer, has endured a torrid start to his career with Cambridge and will be delighted to have got his name on the scoresheet for the first time this term.
However, any celebrations at the New Lawn were cut short straight after the final whistle as news filtered through that former Rovers defender Matt Gadsby had collapsed and died while playing for Hinckley United earlier on in the afternoon.
Gadsby, 27, had been with Rovers for two seasons before leaving the club in the summer to link up with Hinckley.