Rovers almost took the lead after two minutes when Alex Meechan headed just wide, but Oxford soon took control.
The first goal arrived after 26 minutes when goalkeeper Billy Turley's long punt was flicked on to Duffy, who cut inside his marker and curled the ball beyond Steve Williams in the home goal.
The lead lasted only 12 minutes before Allan Russell bundled home an equaliser.
Conceding sparked Oxford into action and they went in ahead at the break after Duffy bravely dived to head home Andrew Burgess' corner.
The visitors were made to work hard after the interval and their third goal did not arrive until the 83rd minute, when Yemi Odubade turned away from Darren Jones and smashed a shot past Williams.
Odubade then combined with Eddie Ancalet to create an 89th-minute chance for Duffy, who claimed the match ball with a clinical finish.
There was still time for Oxford to add a fifth when Christian Hargreaves fired home from the edge of the box with 95 minutes on the clock.