Cheltenham controlled most of the first half, with David Hutton and Matt Thornhill both going close during a succession of corners inside the opening five minutes.
Hayles saw a shot held by Bevan and also missed from a good position with a headed effort, but he scored his sixth goal of the season to put the Robins in front at the break.
Torquay improved in the second half and Elliot Benyon shot straight at Brown from a glorious position in the 68th minute. Lloyd Macklin then saw an effort deflected wide and Ashley Eastham did well to block Nicky Wroe's powerful drive as Cheltenham looked like they had held on for three huge points.
But Carayol had only been on the pitch for five minutes when he made space for himself in the penalty area and beat Brown with a low shot.