The veteran Cheltenham striker pounced in the fourth minute of stoppage time to plie the pressure on Bury, who have now lost their opening four home games.
Cheltenham had the game's first chance as Julian Alsop headed over from a Hayles cross. Bury then began to take charge with Dave Worrall having a shot deflected over by Drissa Diallo before Ryan Lowe lobbed onto the top of the crossbar after latching onto an Andy Bishop flick-on.
The Robins replied with Justin Richards curling a 20-yard effort just over the top, before Hayles tested Bury keeper Wayne Brown with a 20-yard drive. The hosts finished the half strongly, though, with Lowe firing wide before Stephen Dawson blazed over from just inside the box, while Bishop fired into the side-netting from a wide angle.
Visiting keeper Scott Brown had to palm away a fierce close-range strike by Lowe following another Bishop flick just after the break but Bury failed to maintain that momentum. Robins substitute David Hutton kept Wayne Brown alert with a close-range effort just after the hour mark, before Bury sub Mike Jones fired wide after cutting in from the left.
Andy Gallimagh was denied late on for the visitors but Hayles struck at the death, rounding Wayne Brown before superbly slotting home from a tight angle.