Gareth Edds opened the scoring for Rovers before John Welsh doubled their lead. And although Glenn Murray pulled a goal back from the spot, before being shown a second yellow card, it was the home side who were left celebrating at the final whistle.
Caretaker manager Parry said: "We clung on for an overdue win and it just feels so fantastic and we are going to enjoy it for now.
"It was a bad time for us to concede a penalty and it led to a nervy finish, but I would have been totally devastated had they got a point after we had been 2-0 up.
"The fans were absolutely fantastic. You need the crowd when you have your backs to the wall and we certainly had our backs to the wall in the closing stages."
He added: "I don't know what is happening over a new manager. There have been indications it will not be announced next week. Now we have won, the club officials can take their time to make the right appointment."