Jonathan Tehoue pulled a goal back for Orient four minutes from time to set up a tense finish but Tranmere were able to hang on.
"It was a massive victory for us but it was a bit of a rollercoaster ride in the closing stages when they missed two great chances," said Parry.
"We could have had a couple more goals before half-time but we did not take all our chances. I would have liked to have kept a clean sheet but it was not to be and luckily they did not score till late on.
"Had their goal came any earlier it would have been even more nail-biting for us. Their substitutes caused us a lot of problems. It is particularly pleasing to get out of the bottom four and we need to stay out for the rest of the season."