Dean Morgan's fourth goal of the season ended Cheltenham's five-match unbeaten run and put the O's six points clear of the drop zone.
A lacklustre affair boiled over in the closing moments with Cheltenham players furious at being denied what appeared a penalty when Jimmy Smith raised his arms to block an Andy Gallinagh cross.
Allen said: "It was a blatant penalty, but we've had that story on numerous occasions home and away this season. If your hands are above your head in the 18-yard box it is a penalty - it doesn't matter if it's three yards.
"We had the same in the FA Cup against Doncaster and it was worth £250,000. The penalty in this game wasn't worth that, but it was worth a lot in terms of keeping the run we've had going."
Allen added: "It was frustrating. We didn't play with the same levels as we have been. I suppose with young inexperienced players you're going to get some ups and downs, but I thought the boys who came on did well."