Oldham led through Alex Marrow's first-half effort but Jon-Paul McGovern drew Swindon level with a fine 30-yard strike with seven minutes remaining before a defensive slip allowed the Robins to go ahead in the 92nd minute when Billy Paynter pounced to score.
However, Swindon could not hang on for the win points as Kieran Lee levelled for the hosts just a minute later.
"It feels like a defeat because we controlled big parts of the game and didn't feel threatened," said Penney.
"They were very poor goals to concede. You can put on all the coaching sessions in the world but you cannot excuse that.
"It was poor defending and goalkeeping and we need to improve that because we gave away a good position. As a team we played quite well other than those two big negatives. We made chances and we should have killed it off."