Almost five minutes of injury time had elapsed when Lee floated in a deep right-wing cross and saw it hit the post before crossing the line.
Swindon were kicking themselves as they had only just gone in front, hitting back with a dramatic double strike from Jon Paul McGovern and Billy Paynter.
McGovern exploited a goalkeeping error to score from 30 yards after 83 minutes, and Paynter was gifted Swindon's second as he latched on to Reuben Hazell's woeful back-pass.
Oldham had controlled the first half, wasting good chances when Keigan Parker missed his kick from the six-yard line and Chris Taylor fired off target.
Debutant Nick Blackman shot just wide before on-loan teenager Alex Marrow struck his first league goal by chasing Taylor's flick and just reaching the ball ahead of keeper David Lucas. Swindon came back into it after the break, and McGovern hit the post from distance before the late drama.