The Cumbrians looked to be heading for their first home victory since January 17 when Michael Bridges fired home a 43rd-minute penalty. Oldham had Daniel Jones sent off for his part in conceding the spot-kick, although Latics boss Joe Royle felt the red card was harsh.
But Carlisle once again got the relegation jitters late on, and Abbott said: "It's a hard one to take when you need the points as much as we do now.
"We could and should have won. But at 1-0 you are always susceptible to set-pieces. The amount of chances we had we should have sealed it long before their free-kick."
He added: "I don't know what I've done to deserve finishing like that. At the minute we can't find the back of the net and we always needed a second goal to calm everyone down.
"Fair play to Oldham. They protected their goal well and got there in the end."