James McFadden scored from the spot in injury-time to break Pompey hearts after a hard-working display from the visitors.
Referee Lee Probert pointed to the spot after Kevin Phillips' cross was headed away by Younes Kaboul and David James, who was watching the ball and not the player, ran into Sebastian Larsson at the far post, sending him crashing to the ground.
Earlier, the Pompey keeper had pulled off a magnificent save to deny Phillips, while Hayden Mullins shot wide when unmarked six yards out.
Hart told Sky Sports: "Well having looked at it, I think it is a little bit harsh.
"If you look at David, he is looking at the ball, his arms are out-stretched to catch it and they bump into each other - he didn't push him out of the way.
"So I think it is harsh and I think the referee had a perfect view and relied on his linesman who can't have seen it too well."
Hart believed his side had been worthy of a point after their hard-working display at St Andrews.
"I think we were wasteful in possession but certainly (we deserved something) with the effort we put in and their determination to get something from this game.
"I said to the boys 'my only complaint has been we were so wasteful' - if we had kept better possession I think we would have won the game."