Aaron Mokoena was the guilty party after needlessly hauling down Richard Offiong, but Pompey regained their composure quickly and strikes from Aruna Dindane, Danny Webber and Anthony Vanden Borre - all their first for the club - sealed a morale-boosting 3-1 victory.
The win provided some welcome respite from the record-breaking run of six defeats with which they have opened their Premier League campaign.
Hart handed a debut to Ivory Coast striker Dindane having been forced to make several changes due to players being cup-tied, with Webber and Hassan Yebda also brought into the line-up for their first starts for the club.
Angelos Basinas, Hayden Mullins and John Utaka were also given starting spots, while Asmir Begovic deputised for David James in goal as he had in the previous round. Frederic Piquionne and Michael Brown were dropped to the bench from the side that lost 2-0 at Aston Villa.
Carlisle boss Greg Abbott rewarded on-loan Chelsea youngster Tom Taiwo for an impressive cameo at the weekend with a place in the Carlisle first XI, while Joe Anyinsah came in for Scott Dobie in the only other change for the hosts.
The Premier League side started the match in much the same way as they have their entire season, suffering the humiliation of going a goal down against their League One opponents in only the second minute.
Pompey only had themselves to blame though as Mokoena blatantly pulled Offiong back as he received the ball in the area, leaving referee Neil Swarbrick no choice but to point to the spot.
Veteran defender Harte coolly slotted the resulting penalty home, sending Begovic the wrong way and piling the pressure on the beleaguered Hart.
The visitors attempted to hit back immediately and after Younes Kaboul had tried a speculative effort from range, Utaka was unlucky to lift the ball over the bar from Yebda's lay-off.
Yebda then had a shot blocked himself in the box and Nadir Belhadj sent a free-kick straight into the arms of Lenny Pidgeley as Portsmouth tried to save face with a swift equaliser.
The breakthrough came on 26 minutes as Dindane marked his debut with a strike to bring Pompey level.
Utaka brought down a long, hoisted ball over the Carlisle defence and hooked it around Pidgeley, leaving Dindane to simply tuck it away from just a few yards out.
Utaka nearly made it two moments later after rifling the ball through a mass of bodies, but the visitors were finding theistride and soon after, had turned the game on its head.
This time it was Webber chalking up his first Pompey goal on 33 minutes, the former Sheffield United forward flicking Yebda's cross over Pidgeley and into the top corner to send the tiny band of travelling fans into raptures.
Dindane was then unlucky not to score his second, lofting a chip narrowly over the bar as Portsmouth broke incisively, and Webber was also close to finding another goal, drilling a shot straight at Pidgeley when fed through by Utaka.
Carlisle started the second half positively but were unable to make the most of their chances, Peter Murphy dragging his shot across the face of goal after being picked out by a wonderful Harte delivery and Anyinsah directing a header just wide.
They were made to pay for their profligacy just after the hour mark when Dindane capitalised on an error in the hosts' defence and cut the ball back for Vanden Borre, who lashed it past Pidgeley, becoming the third player of the evening to open their Portsmouth goalscoring account.
Piquionne came on in the dying minutes and missed two glorious opportunities to increase Portsmouth's advantage but with Carlisle unable to offer anything else, the visitors' place in the fourth round was secure.