Up stepped McFadden to drill the ball past the 39-year-old, who had earlier denied the home side with fine saves from Larsson and veteran striker Kevin Phillips.
James' efforts looked to have earned his side a valuable point on the day Pompey confirmed a new consortium led by chief executive officer Peter Storrie was in negotiations to buy the club.
Blues boss Alex McLeish is desperate for his side to make St Andrews into a fortress after eight draws in his 13 home games in charge two seasons ago were a major factor in them being relegated.
But for all their endeavour and purpose, with Barry Ferguson impressive in midfield, they were unable to unlock the visitors rearguard, in which Sylvain Distin was a key performer, until the late finale.
McLeish made one change from the side beaten by a single goal at Manchester United with striker Garry O'Connor brought into the starting line-up at the expense of midfielder Lee Carsley.
Portsmouth boss Paul Hart handed a debut to on-loan Genoa midfielder Anthony Vanden Borre after international clearance was finally obtained yesterday. He came into the side at the expense of John Utaka.
Before the kick-off a minute's applause was undertaken in honour of former Blues chairman Jack Wiseman who died last week at the age of 92.
Blues striker Cameron Jerome had the first shot on target after some sloppy defensive work by the visitors but James held onto his low drive away to his left.
Jerome then teed up Fahey on the edge of the box but his low shot, although taking a slight deflection, did not overly trouble James.
But Larsson came close to putting City in front after 18 minutes.
The Sweden international got on the end of a deep first-time centre from left-back Gregory Vignal but he could not keep down his header.
It was 35 minutes before Portsmouth managed a worthwhile effort on target and then on-loan Joe Hart was well positioned to hold onto a long-range attempt from Niko Kranjcar.
Portsmouth were forced into a substitution after 42 minutes when Papa Bouba Diop limped out of the action to be replaced by Angelos Basinas.
Hart parried a low effort around the post from Frederic Piquionne although the shot was probably heading just wide before his intervention.
Portsmouth boss Hart made a half-time substitution with Utaka replacing Vanden Borre.
McFadden made a good break down the left before releasing Jerome but he lacked the necessary composure and curled his shot well wide.
Vignal looked to have picked up a knock and he made way for Blues defender Stuart Parnaby after 49 minutes.
Then Jerome was bundled off the ball by Bassinas, who was yellow carded, and after pitch-side treatment he was replaced by Phillips.
City tried to build up some momentum and James turned over a curling free-kick from Larsson after Phillips had been brought down by Mullins.
At the other end, City stopper Hart had little trouble in holding onto an effort from a similar range by Kranjcar.
James made a fine save to turn a curling shot from Phillips past the post after O'Connor had headed the ball into his path.
But St Andrews erupted after McFadden's late decider.