However, the game turned in the visitors' favour after Perth defender Graham Gartland was sent off seconds into the second half for a professional foul on Fortune.
The former Nancy striker equalised with a wonderful angled drive in the 64th minute and notched a second from close range, either side of a Georgios Samaras drive, with substitute Paddy McCourt sealing the win with a late solo goal.
Gartland's dismissal has to be taken into consideration but the Parkhead side, without a single Scot in their starting line-up, certainly looked in the mood and must be confident of taking three points in the game in hand against Kilmarnock to put pressure on Rangers.
Celtic got off to the worst possible start when Loovens restrained Milne to prevent the Perth striker latching on to a clever one-two with Kenny Deuchar and referee William Collum pointed to the spot. Craig stepped up to send Celtic goalkeeper Artur Boruc the wrong way with a well-taken penalty.
The visitors responded quickly and moments later midfielder Zheng Zhi's effort sped through a packed penalty box only to be cleared off the line by Saints midfielder Jody Morris.
Loovens' woes continued when he was replaced midway through the first half after clashing heads with Deuchar, Paul Caddis coming on as substitute and Darren O'Dea moving into the middle alongside Josh Thompson.
As McDiarmid Park became enveloped in increasing darkness, with the floodlights apparently not fully working, Celtic took time to adjust to the switch.
However, on the half-hour mark, as Celtic broke with pace, midfielder Aiden McGeady set up Niall McGinn wide on the right but the Northern Ireland international smashed his shot high and wide.
The hosts looked comfortable in what remained of the first half but two minutes into the four minutes of added time, after a mistake by Gartland, McGinn smashed the ball against Perth goalkeeper Graeme Smith from close range when it looked like an equaliser was certain.
Around 14 seconds after the restart Gartland was shown a straight red card by referee Collum for a tackle on Fortune at the edge of the Saints box when he was judged to be the last defender.
McGeady drove the resultant free-kick against the Perth side's defensive wall and the home side's punishment remained at playing most of the second half with 10 men.
Moments later, Saints substitute defender Steven Anderson came on for Milne as the home side reshuffled.
However, Celtic gained complete control of the game and in the 53rd minute McGeady played a one-two with Fortune but slid his shot from 12 yards high over the crossbar.
Samaras then mis-kicked eight yards out from McGinn's cut-back before the Greece striker missed an even easier chance, hitting the outside of the post from 10 yards after being set up by Fortune.
Hoops boss Tony Mowbray threw on McCourt for right-back Andreas Hinkel and the pressure on St Johnstone intensified. Smith made a fine point-blank save from O'Dea's close-range volley but the home side survived yet another corner, before Paul McGowan replaced Zheng with Peter MacDonald replacing the tiring Deuchar soon after.
But the Celtic goal eventually arrived in the 64th minute when Fortune took a McGowan pass and from 16 yards fired a wonderful drive past the flailing Smith and into the far corner.
St Johnstone somehow survived in the 73rd minute when O'Dea missed from barely six yards out after Smith had parried McGeady's shot into his path.
Smith then made a fine save from McCourt but the respite lasted only seconds, until the 76th minute when Samaras pounced on Thompson's knock-down inside the St Johnstone box to send a left-footed volley low into the corner.
With 10 minutes remaining the beleaguered Smith did well to parry McCourt's long-range drive but Fortune was perfectly placed to send the rebound through the goalkeeper and into the far corner.
McCourt then rattled in a fourth after a trademark run to give added gloss to the win which, it has to be said, looked less likely at the break.