Second-half substitute Craig Beattie netted a late winner to get Alex McLeish's managerial reign off to a perfect start and keep Scotland top of their qualifying group.
Scotland almost made the perfect start to the match after four minutes when David Weir sent a header just wide of the target.
But Scotland did not have to wait long for the opening goal, as Gary Teale swung in a cross from the right flank and goal machine Kris Boyd was on hand to head home his fifth goal in six matches for the national team.
Teale should have doubled Scotland's lead minutes later when he was unmarked six yards out with only the keeper to beat. But the Derby County midfielder inexplicably fired straight at the keeper when it appeared easier to score.
Georgia grew in confidence as the first half progressed and following a flurry of corner kicks Georgi Demetradze almost levelled, but Craig Gordon pulled off a fine reaction save to parry the ball clear.
Demetradze was causing problems down the left and his trickery earned Georgia an equaliser just before the break, when he supplied the cross from which Shota Arveladze despatched the ball into the net with an excellent diving header.
Scotland continued to create chances after the interval and Kenny Miller was denied by the Georgian keeper, before Boyd somehow managed to completely miss the ball when it appeared as if he was going to restore the Scots' lead.
Just as the game appeared to be heading towards a stalemate, late substitute Beattie popped up with the winning goal to keep Scotland's qualification hopes on track.