Bayern Munich centre-back Demichelis fired home from close range in the 77th minute after his header from Lionel Messi's corner bounced back off team-mate Diego Milito.
- Greece 0-2 Argentina - As it happened
- Group B table
- Group B fixtures
Boca Juniors striker Palermo tapped in the rebound from Lionel Messi's shot to complete the victory a minute from time.
It sent Argentina into the knockout phase to face Mexico full of confidence on the back of three successive wins as Greece's best defensive efforts ultimately proved to be in vain.
South Korea's 2-2 draw with Nigeria put them through as Group B runners-up.
For long periods Diego Maradona's side were frustrated by Greece, for whom defending is what they do best - having used the tactic to become surprise European champions in 2004.
However, what Otto Rehhagel's side desperately needed more than anything was a goal and despite the unstinting efforts of Celtic striker Georgios Samaras - who at times appeared to be the one in a 10-1 formation - they just did not threaten enough.
Argentina, by contrast, did not really need to exert themselves as they were virtually guaranteed to progress as group winners before the match kicked off.
Messi captained the side after Maradona made seven changes from victory over South Korea but found himself closely marked throughout.
He had one early trademark run towards the penalty area which was halted by Sokratis Papastathopoulos but otherwise was effectively shackled throughout.
Sergio Aguero's first sight of goal saw the busy Alexandros Tzorvas palm his shot away for a corner in the 18th minute and the goalkeeper then immediately had to tip over Juan Sebastian Veron's 30-yard strike.
Tzorvas almost undid all Greece's hard work when he spilled Diego Milito's low cross into the path of Aguero but Loukas Vyntra charged down his shot at the far post.
But he redeemed himself in first-half added time when he saved from Liverpool midfielder Maxi Rodriguez and Messi in quick succession.
Three minutes into the second half Greece almost snatched the lead when Samaras beat Demichelis to another long ball out of defence and raced into the penalty area but drove wide of far post with his left foot - Greece's first shot of the match.
But injuries were not kind to Rehhagel's side as they lost Georgios Karagounis, Konstantinos Katsouranis and Vasileios Torosidis all within 10 minutes of the restart.
Argentina pressed on regardless with Milito inches from heading in a cross from left-back Clemente Rodriguez, who himself hit a dipping shot just wide.
In the 68th minute Messi's long-range free-kick brought another save out of Tzorvas and from the resulting corner the ball dropped to defensive midfielder Mario Bolatti but he shot straight at the goalkeeper from six yards.
It was 24 years to the day since Maradona scored his infamous Hand of God goal against England in the quarter-finals of 1986 Mexico World Cup - Argentina's last triumph at this tournament - and the frustrated players might have felt like indulging in similar skulduggery themselves.
But Demichelis saved anyone the embarrassment by drilling into the roof of the net with 13 minutes to go.
Things opened up after that and Messi smashed a left-footed shot off the post having weaved his way into the penalty area.
His next effort was more productive as Tzorvas parried his strike into the path of the 36-year-old Palermo who took full advantage.