The Iron needed to win to stand any chance of pulling off a great escape but made the worst possible start when Kris Boyd put Forest ahead after just nine minutes.
Their survival hopes were left hanging by an even slender thread when Luke Chambers extended the lead midway through the first half.
Michael O'Connor halved the deficit from the penalty spot in the 36th minute but half-time substitute Paul Anderson restored the Reds' two-goal cushion just three minutes after his introduction before late goals from Chambers and Boyd completed the rout.
Scunthorpe will return to League One next season, while Forest look set for a second successive assault on the play-offs.
Billy Davies' side go into the final game of the season three points in front of seventh-placed Leeds and are six goals better off on goal difference.
Scunthorpe started brightly and Mark Duffy forced Lee Camp into a parrying save with a shot from the left side of the box after seven minutes.
But two minutes later, Boyd opened the scoring with a low 12-yard effort.
Joe Murphy kept Scunthorpe in the game with two saves in quick succession to deny Robert Earnshaw and Chris Gunter.
At the other end, Bobby Grant volleyed just wide.
There was little Murphy could do to prevent his side from falling 2-0 behind after 25 minutes as Chambers headed home from close range from Chris Cohen's corner.
Despite the huge setback to their survival hopes, Scunthorpe kept plugging away and Camp tipped a 25-yard effort from Ramon Nunez over the crossbar before blocking a Grant shot.
Their endeavour was rewarded nine minutes before the break when Gunter brought down Nunez and O'Connor emphatically dispatched the resulting penalty into the top corner.
Lee Miller replaced the injured Chris Dagnall immediately afterwards but the change did not affect Scunthorpe's rhythm and O'Connor worked Camp again with a fiercely-struck free-kick that the Northern Ireland international had to push away.
Forest made two substitutions of their own at half-time, with Anderson and Guy Moussi coming on for David McGoldrick and Paul McKenna in a tactical switch by Davies.
It proved to be a masterstroke by the diminutive Scot as Anderson smashed home Lewis McGugan's free-kick from close range in the 48th minute.
That goal seemed to suck the life out of Alan Knill's team, who created little after Nunez had speculatively shot wide from long range after 53 minutes.
Forest too appeared to have less urgency, with their only chance of note in the second half coming after 70 minutes when Murphy pushed away a curling Earnshaw effort from the right side of the box.
That was until eight minutes from time when Chambers headed his second goal of the game from another Cohen cross.
On-loan Middlesbrough striker Boyd then netted his six Forest goal and his 12th of the season with a superb 25-yard free-kick in the final minute.
Nottm Forest 5-1 Scunthorpe: Match Report - view commentary, squad, and statictics of the game as it happened.