After a quiet opening to proceedings, the Millers, who made five changes from their weekend win at Brentford, took the lead in the 17th minute with Joe Skarz's cross unfortunately striking Chris Smith and diverting past Michael Ingham, who was rooted to the spot.
York were unlucky not to restore parity prior to the break as Elliott Whitehouse and Ryan Jarvis both went close with headers.
The hosts did have the ball in the net before half-time but Wes Fletcher's strike was ruled out for offside after he reacted quickest to Adam Collin's parried save from Ashley Chambers' effort and Pablo Mills also cleared a header off the line.
But the League One side gave themselves a two-goal cushion in the 63rd minute as Kieran Agard sent Ingham the wrong way from the penalty spot after Lewis Montrose had brought him down.
Six minutes later the visitors put the result beyond doubt when Alex Revell rose highest in the penalty area to power a header home from Richard Brindley's pinpoint cross.