Paul Connolly's pass set Lee Croft free on the right and with everyone expecting a shot he pulled the ball back across the edge of the area for Gary Teale to drill home.
But Rotherham refused to lie down and they kept Derby pinned in their own half. Pablo Mills went close after 22 minutes and their pressure finally paid off a minute before the break.
Jamie Green's free kick into the penalty area was headed down rather than cleared and Paul Warne pounced for his second goal of the season from 10 yards out.
Nicky Law almost set up Warne for number two just after the break, but with 65 minutes gone the League Two side went in front. Law crossed from the left and Warne laid the ball off for Kevin Ellison to fire home.
County almost rescued it, but Dean Moxey's injury time shot was just inches the wrong side of the post.