Gary Teale gave the Championship side an early lead before Paul Warne levelled shortly before half-time.
Kevin Ellison struck halfway through the second half and the Rams could not find an equaliser to go out at the first hurdle in contrast to last season's semi-final heroics.
Clough, though, had no arguments with the outcome and said: "Rotherham thoroughly deserved it.
"I was disappointed by the manner of the defeat but I thought Rotherham thoroughly deserved their victory, but it's the manner in which you lose, not just the fact of losing.
"We weren't as good as we were on Saturday. Having gone a goal up to go and then concede two goals in normal time and in the manner in which we did, I am very disappointed.
"We didn't pass the ball around, but credit to Rotherham for the way they closed us down. We nicked the goal and were looking to get through to half time, but we didn't."
Rotherham boss Mark Robins said his team were "excellent" from start to finish.
"We started the game brightly and had a couple of half chances, but they scored with their first attack, but we didn't lose any heart from it.
"It was a big scalp for us and I am delighted. I am pleased with the manner of the performance and the manner of the win, and I am delighted for Paul Warne."
The home side started brightly and some early pressure saw Warne shoot just wide, but then it was Derby who went ahead after 11 minutes from their first attack.
Paul Connolly's pass set Lee Croft free on the right and with everyone expecting a shot he pulled the ball back across the edge of the area for Teale to drill home.
But Rotherham refused to lie down and they kept Derby pinned in their own half. Pablo Mills went close after 22 minutes and their pressure finally paid off a minute before the break.
Jamie Green's free kick into the penalty area was headed down rather than clear and Warne pounced for his second goal of the season from 10 yards out.
Nicky Law almost set up Warne for number two just after the break, but with 65 minutes gone Rotherham went in front. Law crossed from the left and Warne laid the ball off for Ellison to fire home.
And, they almost rescued it, but Dean Moxey's injury-time shot was just inches the wrong side of the post.