Ferdinand insisted Grafe should have awarded his side two second-half penalties, one when Ashley Young went down in the box and the other when the ball appeared to hit the arm of visiting defender Milan Jovanovic.
Describing both incidents as "definite penalties", the England captain said: "It was just disappointing really.
"You expect the referees to be able to be up with play and performing at the highest level. Today, we just didn't get that.
"When you're playing at international level, you've got to deal with situations that come about.
"Today we had two really good shouts for a penalty and we didn't get them."
Vice-captain Steven Gerrard, who Ferdinand deposed as skipper, blamed Grafe's assistant, adding: "I saw the handball incident very clearly and the referee was really close.
"I was very surprised that he never saw it but even more surprised that the linesman hasn't seen it because it's on his side and, at this level, you need that little bit of luck, especially when things are difficult and you're finding it hard to break the opposition down."