Grant felt that Oliver should have awarded his side a late penalty after City substitute Jean Beausejour appeared to pull back Lars Jacobsen inside the penalty box, and labelled the decision "a farce".
Grant believed it was the latest in a line of decisions from officials to go against the bottom-placed Hammers, who had looked on course for their first away win of the season after scoring through Frederic Piquionne and Valon Behrami.
But the home side fought back and goals from Cameron Jerome and Liam Ridgewell rescued a point.
Grant said: "Harry Redknapp used the word farce about the decisions of a referee and I agree with him.
"The referee was five metres from the incident when the shirt of Lars was pulled and he did not give a penalty.
"I'm very disappointed because this is not the first time it has happened to us.
"Similar things have happened against Wolves, Stoke, Aston Villa, Bolton and Manchester United where decisions have gone against us.
"What we saw from 50 metres, the referee could see from five metres and he needs to give a penalty. This is a farce.
"Sometimes you don't see things but this was in front of his eyes. I don't understand why he didn't give a penalty."
Grant took solace from his side's performance and said: "You saw a team that deserves to be in the middle of the table.
"We conceded a couple of easy goals and it cost us but if we continue to play like this, and show the same attitude, the points will come.
"We carried on like we have been playing in the last two games and Ben Foster made two unbelievable saves from Carlton Cole and Victor Obinna."
City manager Alex McLeish believes Foster will be pushing for an England spot if he continues to maintain his current form.
He said: "It was a fantastic save from Cole and Ben has been totally on top of his game from the day he arrived.
"He is in the frame for England, and him and Joe Hart are playing out of their skin at the moment."
McLeish admitted: "In the first half there was no penetration from us, we were not a threat but when we went more direct in the second half we caused more problems.
"West Ham are not a bad side. They have good individuals and Avram is trying to get them playing as a team.
"For an hour they played well but, when we got at them, we were a different side."
Ridgewell has often weighed in with important goals for City and he produced another clinical finish to salvage a share of the spoils.
McLeish said: "Liam has had an eye for a goal going back to his youth team days and he's brought that into his time as a senior player.
"He takes a gamble in following the ball into the box and sometimes gets on the end of things."