Clayton McDonald snatched a controversial 81st-minute winner for the Saddlers, a linesman ruling that the ball had indeed crossed the line.
Grayson refused to hide his anger, and said: "The first half was so dire it was unbelievable. We had no enthusiasm, no quality and no passion - every ingredient you need to win a football match.
"I could have taken any player off at half-time because none of them did themselves any justice.
"We got what we deserved out of the game, which was nothing. I am disappointed but I know we have enough in the dressing room to get us results. We just didn't show any quality. We knew Walsall would be hard to beat but our decision-making and desire were missing."