The Bees went ahead through Carl Cort's header before the hosts saw a great chance to level matters come to nothing when Scott McGleish's spot-kick was saved by Nikki Bull.
However, McGleish redeemed himself by grabbing Orient's equaliser before Danny Foster inadvertently diverted Andros Townsend's cross into his own net for the winner.
And Williams was over the moon to see his side come from behind to grab only their second home win of the season, saying: "After the start we had it was a big win for us. A month ago people talked about a home hoodoo. A start like that would have killed us earlier in the season.
"To come back from that, show that bottle and that heart and win the game says a lot about the players."
Meanwhile, McGleish insists he will be the man to continue taking penalties, adding: "There are about seven others who want to take them. But I have confidence in myself and if we get one I'll take it."