Glenn Murray became the sixth Albion player to be sent off this term when he saw red for two bookings - the second a needless handball - during Saturday's 2-1 defeat at Tranmere.
But defender Virgo, who himself was dismissed earlier in the campaign, felt Tranmere duo David Martin and defender Marlon Broomes could have also gone for early baths for denying goalscoring opportunities.
"It's not helping us, I'm part of that as are five other people," he said. "But we kept going and maybe if a few decisions had gone our way we could have had 10 men each. You look at the goalkeeper.
He's stopping a goalscoring opportunity. Look at the defender, it was handball.
"The referee told me it was too wide to give a red card. I can't understand for the life of me why a red card hasn't been given because Gary Hart was through on goal.
"It was a silly red card for us of course and some of the yellows were for petulance which wasn't helpful."