The visitors went ahead through efforts from Gary Hooper and Henri Lansbury inside the first half and were looking in control until Sarginson's intervention.
Adkins said: "At 2-0 up at half-time we were looking forward to playing some good football in the second half on a beautiful pitch.
"The referee had an interesting game to say the least. I haven't seen the tape of the match and I will not comment until I have seen it, but it was a shame we couldn't have 11 players on the pitch.
"We knew we were going to be stretched (when the players were sent off) so we set up two banks of four and said 'break us down if you can'. Swindon used the pitch to their advantage when we were two short."