Orient suffered a first-minute blow when goalkeeper Jamie Jones went down in a goalmouth melee in which the ball was scrambled off the line, and after lengthy treatment was stretchered away.
Striker Kayode Odejayi went on to score the only goal of the game and Williams said: "It was a frustrating afternoon and a tough game physically.
"We knew that was going to happen. We had to defend long throws and set-pieces, which we did. It was all the more frustrating because it was at a set-piece when we lost two headers and that cost us the game.
"We wanted to get the ball down and play and it was very difficult, but give Colchester credit as they are a big strong side and well-organised."