Midfielder Wordsworth met Clive Platt's cross at the far post and the midfielder's scuffed low shot was missed by Shrimpers skipper Adam Barrett on his own goal-line and the ball rolled in.
Scott Spencer and Francis Laurent went close for Southend before half-time but Colchester doubled their lead in the 63rd minute.
The Blues failed to clear David Prutton's corner and after Kayode Odejayi had headed back across goal, Wordsworth tapped in from close range.
Colchester goalkeeper Ben Williams saved from both Southend substitute Matt Paterson and Frank Moussa before Barrett had a goal chalked out, as the U's took the bragging rights.
Colchester stay fourth but now lie just two points behind third-placed Charlton with a game in hand. Southend remain in 17th position, two points above the drop zone.