The United defender also hit out at referee Martin Atkinson's decision to award Carlo Ancelotti's team the free-kick which ultimately led to the 76th minute goal.
Brown insists he was 'held back' by Chelsea striker Drogba and says United did not deserve to be on the losing side.
Grounded: Brown (floor) tumbles under the attention of Drogba while Anelka (cente) heads home Chelsea's winner
He told MUTV: 'I don't think the referee played well and we've been let down by that goal. The decision to award the free-kick was harsh.
'Then Drogba was holding me and pulling me down, obstructing me from challenging for the ball when the free-kick came in the box.
'The ref didn't see that either - what can we do about it?'
'It's very hard to take, we played really well. We were the better team.
'We should have done better with our chances but we definitely didn't deserve this result and it's tough because we go home with no points.'
Chelsea 1 Manchester United 0: Nicolas Anelka lands title blow as champions are floored by disputed late winner Chelsea vs Manchester United matches remembered: What is your favourite clash between the Reds and the Blues?EXCLUSIVE: England boss Capello ends Michael Owen's World Cup dreamChelsea's diamond worthless as they don't have Kaka-type player - Fergie
Explore more:People:WES BROWN, Carlo Ancelotti, Martin Atkinson, Didier Drogba