As he prepares to face their former side West Ham on Sunday and make the most of another chance at Sunderland, the accusation still rings in his ears.
'I've grown up with it,' he said.
We are family: Sunderland's Anton (L)and United's Rio Ferdinand
'It's not something that's new to me. I've been Rio's little brother since the age of nine when I stepped into West Ham's school of excellence.
'I remember going to take a throw-in in a game and heard people standing on the sidelines saying, "Ah, he'll never be as good as his brother".
'He's there for a reason for me, not only as a footballing influence, but as a big brother. I'm sure if you spoke to him you'd get the same vibe off him, too. He wants me to do as well as him, if not better. That's the way we are.
'I'm very proud of him and very happy I can go to him for advice.'
The England captain is Ferdinand Jnr's biggest fan, but also his biggest critic and one of several people close to the Sunderland defender who have tried to keep him on the straight and narrow.
'He has told me to knuckle down and play my football,' Anton, at 25 seven years younger than Rio, said.
'He has said, if you play well, then you can't keep talent down.
Head start: Ferdinand will be given a chance to impress at the heart of Sunderland's defence over the next few weeks with injuries to key men
'There's no point having somebody of that stature on the other end of the phone if you're not going to use them and I talk to my brother all the time, about things to do with my football and stuff. All the things I've said to you come from his mouth as well - just keep working hard, don't think about anything apart from the next game.
'I haven't, due to lack of games, done well enough. Last season I didn't play a lot, this season I haven't really played a lot, but hopefully I'll get a run of games now so I can show my worth to the manager.'
And manager Steve Bruce, without first-choice centre backs Titus Bramble and Michael Turner for at least two months, is ready to give him that opportunity.
The pair had a bust-up at the start of the season after Ferdinand was left out of the side and looked to be heading for Palermo. That frank exchange of views in training helped re-ignite Ferdinand's Sunderland career.
Bruce said: 'This is his third year here so it should not really be his chance, but he has to grasp it and make his claim and give me a headache. He certainly has all the tools to do it. The game is easy to him. He's a good athlete, possibly as good as we have got in the club, he's exceptionally quick, a good footballer on the left and right foot but he has to defend more intensely.
Hammer time: Ferdinand comes up against his former club West Ham on Sunday
'We have to get that defensive side out of him, because we can't all be his brother.'
Bruce will welcome back Irish midfielder David Meyler on Sunday, six months after a serious knee injury which threatened to keep him out for more than a year and could have ended his career.
Bruce said: 'He has worked tirelessly, sometimes 11 hours a day, to get back. It is incredible.'
Welbeck is playing too well to stay at Sunderland, laments boss BruceCole could quit West Ham in January, warns Grant as rivals target swoopSUNDERLAND FC