Ferguson has spent nearly his entire career in Scotland with Rangers, but had a brief foray into England's top-flight with Blackburn a few years ago.
After an injury-plagued stay at Ewood Park, Ferguson opted for a return to the blue-half of Glasgow and continued his career north of the border.
But the 31-year-old claimed that he became an integral part of Rovers' team during that period, a fact he says many of the doubters are quick to forget.
Ferguson told the Daily Mail: "I know people are looking at it. They are saying: 'He had 20 months there, he was out for eight, so what did he prove?'
"I felt I was doing well but I picked up a bad injury at Newcastle. You can still see the scar on my left knee. And that's all they remember. After that, they saw I went back to Rangers.
"I don't want to defend myself or sound arrogant but when Souness left, Mark Hughes wanted to build the team around me.
"I was offered a contract extension, if the people at Blackburn thought that, I must have been doing something right."
The former Scotland international made the switch to Birmingham under a cloud of controversy in the summer after he made obscene gestures on television when sitting out a World Cup qualifier for a previous breach of discipline.
Along with former team-mate Allan McGregor, Ferguson was banned permanently from representing his country and transfer-listed by Rangers.
Although Ferguson accepts that he made a mistake and deserved punishment, he is still disappointed with how the situation was handled by the Scottish FA.
He added: "I'm not the first guy to make a mistake, am I?
"I accept I made those mistakes and I felt it was handled well by Rangers, who did it man-to-man.
"But that didn't happen with Scotland. I was told I wasn't playing again by a piece of paper. I thought it was poor.
"If you have a problem with a guy, go and speak to him face-to-face, That's what my dad brought me up to do.
"Now I realise what went on, I've said my piece. I've got to be honest, after everything that happened at the time, I did wonder whether I wanted to carry on.
"I've got other interests, business interests, that I could have concentrated on away from football."
Tottenham 1/3, Draw 7/2, Birmingham 10/1