Under-pressure manager Steve Bruce substituted the tough-tackling former Middlesbrough and Wigan man after he received his third booking of the season with 20 minutes remaining in the 2-2 draw with West Brom prior to the international break.
It was a change designed to protect Cattermole, who has been booked 47 times and sent off on five occasions in domestic competitions since 2005/06, but he defends his record.
"I feel like they (the referees) are out there for me at the minute," he said. "Maybe it's my stupid mistakes in the past that have got me that reputation, so I can't sit here and blame everyone else. I have to get rid of that reputation, but it's so hard at the minute when I find myself getting a succession of soft yellow cards.
"It affects your game. It's something I have to address, but if you put every foul I've ever made up on the television, I think there'd be one tackle which was actually a bad one.
"That was against Scott Parker when I was playing for Wigan and I've spoken to him since to apologise about it. That's the only bad tackle I've made in my career - all the rest have been genuine attempts to win the ball, not nasty attempts to hurt someone."
Cattermole was furious after being substituted against West Brom and believes Baggies midfielder Steven Reid is just one of several players who attempt to paint him as a villain.
"The tackle I got booked for wasn't a foul, simple as that," he said. "But it's twice it's happened now with Steven Reid. He's six foot four and he rolls around like a girl, simple as that.
"It's twice it's happened and it's a shame. People whinge about the foreigners, but he's a British lad and he's acting like that. I was booked for that challenge, and then the second one was on him again."
Captain Cattermole had returned to Sunderland's starting line-up against West Brom having been dropped for the 4-0 win over Stoke.
"The gaffer probably changes three players for every game, but as soon as it was me it was a big issue and highlighted everywhere," said Cattermole.
"He told me the reasons, asked me if I wanted protecting in the papers and I said I didn't want anything. I just said, 'tell them it's my form and I'll bounce back from it'."