The French striker spent six months on loan at Anfield from Paris St Germain in 2002 but then manager Gerard Houllier decided not to offer him a permanent deal, instead recruiting El-Hadji Diouf.
While the Reds now rely heavily on Fernando Torres to provide their goals, Anelka is thriving at Chelsea and was the Premier League's top scorer last season.
Carragher has been impressed by Anelka's performances and wishes Liverpool had opted to keep him when they had the chance.
"I think it's a bit strong to say it was one of the worst decisions ever, but it's fair to say it was a mistake," he told The Sun.
"You look at Anelka now - he was top scorer in the Premier League last season. It's in the past now, but it was probably something that never went our way."
Title hopeLiverpool are in danger of missing out on the league title again this season after a poor start to the campaign but Carragher is refusing to give up hope.
He is determined to help the Reds claim top spot for the 19th time in their history and is not thinking about the prospect of fierce rivals Manchester United overhauling their record.
"I fear not winning the title more than seeing United go above us in title wins," he explained.
"Don't get me wrong, I don't want to see them going above the record we've held for a long time, but we can't keep holding on to that record and looking into the past.
"We have to do it ourselves rather than hope Manchester United don't win it."