Ivanovic hardly figured for Chelsea during Grant's brief time as manager of the Blues and the Serbia defender, who was named in the PFA Premier League team of the year, admits it was the most difficult time of his career.
He is confident of overcoming a minor knee injury to play against Grant's Portsmouth team and hopes to go into the history books alongside the rest of his Chelsea team-mates by adding the FA Cup to the Premier League title they secured last Sunday.
Ivanovic was signed from Lokomotiv Moscow for £9million in January 2008 but injury kept him sidelined for much of the season as Grant, who took over from axed Jose Mourinho four months earlier, kept Chelsea in the hunt for the domestic league title and the Champions League. But when he was fully fit, he found himself isolated under Grant's regime.
"I have no ill-feeling towards Avram but at that moment I was really angry," said Ivanovic.
"I understand him now because I couldn't change a lot of things in that moment.
"When I arrived it was difficult for a lot of reasons. I had come from the Russian league when there was a break between seasons.
"I was not ready to play. I needed time to be able to get used to the training, the play and everything about the club.
"But after that I picked up an injury and when I returned it was near the end of the season and every game for Chelsea at that stage was important.
"The squad was very strong and they were trying to get to the Champions League final. They were also involved in a very tough title race with Manchester United, like it was this year. However, it was the most difficult time in my career."
Ivanovic admits he came close to quitting Stamford Bridge but is now enjoying his football.
"I don't think you can find a player in the world who is happy when they're not playing," said Ivanovic.
"This is normal, it's a human thing. You think you deserve to play but a manager's job is to find the players he wants to play. Of course I was very angry. Maybe I was thinking [about it] but that was two years ago. Things have changed now. It wasn't in just one day, it changed step by step."
It was advice from former Chelsea striker Andriy Shevchenko which convinced him to stay put.
Shevchenko also found it hard to adapt to English football and Ivanovic was grateful for his help.
"I can't forget what Andriy did," added the Serbian.
"He really helped me. He tried to give me confidence. He told me about the style of play in England, what I had to do as a defender and where I could improve.
"It was a difficult period for him, too. He had been outside the team, he was also injured, but he wasn't just thinking about himself, he was thinking of me too. I can only say a big thank you to him for everything he did because he helped me become prepared for when I would play."