The club announced his departure by "mutual consent" hours after Calderwood led the Dons back into Europe.
The former Dunfermline manager, who secured a top-six place in the Scottish Premier League in each of his five seasons in charge, admitted some board members did not believe he could take the club any further.
But he hinted that his exit may be a relief in some ways as the club expected him to produce a fifth top-four finish despite cuts in his budget.
"People might have doubts about you and you might have doubts about them and where they want to go to," Calderwood told BBC Scotland.
"But it's been wonderful. There's no animosity whatsoever.
"After five years at a big club and not the easiest club in the world to manage although it is a wonderful club and a wonderful city, it was probably best for both parties that we decided to call it a day .
"We want to go one way, the club want to go another way. Maybe they don't think that you can take it any further or certain people might not think you can take it any further than we think we can."
Aberdeen face losing skipper Scott Severin, defender Lee Mair and midfielder Jamie Smith amid the budget cuts.
Calderwood added: "We've virtually taken the season to the last day every year.
"It will be very hard to emulate that, especially with younger kids coming through, so I think it was a decision that had to be made by both parties.
"We could've gone with the kids and maybe the club would've still been expecting us to get into third position which is sometimes very difficult.
"We've built probably three teams and it looked like Scott Severin, Jamie Smith and Lee Mair are going to go and it was going to be another rebuilding job.
"You could go on and on but you can't keep pulling rabbits out of hats. It gets very, very difficult."
Calderwood admitted disappointment that he will not be able to lead Aberdeen back into Europe.
The former Birmingham player took the Dons to the last 32 of the UEFA Cup and guided them into the inaugural Europa League with a 2-1 victory over Hibernian on Sunday to clinch fourth spot in the SPL.
"We're obviously disappointed because we got back into Europe and the last time was a wonderful journey and this season's been a little bit up and down to say the least," Calderwood said.