The defender ended a 15-year spell at Old Trafford to join Sunderland having made his senior debut for the Red Devils as an 18-year-old in 1996.
But after seeing his playing opportunities decline, Brown felt it was time to draw a line under his time with the club.
He said: "You sort of know what's coming anyway. So in a funny kind of way it's a bit of a relief when someone like Sir Alex sits you down and tells you how it is. Sir Alex's decision wasn't a surprise. I think I'd got my head around it.
"We had a good chat and everything was fine. In a way, I think it meant we could both draw a line under things and move on.
"It was a difficult decision to leave. It was really only this summer where things started to come to a head."
Brown, who did not feature again for United after being substituted in their victory over Bolton in March, claims several teams were after him.
But the 31-year-old defender says he could not resist the lure of Sunderland after talking with manager Steve Bruce, who has also signed Brown's former United team-mate John O'Shea this summer.
"A few clubs came in for me, but I spoke to Steve Bruce," he added.
"I've known him for a long time and what he was saying about taking the club forward and the players he was bringing in showed the club's ambition."