The 23-year-old midfielder played for his beloved Leeds when they imploded, then moved to Newcastle where Graeme Souness said he would not be "buying a team of James Milners".
It is only this season at Aston Villa where he has felt stability and it has coincided with a call-up to the England senior squad and playing a key role in the under-21s reaching the semi-finals of their tournament in Sweden.
"Those setbacks help you mentally," Milner said.
"It's how you show your character."
That resilience was displayed in Gothenburg when Milner missed a first-half penalty against Spain but picked himself up and scored the second goal in a victory that sealed a place in the last four.
Milner is suspended for the Group B clash against Germany next week but will return for the semi-final and hopefully the final.
Even if they win the tournament, Milner is teetotal and will not be celebrating with alcohol.
"You get one chance at football," he said.
"There is nothing wrong with having a drink at the right times, you see (Cristiano) Ronaldo having champagne and it doesn't make him a worse player. It's not in my nature and that is the sacrifice you make to make it to the top. Your friends are going with girls and drinking cider - it's a decision you make."
Milner's mental toughness is something he developed at Elland Road.
"It probably helped me playing at 16 in a Leeds team where things weren't going right," he admitted.
"There were things going on off the field, court cases, relegation, financial troubles. That stands you in good stead and helps you to be strong mentally.
"It seems that until last season there was always instability at the club I've been at and I've been worrying about managers getting sacked. Obviously that is not easy, you have to prove yourself all over."
That was the case at St James' Park where there was a quick turnover of managers and Milner being name-checked by Souness when the Scot was outlining the problems of the club.
"It's not an ideal thing for your manager to say," Milner said.
"The mentality from me was to prove him wrong, that is all you can do as a player. It's down to you to prove your ability and show those people who don't rate you that they are wrong.
"You remember the lows more than the highs I suppose and it is how you learn from those."
Ironically, Milner was in the Villa side that defeated Newcastle to relegate them last month, and he now fears them plummeting like Leeds.
"Touch wood it doesn't happen but you look at the bottom of the Championship from last season and you see how many teams there have been in the Premiership," he added.
"Whether it's players wanting to leave, wages, getting that stability, bringing new players in - it's not easy to come back."
Milner is confident that Villa will maintain their stability despite losing Gareth Barry to Manchester City, adding: "I'm sure the manager will make the right signings."
Villa boss Martin O'Neill has certainly sung the praises of Milner this season, with his midfielder now hoping to sign off from under-21 football with a winners' medal after getting to the semi-finals in 2007. He also got to the last four with the under-17s in 2003 but on both occasions lost on penalties.
Milner believes there is more at stake than just silverware - he wants England to break through the mental barrier of crashing out on penalties.
"We went very close last time, losing in a shoot-out in the semis," Milner said.
"You can get into a mentality of 'here we go again'.
"So if we won it would be very important. It is important we get out of that trend of losing on penalties in every major tournament.
"Look at the Spanish players who won the European Championship and so many played in the under-21s. If we get into the winning mentality at youth level we can take that into the senior side."