Milner, who has two years left on his Villa contract, has also been tipped as a target for Manchester United.
But Villa, who paid £12 million to sign Milner from Newcastle United two years ago, plan to open contract talks with him at the end of the season.
Commenting on the Milner rumours, O'Neill said: "It is irksome at this stage of the season.
"It doesn't do anyone any favours particularly because the player has had a fantastic time and is enjoying his football immensely here.
"He has had a really brilliant time, particularly since moving into the middle of midfield and I am delighted for him.
"James is still under contract with us and it's a case of dampening fires that weren't there in the first place.
"We want the best players to stay here at Aston Villa so we can keep the momentum going.
"But I have kind of accepted it now.
If it is not James it will be Ashley Young and if it is not Ashley it might be Richard Dunne and then someone else the following week."
O'Neill also denied he knew of any contact from City about the prospect of Milner moving to Eastlands in the summer.
"It has got ridiculous now and I can categorically say that I have not heard anything from Manchester City," he said.
"I spoke to my chief executive late Monday evening and he never mentioned it so I assume they have not been in touch with him either.
"You almost get the impression that stories are put in there to kind of upset you at this stage of the season.
"It was interesting to hear Arsene Wenger talking about Barcelona and having to rebuff stories about someone who was under contract at Arsenal.
"They are having to get Barcelona to make statements about them not coming after a player."