The Frenchman, who has 12 months remaining on his contract, was set to join Birmingham last summer for £9million only for the deal to fall through over personal terms.
Whelan said there were a number of sides interested in the 25-year-old winger this time around - Sunderland and Newcastle have also been linked - and although he expects the player's departure from the DW Stadium it will only be for the right money.
"I do think he will go this summer, but we want the right price before we allow that to happen," said the Latics chairman.
"We have a fee in mind that we want, and we are fighting to get that fee.
"There's been an awful lot of interest in Charles. We've had three or four clubs biting, wanting to be kept in the know.
"There's been no official bids yet, and I think it'll be another week to 10 days before things get as serious as that.
"But he's worth a hell of a lot more than the fee we're asking.
"Whichever club gets Charles will be getting a great player for far less than what he is actually worth."
Whelan admits, however, N'Zogbia's aborted transfer to St Andrews last year could have tainted his reputation.
"Clubs are perhaps looking at him and thinking twice, thinking he's difficult, but that wasn't his fault at all," he told the Wigan Evening Post.
"It was the agents who were to blame for what happened, not Charles."