Spurs reacted furiously to Chelsea's £22million bid for Modric, with manager Harry Redknapp branding the offer "ridiculous" on Thursday.
Redknapp wants to hang on to Modric, who performed superbly for the club last season, and insisted on Friday morning that he will not sell the Croatian at any price.
The Blues are determined to bolster their midfield this summer though and could return with an improved bid, while Manchester United and big-spending neighbours City have also been linked with the 25-year-old.
Tottenham sold Dimitar Berbatov to Manchester United three years ago, but they have generally stuck to their policy of not selling their best players and chairman Daniel Levy has assured supporters that will be the case this summer.
Redknapp thinks Modric will be key to the club's future chances of gaining the top-four spot they missed out on last year, and Lineker spelt what he sees as the potential consequences of selling the star when he said: "This is a defining moment where we will find out what will happen in the future for the club.
"These are really testing times for Tottenham. It's crunch time for them.
"Once you become a club that loses your best players it becomes awfully difficult to continue to compete and try to get back into the Champions League.
"If you lose someone like Modric he will be difficult to replace and for me, the club are now either going to go on to be one of the big clubs or one of the ones that are in the pack behind. If they do drop into that pack, they'll probably be there for a long time."
Modric admitted earlier this month that he would be ready to discuss a transfer, but only if the club shifted their stance and were willing to sell him.
With Chelsea, United and City all able to offer the former Dinamo Zagreb man the Champions League football that Spurs cannot, Lineker fears the player may leave.
"My gut feeling says yes he will leave," said Lineker, who was speaking at a Sony-sponsored event with British tennis star Heather Watson on Friday morning.
"It largely depends on Modric now. Players have a lot of power. If they want to leave they will make it awkward enough to enforce that."
After guiding Tottenham to the Champions League quarter-finals last season, Redknapp has lofty ambitions for a club he took over when they were at the bottom of the table in October 2008.
He thinks he can bring major silverware to the club and Lineker reckons they have the potential to challenge for the title if they hang on to their best players and bring in one or two top-class additions.
"I think Spurs are genuinely only a couple of players short of being a side that can really compete with the big boys," he said.
The former Tottenham frontman thinks a striker must be a priority after the poor performances of Jermain Defoe, Peter Crouch and Roman Pavlyuchenko last term.
"They need a top-quality striker who will get them 20 or 30 goals a season guaranteed," Lineker said.
"That would make a significant difference. That's what they lacked last season. None of the strikers scored enough goals."