The impressive 19-year-old has attracted attention from a host of top-flight clubs with Aston Villa and Everton both reportedly having bids turned down.
Now Tottenham have declared their interest in taking the England Under-21 international, who has an £8million price tag on his head, to White Hart Lane and Grayson knows it could be tough to keep hold of him this summer.
"We're not talking about him going to a League One team," Grayson told the Yorkshire Evening Post.
"We're talking about top Premier League clubs coming in for him.
"The hardest thing for us would be accept a bid in the first place, but also to deny the player to the chance to go to that level."