Former Huddersfield midfielder Holdsworth, who has also turned down Millwall this summer after being released from the Galpharm Stadium, was offered a permanent deal by McCall after impressing during a trial spell that included two friendly matches.
McCall, who saw his side lose 3-1 in a behind-closed-doors match with a Manchester City XI on Tuesday, told the club's official website: "It's frustrating, but you can't blame the lad.
"I got a call to say he was going to Oldham and wants to carry on playing League One football. He did well when he came here, enjoyed it all and we offered him a deal.
"But once Oldham were sniffing we knew it would be tough and he has decided to go there."