The 28-year-old striker signed for Derby from Charlton in November 2008, initially on loan, before making a permanent move in January the following year. However, he made just 15 appearances for the Rams, scoring only once.
Varney spent almost all of the 2009/10 season on loan at Sheffield Wednesday and last season was on loan at Premier League Blackpool, where he scored five times.
"Luke leaves with our very best wishes. Despite not being in our first-team plans, he has been a great professional and shown an exemplary attitude on and off the training pitch," said Derby manager Nigel Clough.
The former Quorn FC striker rose to prominence at Crewe, where his 35 goals in 108 matches earned him a £2million move to Charlton in 2007.
Varney will now be hoping to help Pompey reach the top-flight after penning a three-year deal.
"I'm delighted to be here," he told the club's website. "Sometimes contracts can drag on through the pre-season, so it important for me to be sorted early on.
"It's really exciting for me to get to training, meet the boys and push on through pre-season. Once I heard that Portsmouth were interested, it was something that I couldn't turn down.
"It's exciting times at the club, people I've spoken to have told that it's a club that's going places and it's something I'm really looking forward to it."