Dickson scored 11 goals in 14 appearances for Gillingham two seasons ago but he joined Bristol Rovers on loan from Charlton rather than the Gills.
"It was really messy, it doesn't help us being so close to them but we have just got to get on with the players we've got," he told BBC Radio Kent.
"The kid just wants to play football. Being the rivals of Charlton didn't help."
Meanwhile, chairman Paul Scally told the Kent on Sunday: "I don't really want to have any dealings with Charlton other than those I have to have, ever.
"Phil Parkinson (Charlton manager) appears to be paranoid that Chris Dickson was going to come and do well under Mark Stimson. That seems to be the basis on which they wouldn't let him come to us."