The Union, who became the 16th MLS team when the league started at the end of March, had been linked with a move for former Arsenal star Pires, who is a free agent after his contract expired with Spanish side Villarreal.
Pires, 36, would have been signed as the club's first 'designated player', meaning the majority of his salary would not have counted towards its salary cap.
But he was reported by French newspaper L'Equipe as having rejected an offer from the Union and MLS to join the club in July because Philadelphia was not "beautiful".
"It does not please me to (go to Philadelphia)," Pires told L'Equipe. "I do not want to be difficult, but if I had to take my family away, I'd rather do it in a beautiful city. Otherwise, I will put an end to my career."
Union chief executive Nick Sakiewicz told the Philadelphia Daily News that Pires had turned down an invitation to visit the city.
"It's a bit of a shock as to where all this is coming from," Sakiewicz said. "A, he's never been here and B, we never made him an offer.
"I think Robert is a good player with a lot of experience and a lot of good football left in him, but he may just not have been for us; and obviously, given his comments, he clearly is not for us."
Union head coach Peter Nowak also criticised Pires for not understanding the intricacies of MLS transfer policy, claiming that having turned down his club he was, in effect, turning his back on a move to any team as a 'designated player' in the single-entity North American league.
"People need to understand the rules before they open their mouth," Nowak told the Daily News.
"Also, I think he is looking at cities like Chicago, New York, Miami and Los Angeles and thinking Philly is a small town compared to them. There are a lot of guys like this, where except the four major [cities], they don't know much about the US.
"When you come to the US, you are trying to mould and improve the game as much as possible. They aren't just here to better their team but to help raise the profile of the sport.
"Mr Pires is supposed to come here and promote soccer, not worrying about being in a vacation resort in some 'beautiful' city."