A number of players had been quoted in the French media saying they were keen to play on as a mark of respect to those killed and injured in Friday's gun attack in the Angolan enclave of Cabinda.
Horror ordeal: Adebayor says Togo are going home
But Togolese prime minister Gilbert Fossoun Houngbo ordered the team must return today.
The players have now agreed and Togolese president Faure Gnassingbe has dispatched his presidential plane to collect them.
Armed guard: Angolan police monitor the Togo team bus as it leaves the Olympic Village
Adebayor told French radio station RMC: 'The head of state has decided we will return.
'There was a meeting between players yesterday (Saturday) and we said we were still footballers.
'We all decided to do something good for the country and play to honour those who died.
'Unfortunately, the head of state and the country's authorities have decided otherwise. We will pack up and go home.'
Anxious moments: Togo's reserve keeper Kodjovi Obilale arrives on a stretcher at Lanseria airport after being evacuated to Johannesburg for medical treatment
Adebayor's comments come after Houngbo announced the government's decision earlier today.
The Togolese prime minister told L'Equipe: 'We understand the approach of players who wanted to express a way to avenge their fallen comrades, but it would be irresponsible on the part of the authorities to let them continue.
'The team must return today. The government's decision is unchanged. It is a conscious decision since Friday. If a team or a person present themselves under the banner of Togo, it would be a false representation.'
On red alert: A member of the Angolan police special forces guards the Olympic village
Manchester City striker Adebayor revealed a conversation he held with Gnassingbe this morning had proved decisive.
'That's what made the difference,' he said. 'It was also our families and loved ones at home who called us.
'They told us we could continue if we wished but that it is the authorities who have the information.
'Is there going to be another attack? Nobody knows. If they asked us back (home), maybe they received a call saying that the threat was not passed.
'We are obliged to respect that. The head of state knows what is good for our careers and our lives.
'The presidential plane will pick us up,' he added.
'He told me that the plane had left Lome. There are about two hours flying between Lome and Cabinda. We will leave in two or three hours.'
The Togo players had voted to continue in the tournament despite an earlierdecision to withdraw as the death toll from the attack reportedly rose to three.
Adebayor admitted he had wanted to play on despite fear of another attack, but accepted the authorities knew best.
'This Friday at 14:30, we were all dead on that bus,' he said. 'We sent our last messages to our families. We called our family to say our last words.
'I told myself: 'If you're still there on the ground in Angola, why not (play in) the CAN?'
'Today, the authorities decided we should return, so we will return.'
Casualty: Togo assistant coach Abalo Amelete was reportedly killed in Cabinda
Adebayor revealed he has spoken to members of the Ivory Coast and Ghana camps - two of Togo's Group B opponents - who appeared to back a boycott but now seem set to continue.
'As captain and spokesman of Togo, I spoke with all authorities. I told them to take the measures necessary for our security.
'I spoke with the captain and the Ivory Coast delegation and with Ghanaians. They expressed their support by saying they were ready to leave the competition if we did.
'(But) at the end of the day, we realised that they were ready to continue. It is still a continent where a World Cup will take place in South Africa.
'If we speak of the dead, the competition should have been cancelled. But CAF(Confederation of African Football) have decided otherwise.
'We're going back and we wish good luck to those who will remain, especially to Burkina Faso, Ivory Coast and Ghana.
'What I have told their leaders is that they may be attacked at any time in Cabinda. I hope they will be cautious.'
Togo's Africa Cup of Nations plans still unclear after horrific gun ambushPremier League stars to flee Africa: Emmanuel Adebayor pulls out as terror attack leaves three deadIAN RIDLEY: After this disaster at the Africa Cup of Nations, let players decide if they go to World CupCapello's number two Baldini risks trip to Angola despite terror attack