Goals from Kirk Hudson and Scott Davies, in between a Tom Craddock leveller, leaves the Hatters 13 points adrift of safety.
Harford, who felt his side deserved something from the game at the EBB Stadium, is now looking to Tuesday's home clash with fellow strugglers Grimsby.
"No disrespect but we shouldn't be losing to Aldershot Town," said Harford.
"The result is disappointing because we seem to get ourselves in a position where we look like we can get in touch with the teams above us but we fall at that final hurdle.
"We'll get ready for Tuesday because we still have the belief, the hope, the burning desire to keep this club in the league."