The disappointing reverse left the Hoops 10 points behind league leaders Rangers, albeit having played a game less than their cross-city rivals.
Celtic's championship aspirations suffered a severe blow after second-half goals from Rudi Skacel and Ryan Stevenson gave the dogged Jambos victory in a pulsating encounter.
Skacel broke the deadlock in the 58th minute with an unstoppable drive from the edge of the box, before Kris Commons was shown a straight card by referee Craig Thomson in the 65th minute for a reckless challenge on Adrian Mrowiec.
Stevenson scored a breakaway second 10 minutes from time to leave Celtic in third place behind Motherwell going in to the international break and with their manager worrying about the rest of the campaign.
"It is a concern, obviously," said the Parkhead boss. "We have a game in hand but we have to win that game and cut it to seven and eat away at it.
"But we have to find more consistency in our performances.
"We have lost three games already this season and last season we lost four.
"It is a difficult position that I am in but I understand that position.
"The players are going to have to turn it around, and I am going to have to turn it around.
"A few players have to look at themselves and ask if they are doing enough at the minute.
"There are six, seven months of the season to go and we have to find consistency."
The Bhoys created a number of chances in the capital, but were unable to take any of them and ultimately paid the price.
And, aside from the defeat, it was the bluntness of their attack that irked Lennon the most.
"My biggest bugbear about today is the amount of chances we missed," continued the Irishman.
"We missed some really good chances in the first half.
"We were wasteful in front of goal and away from home, the longer the game is goalless, you give the opposition encouragement.
"If we had got a goal we might have gone on to win the game comfortably but the first goal changed the nature of the game.
"I am getting 100% from the players but certainly not form.
"The heads dropped after the sending off, which was disappointing and there was a lack of belief that we could get back in the game.
"I said to them afterwards that a draw wouldn't have been the worst result in the world.
"But we weren't clinical enough and that was the biggest disappointment."
Lennon criticised Commons, just back in the side after injury and illness, saying: "One player lost his discipline and let us down badly.
"I didn't get a good look at it but from what I can gather it was a lunge and it is high.
"He came in and held his hand up but it is too late.
"We are disappointed, he let himself down and let us down so he will be dealt with accordingly in-house."
Hearts coach Gary Locke was glad to see the Gorgie side get back on track in the best possible way.
The former Hearts skipper said: "It was really pleasing, we played well and you have to do that against both Old Firm clubs.
"We were disappointed with the result last week against St Johnstone and before that against Ayr United so hopefully so have made up for it today.
"We said to the players that if they apply themselves like they did today in every game then hopefully they will go on to have a very successful season.
"Throughout the team we had to be at our best because it was a very good Celtic team.
"We were right up for the game, the attitude and commitment was there right from the start and I thought we played well at times and made chances.
"But we have to keep it going and build on it, we have to be more consistency.
"We can't get carried away and the other games are just as important."
Locke admitted manager Paulo Sergio was ready to replace Skacel with David Obua moments before the midfielder opened the scoring with a powerful drive.
He laughed: "Yes, he was thinking about it and I'm glad he didn't.
"I actually did not see the goal myself because I was telling big David what to do at corners and free-kicks for and against.
"So I will watch it on the television tonight.
"But it was pleasing that Rudi got the goal. He has been champing at the bit to get a run this season and he came in today and scored another valuable goal."