The Black Cats trailed 2-0 at Carrow Road before left-back Richardson's 86th-minute strike gave them hope of staging a late fightback.
However, Steve Bruce's men came up short to go down 2-1 and leave themselves under pressure once again ahead of West Brom's visit to the Stadium of Light on Saturday.
Richardson told the club's official website: "I was pleased to score but the result overall is very disappointing.
"We were on a high after the Stoke game, but we didn't play well tonight. We had possession in the first half, but it wasn't effective.
"It's a shame because we have let the fans down and I am sorry for that."
Sunderland had arrived in Norfolk full of confidence after trouncing Stoke 4-0 on Wearside last weekend, a result which eased the pressure on Bruce after a difficult start to the season.
However, they fell behind to Leon Barnett's 31st-minute strike and Steve Morison's header three minutes after the break effectively wrapped up the points despite Richardson's late intervention, with substitute Connor Wickham having earlier passed up a glorious opportunity to open his account for the club.
Richardson, 26, said: "I thought we were good enough to get back into the game, but it wasn't to be.
"In the second half, we didn't put our foot on the ball at all. We didn't get in possession, whereas at least we had some of the ball in the first half.
"We still had chances in the second half, though. Connor had a chance with a header and was unlucky - on another day that would have gone either side of the keeper and gone in."
The defeat left the Black Cats in 14th place in the table with only five points from their first six games, and just a single point above the drop zone.
Bruce had hit out at the "hysteria" which had accompanied the club's results during the opening weeks of the campaign in the run-up to Monday night's fixture, but the performance and result at Carrow Road will have done little to appease his detractors.
He continues to enjoy the support of chairman Niall Quinn - Sunderland have privately played down speculation that the Irishman could quit the club for a role at Manchester City - and owner Ellis Short.
However, a rapid improvement in fortunes is required if the manager is to avoid further criticism.