Gary Doherty headed Norwich in front from a free-kick but Shaun MacDonald equalised from the edge of the box. Jon Obika's header looked to have won it but Doherty claimed Norwich's third by deflecting in Russell Martin's shot.
Skiverton said: "It was a great game to watch and no-one can say we aren't providing entertainment. But three goals at home should be enough to win any game and I was not happy with our defending. As a former centre-half myself I think our central defenders should be winning more headers.
"But I don't want to be too negative. Our attacking is good and MacDonald took his goal really well. We have one more match with him on loan from Swansea and then he goes back to Wales for at least 10 days. We will see what happens after that."
Skiverton is hoping to be allowed to keep the Welshman and also goalkeeper Alex McCarthy, who is on loan from Reading.