The Iron came away from Home Park empty-handed after a late penalty from Alan Judge secured only Argyle's second victory of the season.
Hayes, who has scored three goals this term, told the Scunthorpe Telegraph: "You can make all the excuses in the world but we went out there to do a job and we didn't do that on Saturday.
"With us talking about staying clear of relegation, you have to go to places like this and win.
"Only at the start of May, when you know how many points you have got, will you know if it is three points dropped, but we should have won the game on Saturday.
"You have to put pressure on yourself to come to these places and win. If you don't, you're playing catch up and you have to beat the bigger sides and try and get a big result, which will be hard."