Blackburn have picked up just five points so far this season and take on Tottenham , who currently sit in sixth, this Sunday.
The Rovers stopper has admitted their lowly position is difficult to take but believes they can recover and begin to climb the table.
"You can't ignore the league table because you have to be realistic and know that you are in a bad position," he told Lancashire Telegraph.
"We know it is important to pick up points but if we play like we did at QPR then we can pick these points up. We know that."
It has been a turbulent start to the season for the Lancashire club who have finished as high as 10th in recent campaigns.
"That is the reality of football though and we have to think positive and work hard together as a team. That is the way we will get out of it," Olsson said.
"We are all hurting. It is not that we want to be down there.
"We are trying our best in training every day. We think we have been unlucky but hopefully we can start picking up points."
Olsson had mixed feelings about Saturday's draw against QPR as Rovers squandered a couple of late chances to come away with three points.
"We played some good football at times and with the chances we created we could have got the three points," he said.