The 20-year-old's comments are surprising given that the Swede enjoyed a decent run in the first team under Ince at the start of the season.
The youngster has played only twice under Allardyce but has seen enough to suggest that he will lead the club to Premier League safety this season.
After an encouraging start under the former Bolton and Newcastle man, Rovers have not won in the league for four games and find themselves 18th, two points from safety but with a game in hand on their rivals.
"He (Ince) tried his best but when he was here we were low on confidence," Olsson explained.
"Since Sam came in we have been working hard in training and we know we can get out of the bottom three, it will just take a bit of time.
"We work as a team, not individuals. There is more communication and that helps a lot on the pitch.
"Sam has been in the Premier League for a long time compared to Paul Ince. Ince needed to learn more because he was new to the game.
Learning"Sam knows what he is doing and that is why I'm not bothered if I'm not playing because I'll just learn what others are doing on the pitch.
"I think our confidence has gone much higher than when Ince was here."
Olsson is set to feature in the FA Cup fifth round replay against Coventry on Tuesday but as much as he wants to play at Wembley, he admits that Premier League survival is the priority.
"I've been out with injury and viruses but if I get my chance I'll do my best," he added.
"I saw a bit of the last game (against Coventry) and in the second half they were good.
"We have to be focused and start playing how we can and I think we will be fine. We are playing good football when we are playing together.
"We know we are good as a team and we know we can beat Coventry even if we don't have the strongest squad.
"If he (Allardyce) plays the young ones or the ones who don't play every week, then the others can think about the Premier League games.
"It is important game because we want to get to Wembley but the Premier League is more important and we have to focus on that."