Having taken the Premier League by storm in the first half of last season the Tigers have struggled somewhat since.
After leading the club into the top flight for the first time in 104 years Brown has seen recent performances heckled by some sections of the Hull support.
A 4-0 Carling Cup defeat at home to Everton in midweek will not have helped matters but a demoralising 6-1 loss to Liverpool on Saturday leave the team second from bottom of the Premier League.
"There is no speculation at the football club about the manager's position," Duffen told The Times.
"That's just mischief making in the media if there is. We are certainly not aware of it.
Vociferous minority"In football you are always going to find a vociferous minority who will be discontented and I also think second-season syndrome articulates a little bit about fans and a club who got into the Premier League for the first time in 104 years.
"In the second season maybe they have set themselves unrealistic expectations. At the moment we are seven games into a new campaign.
"We are right in the mix, only a few points separates us from the middle of the league and I am really pleased with the signs I see developing in the squad.
"As you could see, we certainly don't lack spirit or heart. Everyone is playing for each other and they are certainly playing for the manager."
Duffen even went as far as to say that the Tigers' first-half showing at Anfield was a good performance with the visitors going in 2-1 down.
"I thought it was a fantastic performance for the first 45 minutes but obviously we went down to a 46th minute goal that made things very difficult," he added.
"I thought for 45 minutes we played really good football and there were good signs of things to come."