Duffen quit as Tigers chief earlier on Thursday, less than 24 hours after the club were forced to issue a statement denying Brown had been sacked.
The departing chairman claimed the club's on-field problems were the reason for his exit, with Hull only narrowly avoiding relegation last term and struggling in the bottom three this season.
But Brown, who alongside Duffen helped guide the club into the top flight for the first time in their history, believes off-field matters were the main contributing factor.
Concerns have been raised over the club's financial stability and there are further reports that owner Russell Bartlett is planning to bring in former chairman Adam Pearson, who quit his post at Derby on Wednesday, to replace Duffen.
SupportSpeaking at a press conference screened live on Sky Sports News, Brown attempted to steer the subject onto Saturday's trip to Burnley and said: "You have been live at the training ground and it was no surprise that you've seen the response.
"I think the players are one million per cent behind me and hopefully that will show in the performance on Saturday.
"As you've seen on the training ground there are some bright lively players looking to put their case forward to play on Saturday.
"The support of Paul Duffen was there for everybody to see, that's gone now. What is around the corner you can only guess, and only time will tell who his replacement is.
"I don't think it was on football results, I think it was on the business of football. It's difficult to separate the two, but you do tend to run a business as well as a football club.
"I run the football side of things and I remain in charge for the game on Saturday.
"I've spoke long and hard with Paul Duffen and Russell Bartlett, the owner, in the last couple of days. He is in extensive talks with a replacement and something will be announced after the game on Saturday.
"It doesn't affect me in terms of my position, as has been made reference to by Russell Bartlett in conversations I've had with him over the last 24 hours."
Ridiculous speculationWhen questioned further on his own future, an increasingly frustrated Brown repeatedly responded: "I'm here to talk to you about the game against Burnley."
The Tigers boss started the press conference by reading out a prepared statement paying tribute to Duffen, who has always staunchly supported the manager despite increasing calls for his exit.
Brown said: "We have had tremendous, unprecedented success with Paul over the last two-and-a-half years and I would personally like to thank him for all his support.
"But unfortunately the game has and always will be about change. We have to adapt to it and we have to respond to it and like in the game of football, the quicker you do, the better for your team.
"We have a big game on Saturday to prepare for and that is my sole purpose at this moment in time.
"I cannot and will not respond to ridiculous speculation surrounding the business of this football club, whether it is reference to the accounts or replacement of Paul.
"I am here specifically to talk to you about the game on Saturday, but I will finish this statement by wishing Paul Duffen all the very best for the future from myself and all of my backroom staff."