The Tigers surprised many at the start of the season when they rose to third in the Premier League but an awful run of form has seen them drop to fourth from bottom.
In March, Brown alleged Cesc Fabregas had spat at the Tigers assistant boss Brian Horton following the teams' FA Cup tie at the Emirates Stadium.
The Arsenal midfielder was subsequently charged with improper conduct while Brown has also been charged by the FA over his criticism of the referee, Mike Riley.
Brown admits it may have altered people's impression of the club.
"Possibly it did," he told The Times. "I can't go overboard about that because it's part of an inquiry and I'm under FA scrutiny, but there was a lot (that) went on after the game.
"I would have dearly loved to speak to Arsene Wenger straight after the game.
"I would have put it to him manager to manager, instead of going public, but there was no access to him.
"If there was access, none of this would have come out. But I couldn't speak to him or get any sense out of anyone else.
"Let's conduct the inquiry and then we'll see what comes out.
"I think it's something we need to look at. For as long as I've been involved in football, there has been a culture where the managers have a chat beforehand and afterwards and it doesn't spill over into personal animosity.
"We talked about a Respect campaign for the players at the start of the season. I think the League Managers' Association have a duty to make sure we all get together and say that this tension just can't go on.
"I know there's long-term battles going on with some of the big boys, but it doesn't have to be that way. We should be able to talk to each other, manager to manager."