Altidore was due to take up a place on the bench for Saturday's game, but was axed from the squad after arriving late at the KC Stadium.
The 19-year-old striker then revealed the reason for his omission on his Twitter page, much to Brown's disappointment.
Brown called the United States international's behaviour 'unacceptable' and acknowledged Altidore would be hit in the pocket.
He said: "It's going to cost him a lot of money, unfortunately.
"That for me is information that stays in house. The reason he wasn't on the bench was our business.
"Jozy was on the bench until 2.10 when I decided to change it.
"It's unacceptable behaviour, full stop.
"You prepare all week for a match day, mentally and physically. To ask a player to arrive an hour and a half before a game is not too much to ask is it?"
Altidore is currently on a season-long loan at Hull from Villarreal, although he has yet to score in six Premier League appearances, five of which have come as a substitute.