The 19-year-old American arrived at the KC Stadium at 2.10pm on Saturday, 40 minutes later than Brown demands his players show up for games. He was dropped from the squad, having initially been set for a place on the substitutes' bench, and Hull took internal disciplinary action.
Brown told the Hull Daily Mail: "He'll be back in contention for the weekend but it's up to him now to prove that he's deserving of a place."
He added: "He's 19, he's at the beginning of his career but Jozy is being paid an awful lot of money to play professional football and it's part and parcel of his working life to be professional."
Brown was unimpressed by Altidore's apology, posted on Twitter shortly after his arrival at the stadium, but it was the time-keeping which riled him most.
Brown said: "You can cut people some slack in given situations, but not on a matchday.
"It's what you prepare for all week and to come in an hour and a half before a game shouldn't be too much to ask. To be 35 or 40 minutes late, it just beggars belief."