But the Midlands club dominated the second period and further goals from acting skipper Ashley Young and Fabian Delph ensured their passage into the last 16 before Nicky Forster's late reply.
Villa boss Martin O'Neill made six changes from the side which had overcome Blackburn in the Carling Cup semi-final second leg in midweek.
James Milner was again an effective performer in midfield after making an early appearance as substitute while Delph showed glimpses of his undoubted potential in the centre of the park.
But Brighton can be proud of their efforts and they were in contention until Young's decisive second goal early in the second period.
Brighton, whose manager Gus Poyet had played for Chelsea against Villa in the 2000 FA Cup final, were roared on by more than 6,000 fans in the near 40,000 crowd but their side were immediately on the defensive.
Ashley Young tested Michel Kuipers inside the first 30 seconds with the Seagulls keeper unable to hold onto his dipping drive but Adam Virgo was on hand to clear.
It came as no surprise when after six minutes Villa broke the deadlock through rookie striker Delfouneso.
Another Villa youngster in Marc Albrighton was the creator with a powerful run to the by-line before cutting the ball back across goal.
Delfouneso, who had netted against Blackburn in the previous round, was the first to react and his near post flick sent the ball into the corner of the net.
England Under-19 striker Delfouneso caused further problems for the visitors after breaking into the box from the left and his low cross was only just too strong for Ashley Young to connect with.
O'Neill was forced to make a change after 13 minutes with Albrighton limping out of the action to be replaced by Milner.
Albion midfielder Alan Navarro became the first player to be show a yellow card after 16 minutes for a foul on Delph.
Brighton started to come more into the game and Villa keeper Brad Guzan had a clearance blocked by Forster but the ball ricocheted to safety.
Brighton squandered a good chance to draw level after 37 minutes when Villa only half cleared a free-kick from Navarro.
The ball eventually broke to Glenn Murray only 12 yards out but he could not keep his shot down with only Guzan to beat.
Then after 40 minutes the Brighton fans erupted when Elphick equalised.
Villa failed to deal with a second corner in quick succession from Elliott Bennett and the ball broke at the near post to Elphick who squeezed his shot inside the post.
Villa defender James Collins was booked a minute before half-time for a foul on Bennett.
Villa needed only two minutes of the second half to regain the lead through a rare headed goal from Young.
Milner made a probing 50-yard break before releasing Downing in space on the left and he sent over an inviting cross for Young whose header beat the dive of Kuipers via the angle of bar and post.
O'Neill made a second substitute shortly afterwards with centre-back Curtis Davies replacing Warnock in his first action for nearly five months since undergoing shoulder surgery.
Brighton were indebted to a fine reflex save from Kuipers for keeping them in the tie.
The influential Milner cut to the by-line and his low cross picked out Delph who looked certain to score but Kuipers somehow blocked his close range effort.
Villa were not to be denied and after 61 minutes Delph scored his first goal for the club.
Milner's through ball sent Delph racing clear of the Brighton defence and he beat the dive of Kuipers with an angled shot into the corner of the net.
Brighton refused to buckle and Guzan tipped over a shot from a narrow angle by Carole while Murray drilled a low drive into the side netting.
But Villa finished on the offensive with Kuipers denying Steve Sidwell from 10 yards out.
The woodwork denied Delfouneso a second goal with five minutes remaining when his 20 yard curler rattled against a post.
But Brighton had the final say in the 90th minute with Forster's shot on the turn from 12 yards.