The 25-year-old has agreed a two-year-deal at the Withdean Stadium.
Crofts, who has 12 caps for Wales, began his career at Priestfield but fell out of favour with manager Mark Stimson last season, and after being stripped of the captaincy was loaned out to Peterborough and Wrexham.
Seagulls boss Russell Slade told the club's official website: "I've been aware of Andrew for some time.
"He's an international player so that's not bad for a start. He's got good pedigree and was an important player for a good Gillingham team at the time.
"He can sit in for you defensively or he can get forward. He has a good work ethic and I'm pleased to get him."