The 36-year-old striker, plucked from non-League obscurity in the summer, rose above the Millers defence to plant a header into the bottom corner to equalise.
It was a painful afternoon for Rotherham caretaker manager Steve Thornber who saw his side utterly dominate the first half but fail to kill off Town after the interval.
Alsop's towering header, from a cross by teenage substitute Theo Lewis, came within moments of Rotherham's Nicky Law missing a glaring opportunity at the other end with Danny Harrison having rattled the post for the visitors five minutes earlier.
The Millers had dominated from the off with Tom Pope, three times, Dale Tonge and Ellison all going close before the visitors took the lead.
Adam Le Fondre finally broke the deadlock from close range after Cheltenham goalkeeper Scott Brown spilled a corner in first-half injury-time.