Tom Hitchcock fired the Millers ahead with a contentious strike after only three minutes, but Ollie Palmer equalised with a low finish before the quarter-hour mark.
Both teams had opportunities to snatch all three points and the Stags did just that in the 79th minute when trialist Craig Lynch broke away and fed Ben Hutchinson, who fired home from 20 yards.
We now advance to the Semi-Finals along with Group A winners Leicester City. Morecambe, Wigan Athletic and Carlisle United are all in contention to win Group C.
On a bitingly cold afternoon in South Yorkshire, we made the worst possible start as the hosts registered an early lead in controversial circumstances.
Nicky Adams threaded a low ball though towards an advancing Hitchcock, who despite being in an offside position – yards in front of any player, slotted past Ian Deakin.
It took our side less than 10 minutes to draw level though, as Jamie McGuire’s defence-splitting pass found an accelerating Palmer, who beat Scott Shearer at a tight angle.
Shearer kept out attempts from Lindon Meikle, Hutchinson and Palmer, but Rotherham spurned several gilt-edge openings in the moments leading up to half-time.
Deakin had already saved two shots from Reece Lucas and a free-kick from Michael O’Connor, before he showed exceptional reflexes to keep us on level terms.
Haris Vuckic latched onto a long downfield ball and was denied by Deakin, who got up quickly and made an instinctive reaction save to Nicky Walker’s follow-up attempt.
Stags’ full-back Liam Marsden then made the first of three goal-line clearances in a two-minute spell, as the Millers – much to their frustration – were unable to find a way through.
After the re-start, Vuckic wastefully blazed over from eight yards following an inviting cross by Richard Brindley, whilst Deakin also made more smart saves.
At the other end, Meikle let fly with an audacious 20-yard attempt on the hour mark, which had to be turned round the post by a full-stretch Shearer.
We took the lead with just 11 minutes remaining when trialist Craig Lynch broke quickly down the centre and played a simple ball to Hutchinson, whose low shot found the bottom corner.
Rotherham almost snatched a share of the spoils on the brink of stoppage-time, when Vuckic latched onto a through ball and rattled the upright, but we held on for an important win.