The 36-year-old threw 84 pitches - 52 for strikes - to record 11 outs with five strikeouts.
The first inning was particularly perilous, with three runs scoring. Tommy Pham kicked off the frame with a solo homer before Travis d'Arnaud collected an RBI single and Joey Wendle tacked on another run with a double.
Verlander last pitched during the opening game of the series, striking out eight over seven shutout frames.
Tuesday's outing marked the first time in his career that he's started a game on short rest. Verlander rested for only three days between Friday's Game 1 and Game 4.
The Rays wound up holding on for the 4-1 win, forcing a Game 5 in Houston on Thursday.