Riley made 6 of her first 7 shots, then missed 17 in a row before warming up again late in the game to carry the seventh-seeded Cowboys into a Saturday semifinal against Baylor or Oklahoma.
Riley also became the Big 12's career leading scorer and set the conference's single-season scoring record.
Anna Prins had 15 points, Kelsey Bolte 12 and Whitney Williams had 11 for Iowa State.
Riley, who scored 37 points against Kansas on Thursday, came into the game tied with Oklahoma's Courtney Paris as the league's career scoring leader. Riley now has 2,772 points.
Riley has 846 points for the season, breaking the conference mark she set two years ago.
Riley, who finished 16 of 44 from the field, hit jumpers to tie it at 50-all and to put the Cowgirls (23-9) up 54-50, and she drove to the basket to make it 58-55 with 45 seconds left.
She hit four free throws in the last 13 seconds to hold off the second-seeded Cyclones (23-7).
The loss ended the Cyclones' streak of making it past their first Big 12 tournament game every year since 2003.
Riley, who scored a tournament-record 37 points against Kansas on Thursday, hit three 3s to help Oklahoma State get out to a 19-7 lead.
Prins, harassed by Toni Young and Tegan Cunningham, fumbled away three entry passes and had another pass go through her hands as the Cyclones committed 11 of their 18 turnovers in the first half and fell behind by 15 points less than 10 minutes into the game.
Yet, in spite of shooting just 28 percent, they were down 24-22 at half. That's largely because the Cowgirls shot worse - 24 percent.
Riley accounted for 18 of the Cowgirls' first 24 points, but she went cold, missing 16 of 17 shots as the Cyclones closed the half on a 13-2 run. Over the first and second halves, she was off the mark with 17 in a row during one stretch, with ISU forcing her to settle for pull-up jumpers in the lane rather than letting her go to the basket.
When the 6-foot-7 Prins knocked down a 3-pointer at the top of the key, Iowa State led for the first time at 28-27.
The teams went back and forth from there, with Iowa State taking its last lead at 50-48 with 5:39 to play.
Iowa State's All-Big 12 point guard, Alison Lacey, didn't play because of her continuing battle with a bronchial illness. She missed last week's 78-70 loss at Oklahoma State and was on the floor for just two seconds last weekend against Colorado so she could have a ceremonial start on Senior Day.