Spanish duo Marcel Granollers and Feliciano Lopez needed to win their doubles decider to send Spain through at Serbia's expense, but Aslan Karatsev and Andrey Rublev came from a set down seal a 4-6 6-2 6-4 victory and top spot for Russia in Group A.
Sweden and Novak Djokovic's Serbia took the two best runners-up spots and a place in the last eight by virtue of their better match win-loss records among the second-placed sides throughout the group stages.
Spain needed to win the tie against Russia to progress to the last eight, with the 40-year-old Lopez defying his years to storm back from a set down and get the better of world No 5 Rublev, getting the defending champions off to the perfect start in Madrid.
Rublev had just picked up his 50th win of the year the previous day, with his power-hitting seeing him take the first set before Lopez turned things around to win the match 2-6 6-3 6-4.
US Open champion Daniil Medvedev settled any Russian nerves in the second singles rubber against Pablo Carreno Busta after making a blistering start to the first set, racing into a 4-0 lead in under 11 minutes, before taking it 6-2.
The world No 2 made hard work of the second, eventually coming through in a tiebreak to win the match and take the tie to a doubles decider, with Russia dramatically edging a tight tussle to end Spain's hopes of retaining their title.
"We were very close but it was not enough," said Lopez.
"It is what it is and I don't want to give excuses. We showed everyone that we are ready to compete in any condition, even with so many issues that we went through. It's so disappointing."
Germany secured top spot in Group F and a quarterfinal against Britain after battling to a 2-1 success over Austria, with Kevin Krawietz and Tim Puetz sealing their progress with a straight-sets victory in the final doubles decider.
Colombia's 2-1 victory over the United States in Turin mattered little as they finished second behind Italy in Group E, unable to grab one of the best runners-up spots, as 32-time champions, the US slipped out with a whimper.
Croatia, Kazakhstan and Britain progressed to the quarterfinals earlier on Sunday after the trio topped their respective groups, while 28-time champions Australia were knocked out.
Britain got off to a poor start against the Czech Republic as they looked to secure a top spot in Group C.
Dan Evans lost his singles rubber against Tomas Machac, before British No 1 Cameron Norrie managed to level and doubles pair Joe Salisbury and Neal Skupski won in straight sets to take the tie 2-1 and secure a top spot ahead of France.
The first quarterfinal gets underway on Monday, as Italy host Croatia in Turin, before Britain face Germany in Innsbruck, Austria the following day.