Rangers will face German side Eintracht Frankfurt in the final of the Europa League.
Leipzig lost just once in 19 games between January and April, but have now been beaten in three of their last four to put their chances of Champions League football next season hanging in the balance.
Tavernier moved out in front as the top scorer in the competition from right-back when he opened the scoring before Glen Kamara turned the tie around inside the first 24 minutes.
The Finnish midfielder dedicated his goal to long-serving kitman Jimmy Bell, who died on Tuesday, on an emotionally charged night at Ibrox.
Christopher Nkunku hit back for Leipzig in the second half, but John Lundstram scored the winner nine minutes from time.
Giovanni van Bronckhorst's men will face Eintracht Frankfurt in the final in Seville on May 18 in just their second European final in 50 years.
"We wanted to do it for Jimmy. I'm super proud of all the effort the players, staff, fans put in today. He will be looking down proud of us," said Tavernier.
"It's an unbelievable feeling, a European final. It's what you dream of.
"We'll go there full of confidence. Frankfurt got there for a reason, but it's one game and we'll fully back ourselves. We're in this competition to win it."
Rangers lost to Zenit St Petersburg in the UEFA Cup final of 2008, but have had to start again from the fourth tier of Scottish football following a financial collapse in 2012.
Last season they ended Celtic's run of nine league titles in a row and will return to the Champions League for the first time since 2010 with a victory over Frankfurt at the Ramon Sanchez Pizjuan in 13 days' time.
Leipzig's 1-0 advantage from the first leg was blown away early on in front of cacophonous noise.
Tavernier timed his run to perfection to turn in Ryan Kent's cross before Kamara guided a wonderful finish low into the far corner from outside the box and kissed his black armband in tribute to Bell.
Joe Aribo should have made it three within half an hour when he missed his kick from point-blank range.
"We knew what to expect, but experiencing this kind of atmosphere in person is something different," said Leipzig CEO Oliver Mintzlaff. "This is part of a recent negative trend and it's come at the worst time in the final stretch."
The Germans enjoyed the majority of the ball in the second half and were back on level terms in the tie when Nkunku steered home his 32nd goal of the season from Angelino's cross.
Rangers then had to dig deep just to keep Leipzig at bay, but they found the winner late on.
Kent's cross to the back post caused panic in the Leipzig defence and the loose ball broke to Lundstram to stroke into an unguarded net and spark wild scenes of celebration.
"I'm very proud of the achievement so far," said Van Bronckhorst. "Once you are in the final, we have to do everything to win it as well."