Kostic drilled a low shot past Manuel Neuer seven minutes from time to hand Frankfurt their first win in Munich for more than two decades and condemn Bayern to their first defeat of the season.
"It always hurts to lose, but this defeat hurts a bit more because it was avoidable," Nagelsmann told the press.
"We were punished today for not taking our chances."
Despite dominating Frankfurt for much of the game, Bayern succumbed to their first league home defeat since 2019 and now sit just one point clear of Borussia Dortmund in the table.
Kostic's winner also gave Frankfurt their first league win of the season, lifting the pressure on new coach Oliver Glasner.
"It was the first time I have won in Munich as a coach, and you don't forget your first time," joked a beaming Glasner.
"We've been chasing this win for a while now and have had plenty of chances. The fact that we've got it against Bayern makes it even sweeter," added Frankfurt captain Hinteregger.
Leon Goretzka gave Bayern a deserved lead after a dominant first half-hour, slotting in a low finish after a smart lay-off from Robert Lewandowski.
But the hosts were left reeling a few minutes later, as Martin Hinteregger sent a thundering header past Neuer at a corner to level the scores.
Lewandowski almost restored the lead shortly before the hour-mark, but Frankfurt keeper Kevin Trapp denied the Polish star at point-blank range with a brilliant reflex save.
It was the first of a string of second-half stops from Trapp, whose heroics kept Frankfurt in the game before Kostic's winner sealed Bayern's fate.
Elsewhere, Nigerian striker Taiwo Awoniyi netted two goals against his old club as Union Berlin fought from a goal down to beat Mainz 2-1 and continue their fine start to the Bundesliga season.
Awoniyi struck twice in four minutes in the second half to give high-flying Union their first away win of the season and fire them up to seventh in the table.
Union loanee Marcus Ingvartsen also scored against his former teammates to give Mainz the lead just before halftime, but it was former Mainz striker Awoniyi who had the last laugh as Union bounced back after the break.
Having missed a handful of chances throughout the game, the Nigerian levelled the scores with a cool finish on the counter-attack 20 minutes from time.
A few minutes later, the 24-year-old scored an almost carbon copy of his equaliser to put Union ahead, once again latching onto a long ball and holding off the Mainz backline before slotting the ball into the bottom corner.