The "Cafeteros" failed to qualify for the global extravaganza for the first time since 2010 following a disastrous campaign in which they set a national record of seven straight matches without scoring a goal.
It was the second time Rueda, a Colombian, had failed to guide his country to a World Cup having also missed out in 2006 during a previous two-year stint in charge.
The Colombian Football Federation said in a statement that it had "agreed with Reinaldo Rueda the termination of his position as technical director ... as well as his coaching team."
Despite boasting a plethora of attacking talents such as Liverpool's Luis Diaz, Duvan Zapata of Atalanta, Rayo Vallecano's Radamel Falcao and James Rodriguez, who plays his club football in Qatar where the World Cup will be held, Colombia's qualification campaign floundered due to a lack of goals.
They finished sixth in the single 10-team South American group and missed out on even an intercontinental playoff.
Rueda, 65, took over in January 2021 after a disastrous start to the qualifying campaign under Portuguese Carlos Queiroz saw Colombia lose 3-0 at home to Uruguay and 6-1 away to Ecuador.
But after Rueda won his opening match 3-0 away to Peru and also guided Colombia to a third-place finish at the 2021 Copa America, the goals dried up and hopes of reaching Qatar disintegrated.
In 22 matches in charge, he won seven, drew 10 and lost five.
The Colombian federation said it was "evaluating the existing options" before naming a replacement.
Rueda previously coached Honduras (2010) and Ecuador (2014) at the World Cup and was in charge of Chile for three years before returning to the Colombia post.