The German took charge of United on a caretaker basis for the last six months of the recently concluded 2021/22 season after Ole Gunnar Solskjaer was sacked in November.
Rangnick led the club to a sixth-placed finish in England's top-flight with 58 points, their lowest total in the Premier League era, as they just did enough to qualify for the second-tier Europa League.
The 63-year-old had been expected to stay at Old Trafford for the next two years in a consultancy post, but last month he was appointed as the new manager of Austria and it appears holding the two jobs at once is no longer a viable option.
"We would like to thank Ralf Rangnick for his efforts as interim manager over the past six months," said a United statement.
"By mutual agreement, Ralf will now focus solely on his new role as manager of the Austria national team and will not, therefore, be taking up a consultancy role at Old Trafford.
"We would like to wish Ralf the best of luck in this next chapter of his career."
Only nine days ago, Rangnick told the media he had been in contact with new United manager Erik ten Hag about "all the different areas where I could maybe of help".
But Ten Hag said Rangnick's consultancy post "is on the club".
United lost in the fourth round of the FA Cup to Middlesbrough and was knocked out of the Champions League in the last 16 by Atletico Madrid.
Under Rangnick, they also failed to qualify for next season's edition of the Champions League, European football's premier club competition.