Awoniyi scored Forest's first Premier League goal in 23 years in first-half stoppage time and it proved to be the difference as both teams had goals disallowed by VAR and West Ham captain Declan Rice had a second-half penalty saved.
Spurred on by a boisterous home crowd, Forest started on the front foot and kept West Ham at bay in a frantic first half before Awoniyi, making his home debut, bundled home Jesse Lingard's miscued shot from Harry Toffolo's cross.
Moments earlier, Said Benrahma had slotted past on-loan Forest goalkeeper Dean Henderson following a West Ham counter-attack but the goal was chalked off after Michail Antonio was adjudged to have fouled Orel Mangala in the build-up.
West Ham midfielder Pablo Fornals struck the crossbar seconds after the restart and the rebound fell to Tomas Soucek, whose header was clawed away by Henderson as the visitors made an electric start to the second half.
Their momentum was soon halted, however, when Forest forward Brennan Johnson - their top scorer last season with 19 goals in all competitions - beat goalkeeper Lukasz Fabianski with a dinked finish in the 54th minute before being ruled offside.
On the hour mark, West Ham winger Benrahma hit the woodwork from distance before Soucek's shot was blocked by the slightly extended arm of Forest defender Scott McKenna, resulting in a penalty for the visitors after a VAR replay.
Rice stepped up to take the spot kick but his effort dragged low to his left, was saved superbly by Henderson to preserve the hosts' slender lead.
In the closing stages, Forest full-back Neco Williams cleared Kurt Zouma's header off the line as Steve Cooper's side saw out seven minutes of stoppage time to pick up their first points of the campaign.
West Ham have now lost two consecutive games while Nottingham earned their first 3 points.