The Californian, who opted for a move up the European single seater ladder as opposed to moving up IndyCar's 'Mazda Road to Indy', had spent the closing stages of the race battling inside the top 10 as his rivals around him pulled into the pits one-by-one as their fuel failed to last the long green flag period that closed the race.
Rossi managed to go the distance however, with his car spluttering to a halt just after he crossed the finish line to win his first IndyCar series race.
"I have no idea how we pulled that off," said Rossi in voctory lane. "We struggled a little bit in the pit stops but Bryan (Herta) came up with an unbelievable strategy. I can't believe we've done this!"
"I don't even know where to begin," he added. "In February I wasn't even thinking about IndyCar, and now we've just won the Indy 500. Thanks to an amazing group of people who gave me an opportunity to come here this year."
The former Formula One driver's Andretti Austosport team mate Carlos Munoz followed him home in second place. The pair were two of four Andretti drivers to have taken a turn at leading the race with 2014 winner Ryan Hunter-Reay and Townsend Bell also spending time at the front of the field before a pit lane collision took both out of contention.
"Something out of our control happened. They said, 'Go, go, go!' It looked like Townsend (Bell) got into Helio (Castroneves) and bounced into me," said Hunter-Reay. "The car was so strong. The only time we ever spent any time (slower) was because I was saving some fuel. Other than that, it was a rocket ship. Such a shame when you have a car like that. The car was great. We could have won this thing today."
"He (Hunter-Reay) was pulling out and I'm not going to stop. What a shame," added Bell. "We had a great race car and we were saving fuel early on. The pit stops seemed pretty good. All of a sudden I'm leading, I think, when I come in. We pit, I get out before Ryan and he starts going. So I'm going around him outside and I didn't know a car was coming down. I guess the three of us were trying to occupy space for two cars. It took me and Ryan out. I'll look at it but I don't know what I could have done differently."
Josef Newgarden and Tony Kanaan who were both also in contention for the win late in the race finished in third and fourth respectively, with Kanaan's Chip Ganassi Racing team mate completing the top five.
2011 Indy 500 runner-up JR Hildebrand, making just his second start of the year, finished the race in sixth while pole sitter James Hinchcliffe came home seventh. Scott Dixon, Sebastien Bourdais, and Will Power rounded out the top 10.
Defending Indy 500 winner Juan Pablo Montoya was the first driver to retire from the race, losing the rear of his car and sliding into the wall at turn two on lap 63.
"The car was just hard to drive and it was a little bit tighter there in that corner and it got away," said Montoya. "It was a disappointing day. People in race cars do a lot of dumb things and we were being careful. So I started making up some ground again and the car was actually pretty good. It was really sloppy, but it felt okay. I went into (Turn) 2 with a big push and when I got on the gas, it just came around. The hit is no big deal and not that bad actually. When the clouds came over the track, my car came to life and that's what we were hoping for. I don't know. It's just disappointing."
Sage Karam, Ed Carpenter, Buddy Lazier, Conor Daly, Stefan Wilson, Mikhail Aleshin, and Takuma Sato were the race's other retirements.
Rossi's win the the #98 Andretti Herta Autosport entry marked a repeat of the 2011 Indianapolis 500 - the 100th anniversary event - when Dan Wheldon won in the Bryan Herta-run #98 Honda.
"This is unbelievable," said Herta, who teamed up with Andertti Autosport at the start of the year to run Rossi.
"Man, it was so close at the end. For a rookie to drive with the poise he did in such a tough situation - I was telling him, 'Don't let anybody pass you but save fuel' - and he did it."
Images: IndyCar Media