Taxiarchis Fountas capitalised on Leno's misfortune to put Rapid ahead in the 51st minute before David Luiz headed Arsenal level and Pierre-Emerick Aubameyang, who started on the bench, gave them three points in the Group B match.
"We knew it was going to be a difficult match. We made it harder in the beginning with a few decisions that we made and obviously the goal we conceded, which if anything is obviously my fault because I demand we do those things," Arsenal coach Mikel Arteta said.
Arsenal gave a first start to new signing Thomas Partey whose €50 million fee is more than the transfer market value of Rapid's entire squad, which according to the specialist website Transfermarkt is around €34m.
"He looked really solid, really comfortable. He held midfield by himself in the second half when we were a little bit more open and starting to take more chances attacking certain places with some players," Arteta said of Partey's performance.
Rapid, backed by around 3,000 fans at the Weststadion, had the better of the first half and deservedly went ahead six minutes after the restart.
Leno received the ball from David Luiz but was dispossessed by Ercan Kara and Fountas turned in the loose ball.
David Luiz levelled with a simple goal in the 70th minute, outjumping the Rapid defence to head in from Nicolas Pepe's floated free kick.
Fountas just failed to capitalise on another Leno mistake, shooting wide from 30 metres after the German charged out of his area and fired the ball against David Luiz, before Arsenal won the game in the 74th minute.