It was the second stunner in Doha since Saudi Arabia came from one goal down to upset Argentina 2-1 in a similar drama on Tuesday.
Like Argentina's loss to the western Asian team, Germany led from a first-half penalty by Ilkay Gundogan, who went to the spot after Shuichi Gonda brought down David Raum in the box.
A mostly uninspired Japanese team woke up after Gonda made four brilliant saves in quick succession around the 70th minute, leveling the score through Ritsu Doan in the 75th minute and snatching the winner through Takuma Asano in the 83rd minute for its biggest World Cup victory to date.
The introduction of Tomiyasu and Minamino made a great impact for Japan, with German making a great mistake to substitute Muller and Gundogan. German Vs Japan Highlights�� pic.twitter.com/pUWqviOHJl
Doan, who plays for the German team Freiburg, pounced on a rebound and tapped it home after Germany goalkeeper Manuel Neuer parried a shot from Takumi Minamino.
Then Asano, who plays for German club Bochum, completed a turnaround in the first competitive meeting between the two nations, shaking off Nico Schlotterbeck and smashing the ball home from a narrow-angle.
Germany is now on the line, with its next rivals being Spain and Costa Rica, who will take on each other late Wednesday.