The CDU general secretary narrowly beat Friedrich Merz, a millionaire lawyer, in a run-off vote in Hamburg.
Ms Kramp-Karrenbauer, or AKK as she is also known, received 517 of 999 votes, and as the head of Germany's largest party, she could now become the next German chancellor.
Speaking after her victory, Kramp-Karrenbauer said, "I learned what it is to lead, and above all learnt that leadership is more about being strong on the inside than being loud on the outside."
She called for party unity in her first address as leader and brought Mr Merz and her other opponent, Health Minister Jens Spahn, to the stage, saying she would be "very happy" if they joined her in this task.
The 56-year-old was Merkel's choice to succeed her with the former Chancellor praising AKK for her 2017 electoral success in Saarland.
Kramp-Karrenbauer became the first woman to serve as a state minister for internal affairs in 2000 and the first woman to serve as prime minister of Saarland, a position she held from 2011 to 2018.