Trump, speaking to reporters as he left the White House to attend the annual football game between the Army and Navy military academies in Philadelphia, said he would name a replacement for Kelly, possibly on an interim basis, in the next day or two.
"He's a great guy," Trump said of Kelly, who led the Department of Homeland Security before joining the White House. "I appreciate his service very much."
A White House official later told Reuters that Trump has been in discussions for months with Nick Ayers, an aide to Vice President Mike Pence, about becoming White House Chief of Staff.
Trump wants Ayers to do the job for two years, the official said. But Ayers is unable to make the commitment because of his family, including two young children, and has agreed to serve only until the spring of 2019.
A freewheeling former businessman, Trump brought Kelly in last year to restore order to his often chaotic White House, but has clashed repeatedly with the retired Marine Corps general in recent months.
Kelly, 68, had some success in restoring order to Trump's White House after he was appointed in July 2017, but the president bristled at his style.
Their rocky relationship was highlighted in "Fear: Trump in the White House," a book by Washington Post reporter Bob Woodward that was released in September.
In it, Kelly was shown as one of several White House figures questioning Trump's abilities and was quoted as calling the president "unhinged" and "an idiot."
"It’s pointless to try to convince him of anything," the book quoted Kelly as saying during a meeting. "He’s gone off the rails. We’re in Crazytown. I don’t even know why any of us are here. This is the worst job I’ve ever had.”
As head of Homeland Security, Kelly backed Trump's ban on travel from certain predominantly Muslim nations but at times appeared at odds with the president's agenda.
In January, he told Fox News that Trump was not "fully informed" when he made his signature promise to build a wall on the U.S. border with Mexico to block illegal immigrants and drug smugglers.