Paramedics told media the man fired a revolver and a .38-caliber pistol inside the cathedral at 1 pm as mass was going on, also wounding several people before killing himself.
"A man opened fire randomly on people inside until police intervened and shot at the gunman who then killed himself," the Security Secretariat for the state of Sao Paulo said in a statement. It said, "as well as (the gunman), four people died and four people were wounded."
That corrected an earlier police toll that had given a total of six dead including the gunman and three wounded.
The secretariat said footage from the cathedral's security cameras had been obtained and would be analyzed.
Television images showed the lifeless body of the gunman, wearing a blue t-shirt, lying inside the cathedral, holding a discharged pistol in his right hand, and a revolver tucked into his jeans.
Paramedics were also seen treating several people outside the cathedral and there were multiple police cars in the street.
"Everybody ran out. You can imagine that I'm shocked," one woman who was attending the mass told Globo TV.
The Archdiocese of Sao Paulo said on its Facebook page that the killings provoked "deep pain" and added that the cathedral would be closed for as long as required for the police investigation.
A police officer told Globo TV that most of those in the cathedral at the time were elderly.
Brazil is one of the most violent countries in the world. Last year it registered a record 63,880 murders. Its homicide rate of 30.8 per 100,000 inhabitants is three times higher than the level the UN considers to meet the definition of "endemic violence."
Seventy percent of murders in Brazil are committed with a firearm, according to a monitoring group, Forum for Public Security.
The country's incoming president, Jair Bolsonaro, who takes office on January 1, easily won November elections on a platform that included promises to ease gun laws to allow "good" people to defend themselves.