Lord Eric Pickles turned a blind eye while giving evidence at the inquest into the disaster.
He was secretary of state at the Department of Community and Local Government from 2010 to 2015.
The former Conservative Party chairman appeared ahead of the investigation into the west London tower fire that killed 72 people in 2017.
However, the former MP for Brentwood and Ongar said 96 people had died in the fires.
Grenfell United, the group of survivors and bereaved, said: “Eric Pickles’ disrespect in the investigation has left us speechless. How dare he call our loved ones we lost that night ‘anonymous 96’. Seventy-two people have died in Grenfell and none of them are unknown.
“His complete disregard for what has happened and who is no longer with us is appalling, given his ability as Housing Minister to reform. building safety.
“Eric Pickles must be removed from Government and Lords.”
Lord Pickles made a mistake at the end of his proof while answering if anything he would have done differently.
He told the inquest: “My answer, which I obviously prepared because I watched you do this and I realized I was going to be asked – was completely different from the answer. which I will give.”
He consulted the coroner’s recommendations after the 2006 Lakanal House fire in Camberwell, which killed six people and injured 20 others.
The first report from the Grenfell Inquiry showed that lessons from the Lakanal House fire had not been learned by the time of the Grenfell disaster eight years later.
Lord Pickles said: “What I was going to say was maybe I should put in the letter a simple ‘and I accept the coroner’s offer’ – does that change things?”
“And your diligence, and your choice of examples, made it clear to me, I don’t think it’s going to make any difference from what I thought it would.
“There’s a type of mindset that exists in parts of the department that simply ignores what’s going on, makes a point of what we are, and does it.”
He said the investigation should “never lose sight” that “this is not about deregulation”.
Instead, he said it was about the “unnamed” victims of the Grenfell Tower fire.
He added: “It concerns Michelle (Udoaka, who died in the Lakanal fire) and the 96 unknowns who perished in the Grenfell fire. They are the people we should think of when we argue about pitching.
“In the end, as I said earlier, the dead deserve a name and the dead deserve the dignity of a resolution. And I’m sure you’ll get there.”
Seventy-two victims of the Grenfell Tower fire have been named and identified.
Earlier, on Thursday, the second and final day of evidence, Lord Pickles also expressed frustration with the amount of time the investigation had taken up.
He told lead counsel for the investigation Richard Millett QC: “By all means, sir, feel free to ask me as many questions as you like, but I can respectfully remind you that you promised that we will be away this morning and I have changed my schedule to accommodate this. I have an extremely busy day.
“But this is more important than anything, but I advise you to use your time wisely.”
Mr. Millett replied: “Yes… May I have an answer to my question, please?”
However, at the end of his evidence in the afternoon, Lord Pickles thanked the inquest for the “professional and polite” treatment he received.
In return, Grenfell’s chairman of investigations, Sir Martin Moore-Bick, thanked him for his time.
He added: “I’m sorry that got in the way of your arrangements for today, but there are things we need to ask you.”
Standard has approached Lord Pickles for comment.