> And crucially, we made sure to tell the model not to guess if it wasn’t sure. (AI models are known to hallucinate, and we wanted to guard against that.)
Prompting an LLM not to confabulate won't actually prevent it from doing so. It's so disappointing to see an organization like this, that's mission is to inform the public, used AI not understanding the limitations and then making a claim like this.
> Of course, members of our staff reviewed and confirmed every detail before we published our story, and we called all the named people and agencies seeking comment, which remains a must-do even in the world of AI.
That sounds to me like they absolutely do understand the limitations of the technology they are using.
Criticism feels harsh. Of course models don't know what they don't know. Reporters can have the same biases. They could have worded it better "lowers the probability of hallucinating", but it is correct it helps to guard against it. It's just that it's not a binary thing.
ProPublica does outstanding investigative reporting, to include among many more examples their investigative reporting on the nation's stillbirth crisis [1], our broken healthcare insurance system [2], and how the rich and wealthy avoid paying taxes [3]. Your rattling about "left-leaning this, right-leaning that" comes across as out-of-touch and maybe even a tad unhinged.
That's a chemical manufacturing corporation lobbying group you're quoting on the risks of one of their products, not the American Chemical Society.
The American Chemical Council (previously named the Chemical Manufacturers' Association) have a section on their Wikipedia page accusing them of pretending to self regulate the industry by setting up green-washing programs that make progress slower than sites not affiliated with the program.
So without any further knowledge, I'd go with the journalists on this one.
I don't have clear context on this but I also want to say you could cherry pick PRs from my career to make me look however you want. I think especially for outlets with a wide variety of writers and editors and a huge backlog of stories people get caught up on specific mistakes.
Not to put too fine a point on it but even in your post, which was made in good faith, you said "American Chemical Society" when the link clearly says it's "American Chemistry Council" (though maybe there's some context I'm missing).
That medium article is pretty bad. Arguing that "prediction rates are similar" means a lack of a problem is a really shitty argument. Imagine african americans and white people both re-offended 50% of the time. If COMPAS predicted a 10% recidivism rate for white people and an 80% recidivisim rate for african americans, it would be horribly biased yet more accurate for african americans.
The details got worse too.. the Medium author is arguing that Propublica is mistaken when they call the tool inherently biased because you can group the recidivism rates such that the AA rate is ~close to the claimed one but then they admit;
> One point about the ProPublica criticism did appear correct. Caucasians and Hispanics are predicted to recommit crimes at a substantially lower rate than they actually do.
> For Caucasians, they are predicted to recommit 25% of the time, but they actually recommit 39% of the time. And the prediction accuracy for both groups are substantially lower than for African Americans.
So the software isn't biased because if you use a different measure it ~accurately predicts recidivism for AA folks and dramatically underestimates how often Caucasian convicts reoffend? what?
> I get that they are left-leaning, however, every single thing they write comes from the perspective of “let’s start under the assumption that a conservative perspective is wrong and let’s build a case to bolster that assumption.” They’re published not a single story that doesn’t lean left. The assumption that the “left” is always correct until proven otherwise is just a slightly more sophisticated appearing version of what Fox News does more overtly.
> The process first ran off the rails under President Donald Trump. It then festered under the administration of President Joe Biden — which ProPublica’s reporting shows impeded the FAA when it finally decided to act — until a crisis forced an intervention.
..
> But a new impediment had arisen: the Biden administration. The White House was discouraging any public action, as was the FCC, which was now operating with a Democratic chair but was every bit as supportive of 5G and the telecom industry’s position on it as Pai had been. They assured the FAA that the agencies and industries could somehow still work out the problem quietly. (A senior FCC official denies the agency requested any delays.)
Their focus on civil rights, criminal justice and corporate abuses could be coded as left-leaning I guess, but those things seem to be keeping them pretty busy.
> And crucially, we made sure to tell the model not to guess if it wasn’t sure. (AI models are known to hallucinate, and we wanted to guard against that.)
Prompting an LLM not to confabulate won't actually prevent it from doing so. It's so disappointing to see an organization like this, that's mission is to inform the public, used AI not understanding the limitations and then making a claim like this.
From that same article:
> Of course, members of our staff reviewed and confirmed every detail before we published our story, and we called all the named people and agencies seeking comment, which remains a must-do even in the world of AI.
That sounds to me like they absolutely do understand the limitations of the technology they are using.
Criticism feels harsh. Of course models don't know what they don't know. Reporters can have the same biases. They could have worded it better "lowers the probability of hallucinating", but it is correct it helps to guard against it. It's just that it's not a binary thing.
> we made sure to tell the model not to guess if it wasn’t sure
Fair enough, but it's kind of ridiculous that in 2025 this "hack" still helps produce more reliable results.
It definitely does mitigate the risk (pretty substantially in my experience!)
[flagged]
ProPublica does outstanding investigative reporting, to include among many more examples their investigative reporting on the nation's stillbirth crisis [1], our broken healthcare insurance system [2], and how the rich and wealthy avoid paying taxes [3]. Your rattling about "left-leaning this, right-leaning that" comes across as out-of-touch and maybe even a tad unhinged.
[1] https://www.propublica.org/article/stillbirth-prevention-leg...
[2] https://www.propublica.org/topics/health-insurance
[3] https://www.propublica.org/article/the-secret-irs-files-trov...
That's a chemical manufacturing corporation lobbying group you're quoting on the risks of one of their products, not the American Chemical Society.
The American Chemical Council (previously named the Chemical Manufacturers' Association) have a section on their Wikipedia page accusing them of pretending to self regulate the industry by setting up green-washing programs that make progress slower than sites not affiliated with the program.
So without any further knowledge, I'd go with the journalists on this one.
I don't have clear context on this but I also want to say you could cherry pick PRs from my career to make me look however you want. I think especially for outlets with a wide variety of writers and editors and a huge backlog of stories people get caught up on specific mistakes.
Not to put too fine a point on it but even in your post, which was made in good faith, you said "American Chemical Society" when the link clearly says it's "American Chemistry Council" (though maybe there's some context I'm missing).
That medium article is pretty bad. Arguing that "prediction rates are similar" means a lack of a problem is a really shitty argument. Imagine african americans and white people both re-offended 50% of the time. If COMPAS predicted a 10% recidivism rate for white people and an 80% recidivisim rate for african americans, it would be horribly biased yet more accurate for african americans.
The details got worse too.. the Medium author is arguing that Propublica is mistaken when they call the tool inherently biased because you can group the recidivism rates such that the AA rate is ~close to the claimed one but then they admit;
> One point about the ProPublica criticism did appear correct. Caucasians and Hispanics are predicted to recommit crimes at a substantially lower rate than they actually do.
> For Caucasians, they are predicted to recommit 25% of the time, but they actually recommit 39% of the time. And the prediction accuracy for both groups are substantially lower than for African Americans.
So the software isn't biased because if you use a different measure it ~accurately predicts recidivism for AA folks and dramatically underestimates how often Caucasian convicts reoffend? what?
Just an absolutely terrible take - ProPublica is one of the most reliable journalistic outfits out there.
Read the original story that prompted the chemical lobbyists to balk and point out where ProPublica says anything problematic. It's an extensively researched and data-backed story: https://www.propublica.org/article/formaldehyde-epa-trump-pu...
> I get that they are left-leaning, however, every single thing they write comes from the perspective of “let’s start under the assumption that a conservative perspective is wrong and let’s build a case to bolster that assumption.” They’re published not a single story that doesn’t lean left. The assumption that the “left” is always correct until proven otherwise is just a slightly more sophisticated appearing version of what Fox News does more overtly.
Is holding Biden's VA to account "left leaning": https://www.propublica.org/article/when-veterans-cant-access...
Or reporting on the Biden administration's family separations at the border: https://www.propublica.org/article/family-separations-biden-...
Or their lack of follow-through on investigating cyber attacks: https://www.propublica.org/article/cyber-safety-board-never-...
Or the failed FAA 5G rollout with the following quotes (https://www.propublica.org/article/fcc-faa-5g-planes-trump-b...):
> The process first ran off the rails under President Donald Trump. It then festered under the administration of President Joe Biden — which ProPublica’s reporting shows impeded the FAA when it finally decided to act — until a crisis forced an intervention.
..
> But a new impediment had arisen: the Biden administration. The White House was discouraging any public action, as was the FCC, which was now operating with a Democratic chair but was every bit as supportive of 5G and the telecom industry’s position on it as Pai had been. They assured the FAA that the agencies and industries could somehow still work out the problem quietly. (A senior FCC official denies the agency requested any delays.)
Their focus on civil rights, criminal justice and corporate abuses could be coded as left-leaning I guess, but those things seem to be keeping them pretty busy.
To quote Steven Colbert: "reality has a strong liberal bias"
(Not enjoying how folks on HN conflate liberalism with "the left" when they are two VERY different things)