Digital Qualified Immunity: When Software Decides and Nobody Answers
Ep. 82

Digital Qualified Immunity: When Software Decides and Nobody Answers

•

Episode description

When the government hands a decision about your life to software, who answers for it — and how does anyone check its work?

Joshua and Will borrow a real legal doctrine, qualified immunity, to name something newer: digital qualified immunity. The agency says the software decided. The vendor says the software is a trade secret. The person who lost their benefits or their job is left with nobody to question.

In this episode:

  • Arkansas ARChoices. A formula replaced nurses’ judgment and cut home care hours for disabled and elderly Medicaid recipients. A judge blocked it in 2018, not because the math was wrong, but because nobody was told the rules changed.
  • Houston ISD. Teachers were rated and fired using SAS’s EVAAS scores, a model so secret that the district itself couldn’t check it. The case settled for $237,000, and the district stopped firing teachers on those scores.
  • Michigan’s MiDAS. An automated system falsely accused tens of thousands of people of unemployment fraud. The state’s own review found a 93% error rate. That led to Bauserman, a $20 million settlement, and Cahoo v. SAS Analytics, where the vendors were sued by name.
  • Due process 101. Goldberg v. Kelly, the Mathews v. Eldridge balancing test, and why private prison guards can’t claim qualified immunity.
  • California SB 947, the “No Robo Bosses” Act. What “relies primarily” might mean, and whether a manager skimming an AI-written summary counts as human review.
  • What the contract should say. No trade-secret shield against the person affected, reproducible versioned models that auditors can run, vendors sharing liability, and treating any change to scoring weights as a rule change.

Consistent isn’t the same as correct. If software made a decision about your life this year, how would you know?

Note: We ran into some audio/video sync issues during recording. You may notice drift or more crosstalk than usual in places. We fixed what we could in editing. Thanks for bearing with us.