03 July 2012

Drug Dogs & Information Retrieval

The Agitator | Radley Balko | "Good Enough" for Probably Cause

“Bono” is a state police drug dog in North Carolina. Of the 85 times Bono has alerted his handler to the presence of drugs, a subsequent search turned up actual drugs just 23 times.

No matter. Federal District Court Judge Glen Conrad ruled late week that Bono’s record is still good enough to establish probable cause.
Bono “may not be a model of canine accuracy,” Conrad wrote in an opinion filed Thursday in U.S. District Court in Roanoke.

However, the judge ruled that other factors, including the dog’s training and flawless performance during re-certification sessions, were enough to overcome a challenge raised by Green’s attorney, public defender Randy Cargill.
Conrad’s justification for allowing the search illustrates just how clueless federal judges can be about these things—and why they can be such poor custodians of the Fourth Amendment. Judges have far too little skepticism for law enforcement officials.
Too little skepticism for law enforcement and too little knowledge of math. A precision of 0.27 is astoundingly bad, especially for this circumstance.

And so what if he performed flawlessly in training? Training error doesn't count! No self-respecting, honest assessor would put any stock in the training error of a decision or detection system. You can only evaluate against samples not seen in training. I would fail an undergrad for not understanding that.
What’s incredible is that even if everything the U.S. Attorney says is true, the dog and his handler still have a 50 percent rate of error. Which means they’re no better than a coin flip. A coin flip is good enough for Judge Conrad.
No, it's even worse than that. Even if everything the USA says is true, the dogs have a precision of 0.5. Since they certainly have non-zero false negatives, their accuracy is even worse than a coin flip. Not to be pedantic about this, but the distinction between false positives and false negatives is a really important one in this context.

(We should be especially vigilant against false positives in this context because — in theory at least — our judicial system puts a higher cost on wrongly punishing the innocent than on mistakenly letting the guilt off.)

PS I wonder how this judge would feel about this new system for detecting fraud and graft in the judiciary I'm developing. It causes more false positives than true positives, but there are "other factors" that I think are more important. I'm sure he'd be fine if I audited him and everyone else in his courthouse whenever my worse-than-a-coin-flip system indicated that a judge was corrupt. Right?

1 comment: