The Column – Cyber Security Week
Let’s start with the news. Joel Feder, automotive journalist from The Drivewas testing a Range Rover, made available by the manufacturer, with a special New Jersey license plate: 34 10 DTM. A similar license plate, 34 03 DTM, was reported stolen in Los Angeles, but was incompletely recorded in the report as 34 DTM. Flock Safety’s automatic cameras would have read only the larger characters of Feder’s license plate, ignoring the “10”, written in smaller characters.
The correspondence triggered a shared alert and the vehicle would be followed for days, until it was stopped in Minnesota by four patrols. After about an hour and a check with Jaguar Land Rover, it emerged that neither the car nor the license plate were stolen; according to the officers, elsewhere the arrest could have taken place with guns blazing.
This story does not tell of a machine gone mad, but something more ordinary and therefore more serious: a human error entered an automatic system, encountered a partial reading and came out dressed in certainty. The sequence is simple and therefore almost elegant. An incomplete piece of data becomes a correspondence, which becomes an alert, and then turns into a tailing and then reaches completion with an arrest. Nobody, along the way, adds proof. What was written on a “34 DTM” piece is an approximation, but it is fed to a system that adds maps, timetables and notifications, and changes into an incontrovertible truth. An automatic license plate reader doesn’t know that a car is stolen. IT simply extracts characters from an image, compares them to a list and reports a similarity. We do the rest, often with an almost religious deference towards everything that flashes on a monitor, but we forget some not insignificant details. Thousands of cameras, millions of swipes, shared lists, notifications arriving while others are already arriving. More data should mean more control, but without brakes and priorities the opposite happens: control turns into delegation. The operator does not ask himself the question “Is this car really the one you are looking for?”, but accepts the answer “The recognition system is not wrong”. Technology, in such cases, does not necessarily create the error. It offers him speed and memory. A typo in a report can remain in a drawer for weeks, but connected to a network of sensors, instead, it crosses states, is replicated between agencies and waits for the wrong person in the right parking lot. If we don’t start designing useful frictions it will get worse and worse. A weak match should force a second check; a license plate with different sized characters should lower the confidence level; an alert involving multiple similar vehicles should raise doubts, not multiply suspicions. Human control only makes sense if it comes before the intervention, not after four police patrols have wasted time.
When the decision is distributed between those who insert the data, those who build the algorithm, those who manage the list and those who intervene in the field, each one only has a fragment of the story which is evidently not enough to cause the brain to turn on.
The lesson, therefore, is not about the infallibility of machines, which has never existed. Rather our habit of mistaking automation for proof and speed for accuracy. A system can be very useful and, at the same time, dangerous when it works in degraded mode without declaring it. When an error enters an automatic system, a serious problem arises for us human beings: it seems more difficult to contradict.
