A DEF CON Guide to Adversarial Testing of Software

Architecture and Background

Software is increasingly used to make huge decisions about people's lives and often these decisions are made with little transparency or accountability to individuals. If there is any place where transparency, third-party review, adversarial testing and true accountability is essential, it is the criminal justice system.

Vulnerability Analysis

Nevertheless, proprietary software is used throughout the system, and the trade secrets of software vendors are regularly deemed more important than the rights of the accused to understand and challenge decisions made by these complex systems. In this talk, we will lay out the map of software in this space from DNA testing to facial recognition to estimating the likelihood that someone will commit a future crime.

Exploitation and Demonstrations

We will detail the substantial hurdles that prevent oversight and stunning examples of real problems found when hard won third-party review is finally achieved. Finally, we will outline what you as a concerned citizen/hacker can do.

Remediation and Defensive Strategies

Nathan Adams will demo his findings from reviewing NYC's FST source code, which was finally made public by a federal judge after years of the city's lab fighting disclosure or even review. Jerome Greco will provide his insight into the wider world of software used in the criminal justice system—from technology that law enforcement admits to using but expects the public to trust without question to technology that law enforcement denies when the evidence says otherwise. Jeanna Matthews will talk about the wider space of algorithmic accountability and transparency and why even open source software is not enough.