Machine learning techniques currently make "high-stakes" judgments in areas as diverse as criminal justice, credit risk, social welfare, hiring, and congressional redistricting. Such techniques make these decisions using patterns learned from historical social data. Emphasis on prediction rather than the circumstances of dataset creation have led to machine learning systems that preferentially target vulnerable populations for disparately adverse social judgments while making it more difficult for those subject to these decisions to protest unfair treatment.