" Yet experts and researchers have long called out the bias algorithms can perpetuate. Amazon built a hiring tool that discriminated against women. Tweets from black people were more likely to be dubbed “toxic” in a Google-funded AI tool. Facial recognition tools used by the US government have been shown to misidentify black people much more often than white people.
An algorithm is also at risk of reflecting structural inequality, as with zip codes. In the 1930s, the US explicitly segregated African-American neighborhoods from white ones through policies known as “redlining.” Today, these policies have lasting impact on racial makeup on zip codes — which means using these zip codes in algorithms perpetuates racial inequality.