The Algorithm That Learned Bias (And Who Taught It)

In 2018, Amazon quietly shut down an experimental hiring tool it had been building for several years. The system was designed to automate the screening of job applicants, using machine learning to identify the best candidates from a flood of resumes. It was an ambitious project, and it failed spectacularly. The reason was not a bug in the code. It was a flaw in the data.

The System

Amazon’s engineers built the recruiting engine to do what machine learning does best: recognize patterns. The system was trained on a decade’s worth of resumes submitted to Amazon. It analyzed thousands of data points, words, phrases, educational backgrounds, employment history, and learned to associate certain patterns with successful hires.

The goal was to identify candidates who looked like the people Amazon had already hired. The system was given a simple objective: score resumes on a scale of one to five stars, mimicking the way human recruiters rated candidates.

The Discovery

In 2018, Reuters reported that the system had developed a significant and deeply problematic bias. It was penalizing resumes that included the word “women’s”, as in “women’s chess club captain” or “women’s studies.” It was downgrading graduates of two all-women’s colleges. It had effectively taught itself that women were worse candidates than men.

The reason was buried in the training data. Amazon’s workforce, particularly in technical roles, was overwhelmingly male. The resumes the system learned from were predominantly from men. The successful hires it was trained on were mostly men. The system learned that male was the default.

It did not learn this because anyone programmed it to be sexist. It learned it because the data reflected a world where men had historically been favored in tech hiring. The algorithm was a mirror. And the reflection was ugly.

Who Taught It?

The answer is uncomfortable. The algorithm was not taught by a malicious engineer or a biased dataset deliberately seeded with sexist patterns. It was taught by Amazon itself. It was taught by a decade of hiring decisions that had resulted in a predominantly male workforce. It was taught by a culture that had, for years, failed to recruit and retain women in technical roles.

The algorithm learned from the past. The past was biased. The algorithm reproduced the bias. The problem was not the technology. The problem was the institution that created it.

The Response

Amazon’s engineers attempted to fix the problem. They edited the algorithm to neutralize the terms that had become problematic. But they could not fix the underlying issue. The system was still learning from data that reflected a world that had not yet achieved equity. Any attempt to correct the bias was superficial. The system would find new proxies for gender, new patterns to discriminate against.

By 2018, the project was abandoned. Amazon’s recruiters had lost confidence in the tool. The system that was supposed to make hiring fairer had become another obstacle for women trying to enter the company.

The Lesson

The Amazon hiring tool is not an outlier. It is a case study in a fundamental problem with machine learning. Algorithms learn from data. Data reflects the world as it is. The world is biased. Therefore, algorithms are biased.

This is not a flaw that can be fixed with better engineering. It is a flaw that requires addressing the root cause: the systems and institutions that generate the data. An algorithm trained on a decade of biased hiring decisions will perpetuate that bias. It does not matter how sophisticated the model is. The model is only as good as the world it learns from.

The responsibility does not lie with the algorithm. It lies with the people who created it, trained it, and deployed it without questioning what it was learning. The algorithm learned bias because the people who built it did not understand what the data was teaching. That is the lesson. And it is one that every designer, engineer, and product manager working with AI must internalize.