- Media - Feb 11 Gender and skin- type bias in commercial artificial- intelligence systems
- Media - Feb 5 High Number of Science Enthusiasts in Switzerland
- Media - Jan 29 The four personas including the Lurker and the Geek that explain teenagers’ online behaviour
- Media - Jan 19 MLK commemoration highlights the struggle for civil rights and inclusion
- Law - Nov 27 After 40 Years, Copyright Law Needs To Be Tweaked
- Media - Nov 16 3 Questions: Lisa Parks on drones, warfare, and the media
- Physics - Nov 10 The Path Length of Light in Opaque Media
- History - Nov 10 Stanford University Press celebrates 125th anniversary
- Media - Nov 1 Social Media’s Dark Side Tainted Our Democratic Institutions
- Media - Oct 25 Media attention to drought produced water savings
- Media - Oct 19 Hear My Voice: News project makes space for inclusive conversation
- Media - Oct 19 Distrust of Washington and Media Grows Since 2016 Election
- Media - Oct 11 Summer of Love language: Still compelling after all these years
- Environment - Oct 10 The 2015 Paris Climate Conference: For Germans, the intensive media coverage had a calming, not a mobilizing effect
- Media - Sep 29 New podcast explores people, places behind the food we eat
- Media - Sep 28 ’Junk news’ concentrated in key US swing states before 2016 election
Gender and skin-type bias in commercial artificial-intelligence systems
Three commercially released facial-analysis programs from major technology companies demonstrate both skin-type and gender biases, according to a new paper researchers from MIT and Stanford University will present later this month at the Conference on Fairness, Accountability, and Transparency.
In the researchers’ experiments, the three programs’ error rates in determining the gender of light-skinned men were never worse than 0.8 percent. For darker-skinned women, however, the error rates ballooned - to more than 20 percent in one case and more than 34 percent in the other two.
The findings raise questions about how today’s neural networks, which learn to perform computational tasks by looking for patterns in huge data sets, are trained and evaluated. For instance, according to the paper, researchers at a major U.S. technology company claimed an accuracy rate of more than 97 percent for a face-recognition system they’d designed. But the data set used to assess its performance was more than 77 percent male and more than 83 percent white.
"What’s really important here is the method and how that method applies to other applications," says Joy Buolamwini, a researcher in the MIT Media Lab’s Civic Media group and first author on the new paper. "The same data-centric techniques that can be used to try to determine somebody’s gender are also used to identify a person when you’re looking for a criminal suspect or to unlock your phone. And it’s not just about computer vision. I’m really hopeful that this will spur more work into looking at [other] disparities."
Buolamwini is joined on the paper by Timnit Gebru, who was a graduate student at Stanford when the work was done and is now a postdoc at Microsoft Research.
The three programs that Buolamwini and Gebru investigated were general-purpose facial-analysis systems, which could be used to match faces in different photos as well as to assess characteristics such as gender, age, and mood. All three systems treated gender classification as a binary decision - male or female - which made their performance on that task particularly easy to assess statistically. But the same types of bias probably afflict the programs’ performance on other tasks, too.
Indeed, it was the chance discovery of apparent bias in face-tracking by one of the programs that prompted Buolamwini’s investigation in the first place.
Several years ago, as a graduate student at the Media Lab, Buolamwini was working on a system she called Upbeat Walls, an interactive, multimedia art installation that allowed users to control colorful patterns projected on a reflective surface by moving their heads. To track the user’s movements, the system used a commercial facial-analysis program.
The team that Buolamwini assembled to work on the project was ethnically diverse, but the researchers found that, when it came time to present the device in public, they had to rely on one of the lighter-skinned team members to demonstrate it. The system just didn’t seem to work reliably with darker-skinned users.
Curious, Buolamwini, who is black, began submitting photos of herself to commercial facial-recognition programs. In several cases, the programs failed to recognize the photos as featuring a human face at all. When they did, they consistently misclassified Buolamwini’s gender.
To begin investigating the programs’ biases systematically, Buolamwini first assembled a set of images in which women and people with dark skin are much better-represented than they are in the data sets typically used to evaluate face-analysis systems. The final set contained more than 1,200 images.
Next, she worked with a dermatologic surgeon to code the images according to the Fitzpatrick scale of skin tones, a six-point scale, from light to dark, originally developed by dermatologists as a means of assessing risk of sunburn.
Then she applied three commercial facial-analysis systems from major technology companies to her newly constructed data set. Across all three, the error rates for gender classification were consistently higher for females than they were for males, and for darker-skinned subjects than for lighter-skinned subjects.
For darker-skinned women - those assigned scores of IV, V, or VI on the Fitzpatrick scale - the error rates were 20.8 percent, 34.5 percent, and 34.7. But with two of the systems, the error rates for the darkest-skinned women in the data set - those assigned a score of VI - were worse still: 46.5 percent and 46.8 percent. Essentially, for those women, the system might as well have been guessing gender at random.
"To fail on one in three, in a commercial system, on something that’s been reduced to a binary classification task, you have to ask, would that have been permitted if those failure rates were in a different subgroup?" Buolamwini says. "The other big lesson... is that our benchmarks, the standards by which we measure success, themselves can give us a false sense of progress."
"This is an area where the data sets have a large influence on what happens to the model," says Ruchir Puri, chief architect of IBM’s Watson artificial-intelligence system. "We have a new model now that we brought out that is much more balanced in terms of accuracy across the benchmark that Joy was looking at. It has a half a million images with balanced types, and we have a different underlying neural network that is much more robust."
"It takes time for us to do these things," he adds. "We’ve been working on this roughly eight to nine months. The model isn’t specifically a response to her paper, but we took it upon ourselves to address the questions she had raised directly, including her benchmark. She was bringing up some very important points, and we should look at how our new work stands up to them."
- Cornel West advocates the "examined life" on campus
- Joy Buolamwini wins national contest for her work fighting bias in machine learning
- Institute launches the MIT Intelligence Quest
- Designing virtual identities for empowerment and social change
- President Obama discusses artificial intelligence with Media Lab Director Joi Ito
Last job offers
- Media - 12.2
Postdoctoral position on Artificial Intelligence and Communication
- Media - 9.2
Postdoctoral position on Mobile Communication Research
- Media - 31.1
Universitätsassistent/in post doc Non Tenure Track oder Universitätsassistent/in prae doc
- Media - 16.1
- Media - 22.4
Professuren, Carl-Friedrich-Gauß-Fakultät, Universitätsprofessur für Kommunikations- und Medienwissenschaften...
- Media - 6.2
Research Associate - Transitional Justice and Transitional Journalism
- Media - 19.1
Research Fellow in Fibrous Filter Media Development
- Media - 1.11
Center for Digital Art & Experimental Media (dxarts) - Research Associate (nn26118)