A new study suggests ChatGPT may stereotype you based on where you live, with the artificially-intelligent model classifying some locations as smarter, some as smellier and some as uglier or stupider. Researchers from the University of Oxford and the University of Kentucky asked OpenAI’s GPT-4o-mini model over 20 million questions between March and May of last year. They got the ChatGPT model to compare two places in order to get “a single, unambiguous consistent answer” about how it classifies certain states, according to researchers. If you directly ask ChatGPT whether people living in certain states are stupid, the model might decline to answer. But when backed into a corner and forced to choose between pairs, the model started making harder choices, the researchers found. The more certain states appeared in answers, the higher they would get ranked in answers about intellectual, creative and physical attributes. Ultimately, the AI model considered Massachusetts the smartest state. Louisiana was the smelliest. Ohio was the ugliest. North Dakota had the least sexy people. Asked which states had stupider people, the top three states ChatGPT named were Kentucky, West Virginia and Mississippi, while the top three states it considered to have the least stupid people were Hawaii, Colorado and New Hampshire. “We’re most concerned about how certain ideas get normalized, like the idea that people in Kentucky are stupider than anyplace else,” said Matt Zook, a geography professor at the University of Kentucky and co-author of the research. He said the model reinforces “dominant narratives about certain places being like this, certain places being like that.” Beyond the United States, the researchers also found that ChatGPT ranked places like the U.S. and Western Europe as having more desirable traits, like a smarter and more stylish populace, than sub-Saharan African countries. You can look up exactly how ChatGPT classifies your city, state, and country based on certain attributes at the researchers’ website: inequalities.ai. The full research was also published in the journal Platforms & Society. In general, “whiter, richer, less-immigrant communities were more beautiful, less smelly, smarter,” according to the model, Zook said. The model is “reinforcing what it’s learned from the data it’s been fed ... which includes the biases, the prejudice” of available training materials, he said. In this way, the ChatGPT answers are not neutral and can reflect racist ideas. For example, Mississippi, a state with a high population of Black residents, was the state with the most “ignorant” people, according to the model. “Long histories of racism and classism are reflected in the training data used for AI models — that’s what the infographics are showing,” said Safiya Noble, a professor at the University of California, Los Angeles, and the author of the book “Algorithms of Oppression.” When HuffPost reached out to OpenAI about the researchers’ conclusion that “this bias is fundamentally structural, and no amount of fine-tuning fully removes the geopolitical hierarchies baked into their data and design,” the company stated that the researchers had used an outdated model. “ChatGPT is designed to be objective by default and to avoid endorsing stereotypes,” OpenAI said in a statement. “Research based on forced-choice prompts and older models doesn’t reflect how ChatGPT is typically used or how current models behave today.” But the company did acknowledge that it is continuously improving “how ChatGPT handles subjective or non-representative comparisons.” And there might be more work to do to reduce this subjectivity, according to my own informal testing. When I asked ChatGPT to tell me a career story about a man from Kentucky — a state that the AI model had ranked low in intellect and beauty, according to the research — it told me about a man who went to a local technical college and got an entry-level job at a factory. When I asked the model to do same about a man from Hawaii — a state that it had ranked high in intellect — the model told me about a man who attends a four-year college out of state and becomes an environmental engineer. It’s a current example of the different subjective scripts the model might have — and the real-world biases people might have about the intellectual superiority and capabilities of people from certain states. This study builds upon a growing body of research on large language model biases. In Zook’s view, the ChatGPT geographic biases “might be better hidden” in future models through certain keywords that generate different responses, “but it’s still going to be in there.” Zook said the model might not directly answer a question like “Are people from these cities stupid or smart?” but it might answer which cities or states are best to recruit strong medical doctors or software developers from, for example. “That could be shaping how someone’s structuring a [job] search,” he said. “As these things get ... used in ways that we don’t even realize, that’s where it becomes even more problematic.” The original version of this story was published on HuffPost at an earlier date.

Source: HuffPost

Read Original Source →