Consumers who want to submit a complaint to an agency such as the Consumer Financial Protection Bureau face a task that, for some, can be daunting: they must fill out a form that requires them to explain the issue, clearly and convincingly, in their own words. Those who are not native English speakers or simply don’t regularly communicate in writing may lack the skills needed to convincingly make their cases.
New research from Jiwoong Shin of Yale SOM reveals that the recent wave of consumer Large Language Models like ChatGPT may offer an unexpected solution to this problem.
Shin and his co-authors, Minkyu Shin of the City University of Hong Kong and Jin Kim of Northeastern University, conducted an analysis of data from the Consumer Financial Protection Bureau (CFPB). They identified a surge in AI-edited complaints against financial firms following the release of ChatGPT in November 2022.
This trend was especially pronounced in places where many residents have limited English proficiency. And the AI assistance seems to have helped—LLM usage, the researchers found, was associated with an increase in the likelihood of receiving relief from firms, whether monetary or non-monetary.
Shin and his co-authors supplemented their analysis of CFPB data with several experiments that further explore the relationship between LLM usage and complaint resolution. These studies show that LLMs help consumers craft clearer and more persuasive complaints, resulting in a better chance of receiving relief.
Together, the analysis of CFPB data and the experiments suggest that “LLMs create a level playing field for everyone,” Shin explains. “By using LLMs, people with limited English proficiency can express their concerns or complaints effectively, and companies can decide to address complaints based on the content, not just the presentation.”
The researchers started with a set of more than a million complaints against financial services firms submitted to the CFPB between March 2015 and March 2024. Under the agency’s rules at the time, firms were required to respond to these complaints by providing relief, whether monetary or non-monetary, or an explanation about why relief was not warranted. (As of March 2025, most of the CFPB’s operations have been halted.)
This dataset was especially valuable because consumers must submit detailed narratives about their complaints and accurate information about themselves, including their geographic locations. Unlike, say, a negative review on a site like Yelp, a complaint to a government agency such as CFPB “has a potential legal implication,” Shin says, “so there is much less of a concern about fake complaints.”
The researchers used a commercial AI detection tool to analyze the text of these complaints. The tool provided scores from 0 to 100 that represented the likelihood the complaint had been written with AI assistance; the researchers coded complaints as AI-written only if this score was 99 or above. Even using this conservative approach, they observed a sudden jump in AI-assisted complaints following the introduction of ChatGPT—from essentially none to 9.8% by March 2024, “which is quite remarkable,” Shin says.
The researchers then looked at the ZIP codes where the complaints had originated. A disproportionate share of AI-assisted complaints, they found, came from areas where a higher proportion of residents have limited English proficiency. In other words, it seemed likely that consumers who didn’t speak English comfortably turned to ChatGPT to help them draft their complaints.
Shin and his co-authors also compared AI-assisted complaints to those the AI detection system had scored as probably human-authored. AI-assisted complaints had a 49.3% likelihood of obtaining relief, as compared to 39.9% for the human-authored complaints.
AI usage may have correlated with a higher likelihood of obtaining relief, but did it cause the improved outcomes? To get closer to answering that question, the researchers used a type of statistical analysis called instrumental variables estimation.
They looked at two factors that should logically be correlated with increased use of AI after the introduction of ChatGPT but shouldn’t otherwise affect the outcome of complaints: internet access and English proficiency, both measured at the ZIP code level. (They controlled for factors including income, education level, and employment status.)
This analysis revealed that, indeed, complaints from areas with fewer proficient English speakers and from areas with higher levels of internet access had better outcomes after ChatGPT was released.
However, even with the help of instrumental variables estimation, the researchers couldn’t say for sure why AI-assisted complaints received better outcomes. They suspected it was because AI allowed for more effective presentation of consumer concerns, but it was possible that some other factor was at work—for example, perhaps people who took the time to refine their complaint with ChatGPT simply had stronger cases for relief.
To untangle the relationship between content and presentation, the researchers ran an experiment in which online participants reviewed complaints that were identical in content but differed in whether they were unedited or edited by ChatGPT to improve clarity, coherence, or professionalism.
Participants—imagining themselves as the party responsible for addressing the complaints—rated from one to seven how likely they would be to provide monetary relief in response to each complaint. The participants, the researchers found, were significantly more likely to provide relief to the AI-edited complaints.
The researchers repeated the same experiment with participants who had previously worked in the financial industry. This time, in addition to rating the likelihood of providing monetary relief, participants also scored the complaints on clarity and professionalism. And as expected, AI-edited complaints were deemed clearer and more professional—and were more likely to receive relief.
To Shin, the results of the second experiment were especially striking. Going into the study, he thought it was possible that financial professionals “might be a little more sophisticated” and therefore able to identify meritorious complaints, regardless of presentation style. However, even among people with more experience, AI-edited content still outperformed unedited content, suggesting that how well a complaint is articulated meaningfully affects its outcome.
In the future, Shin hopes to understand how organizations might respond to an increase in AI-edited complaints. “It seems like they are benefiting people that are using LLMs—is that fair?” Shin asks.
Perhaps, he suggests, firms should integrate AI tools into the forms through which consumers submit complaints, equalizing access. Another response might be to train staff to separate content and presentation when adjudicating consumer complaints.
Amid a sea of concerns about how LLMs might change our society for the worse, Shin sees the study as identifying one valuable upside: “They increase welfare for the entire society, because no one is at a disadvantage simply because of their communication skills.”
Source: https://techxplore.com/
