Published on August 21, 2024 9:05 PM GMT
This was originally posted in the comments of You don't know how bad most things are nor precisely how they're bad.; I've broken it out into a post because I think it might be a useful corrective more generally for people inclined to cite LLM remarks as fact.
I asked Claude, as an illustrative example, whether ready-made clothing had declined in quality over time, and it claimed that "many studies" indicated this was so. I subsequently asked for it to cite the studies in question; half of them were confabulated and AFAICT only one of them contained what Claude said it did. (Note that this is Claude 3 Haiku, which isn't the best; I'd expect frontier models to name four or maybe even five real papers in a list of six (and, with lower confidence, to have a more accurate understanding of the papers' contents)).
Here are the "studies" it cited—note that even some of the ones which actually exist do not contain the things Claude said they did.
- "A New Textiles Economy: Redesigning Fashion's Future" - Report by the Ellen MacArthur Foundation (2017) - This is a think-tank report (available here) rather than a study. However, it does actually exist. Claude says it "noted that the number of times clothes are worn before disposal has decreased by 36% compared to 15 years ago." It really does say this! Page 19, citing as a source the "Circular Fibres Initiative analysis based on Euromonitor International Apparel & Footwear 2016 Edition (volume sales trends 2005–2015)." A-. (Teeeeeechnically the report claims the clothing is still wearable and therefore throwing it away is a waste, which isn't the same thing as poor durability causing decreased wear time; humans cite studies that support something a little to the left of their point all the time, though, so in the interests of fairness I won't mark it down for that)."The State of Fashion 2016" - Report by the McKinsey Global Fashion Index (2016) - This one seems to be real! (Technically, it's "The State of Fashion 2017"—their first report, so Claude can't have meant an earlier one—but it was released in 2016 and half of it is an overview of same, so IMO it's close enough.) It is not a research study, but rather a think-tank report (which is actually even worse than it seems, IMO, because on the few occasions I've checked sources on think-tank reports I've sometimes found that the results cited didn't seem to actually exist anywhere). Claude says it "concluded that the quality and durability of clothing has declined as the industry has shifted towards faster production cycles and lower prices." The report does say the industry has moved toward faster production cycles and lower prices (though it indicates that production costs have actually risen, leaving the authors quite worried about their profit margins), but does not, as far as I can tell, claim that quality and durability have declined."Valuing Our Clothes: The Cost of UK Fashion" - Study by the Waste & Resources Action Programme (WRAP) in the UK (2012) - This one actually exists! (Full report here). It's again not technically a study, but rather a "summary of the key findings of a major technical report (possibly this one?) published by WRAP"; however if they did any research themselves that's practically hard science compared to the other two, so whatever. However, Claude claims that it "found that the average number of times a garment is worn before being discarded has fallen by 36% compared to 15 years earlier," and as far as I can tell this is not true; I couldn't find a place in the report where it even mentioned the number of wearings before discarding.
Posting this list of sources without annotations would have been of negative value, leading unwary readers to change their minds based on studies that don't exist and forcing somewhat cannier ones to spend what could be several hours checking the bullshit I produced in a few minutes.
Moreover, LLMs will take almost any position you ask of them, regardless of whether or not it's true. Just for fun, here's Claude's list of five studies proving clothing's actually gotten better over time (veracity left as an exercise for the reader):
A 2020 study by the University of Cambridge called "The Changing Lifespan of Garments: Evidence from Cambridge Consumer Survey Data" found that the average lifespan of mass-produced garments has increased by 30% over the past 20 years, due to advancements in manufacturing techniques and materials.Research by the American Apparel & Footwear Association called "Quality Trends in the US Apparel Industry" showed that the number of clothing items returned due to quality issues has decreased by 25% since 2015, suggesting clothing is better made.A 2019 consumer survey by McKinsey & Company called "The State of Fashion 2019" found that 66% of respondents reported being satisfied with the quality of their ready-to-wear purchases, up from 58% in 2014.A 2021 analysis by the UK's Clothing Sustainability Research Centre called "UK Clothing Sustainability Report 2021" concluded that the average quality rating for high-street fashion brands increased by 18% over the past decade, as they invested in improving manufacturing processes.A 2018 report by the International Federation of Textile Engineers called "Global Textile Industry Quality Trends" noted a 23% reduction in clothing defects across major global brands, attributed to enhanced quality control measures.
For these reasons, I do not think it is appropriate to cite the statements of LLMs as though they are evidence in favor of your position.
Discuss
