“Don't believe anything I tell you, it's all lies!”: a synthetic ethnography on untruth in large language models

Holmquist, LE ORCID: https://orcid.org/0000-0003-0229-2345 and Nemeth, S, 2025. “Don't believe anything I tell you, it's all lies!”: a synthetic ethnography on untruth in large language models. In: Yamashita, N, Evers, V, Yatani, K and Ding, X, eds., CHI EA '25: extended abstracts of the CHI Conference on Human Factors in Computing Systems. New York, NY: ACM. ISBN 9798400713958

Preview

Text 2442065_Holmquist.pdf - Published version Download (198kB) | Preview

Abstract

Large Language Models (LLMs) have many practical uses in areas like journalism, search, coding and more. However, a growing concern is that they are also prone to presenting incorrect information, sometimes called “hallucinations”. Here, we are not interested in what specific untruths LLMs presents, but how they do it. We used synthetic ethnography, a methodology for the qualitative study of generative models, to study two LLMs with different size and capability. We collected 3 cases where LLMs presented incorrect information and observed the strategies they used to justify this. From these observations we can start to form an understanding of what happens when an LLM reaches the edge of its knowledge-base and takes corrective action. Our conclusion is that the interfaces should be better designed to reveal this tendency of LLMs to “fill in” information they are missing, but that this ability may also be one of their strengths.

Item Type:	Chapter in book
Description:	Paper presented at CHI EA '25, Yokohama, Japan, 26 Apr-1 May 2025
Creators:	Holmquist, L.E. and Nemeth, S.
Publisher:	ACM
Place of Publication:	New York, NY
Date:	26 April 2025
ISBN:	9798400713958
Identifiers:	Number Type 10.1145/3706599.3719939 DOI 2442065 Other
Rights:	© 2025 Copyright held by the owner/author(s) Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
Divisions:	Schools > Nottingham School of Art & Design
Record created by:	Jeremy Silvester
Date Added:	22 May 2025 13:40
Last Modified:	22 May 2025 13:40
URI:	https://irep.ntu.ac.uk/id/eprint/53641

Actions (login required)

Edit View

Statistics

Views

Downloads