After showing how Large Language Models (LLMs) can "guess" mental states from online posts or even rival expert psychiatrists in diagnostic accuracy, the REMEDI Lab (REthinking MEntal health through Clinical and Data Intelligence), part of USI Euler Institute , has now published a new study in Scientific Reports by Nature that raises a radical question: do psychological tests really measure what we think they measure? The work, authored by Antonietta Mira (Full Professor at USI Faculty of Economics), Andrea Raballo (Full Professor of Psychiatry at USI Faculty of Biomedical Sciences and Head of Research and Academic Training at the Cantonal Socio-Psychiatric Organisation (OSC), together with Federico Ravenda (USI), Antonio Preti (University of Turin), and Michele Poletti (IRCCS-AUSL Reggio Emilia), builds on research already presented in recent months and significantly expands its scientific scope.
The new study introduces an innovative perspective: large language models (LLMs) can predict in advance how questions in a psychological questionnaire correlate with each other, based solely on their linguistic similarity. This suggests that what questionnaires "measure" does not depend solely on people’s answers, but is already partly inscribed in the wording of the items themselves.
Through analyses conducted with established instruments-such as the Big Five Personality and the DASS-42--the team showed that the items that are most semantically similar are also empirically highly correlated: 95% of cases in the DASS and 82% in the Big Five. Therefore, it appears that the structure of these tests emerges not only from the data collected but also from the way the items are linguistically constructed.
On this basis, the researchers developed PsychoLLM, an architecture that uses only the semantics of items to predict responses to standard clinical instruments such as GAD-7 and PHQ-9. The model achieved 70% accuracy in predicting responses on one scale from the other, demonstrating the predictive power of linguistic relationships between questions.
The study highlights an important epistemological consequence: item semantics can influence psychological measurement much more than traditionally assumed. This has significant implications for questionnaire construction, data quality assessment, and the correct interpretation of results. The team also emphasises the need to verify whether these effects are also present in other languages and cultural contexts, given that linguistic structures and response styles can vary significantly.
