Large Language Model (LLM)-Based Conversational Survey Design and Comparison with Web-Based Survey

Falana Rofako Hakam; Nori Wilantika

doi:10.32734/jocai.v10.i1-22660

Authors

Falana Rofako Hakam Politeknik Statistika STIS
Nori Wilantika Politeknik Statistika STIS https://orcid.org/0000-0002-6353-9527

DOI:

https://doi.org/10.32734/jocai.v10.i1-22660

Keywords:

conversational survey, large language model (LLM), retrieval-augmented generation (RAG), response quality, user perceptions

Abstract

This study addresses the low response quality often observed in conventional web-based surveys due to respondent satisficing. This study developed a prototype conversational survey powered by a Large Language Model (LLM) that applies prompt engineering and Retrieval-Augmented Generation (RAG) to enable more natural survey interactions. A comparative experiment was conducted between the LLM-based conversational survey and a conventional web survey with 36 respondents whose characteristics were matched. The evaluation focused on response quality and user perceptions. Statistical analyses show that the LLM-based conversational survey significantly reduces satisficing, evidenced by a lower rate of item nonresponse (p-value = 0.0036) and longer per-item response times (p-value = 0.0001), indicating greater respondent engagement. From a user-experience perspective, the LLM-based conversational survey was rated as significantly more enjoyable (p-value = 0.023) and cognitively less demanding (p-value = 0.0002). This study concludes that LLM-based conversational surveys can simultaneously improve response quality and user experience.

Downloads

Download data is not yet available.

References

M. P. Couper and P. V. Miller, “Web Survey Methods: Introduction,” Public Opin. Q., vol. 72, no. 5, pp. 831–835, Dec. 2008, doi: 10.1093/poq/nfn066.

H. L. Ball, “Conducting Online Surveys,” J. Hum. Lact., vol. 35, no. 3, pp. 413–417, Aug. 2019, doi: 10.1177/0890334419848734.

S. Lefever, M. Dal, and Á. Matthíasdóttir, “Online data collection in academic research: advantages and limitations,” Br. J. Educ. Technol., vol. 38, no. 4, pp. 574–582, Jul. 2007, doi: 10.1111/j.1467-8535.2006.00638.x.

N. Berzelak and V. Vehovar, “Mode effects on socially desirable responding in web surveys compared to face-to-face and telephone surveys,” Adv. Methodol. Stat., vol. 15, no. 2, Jul. 2018, doi: 10.51936/lrkv4884.

J. L. Jenkins, J. S. Valacich, and P. Williams, “Human-computer interaction movement indicators of response biases in online surveys,” 2017.

D. Heerwegh and G. Loosveldt, “Face-to-Face versus Web Surveying in a High-Internet-Coverage Population: Differences in Response Quality,” Public Opin. Q., vol. 72, no. 5, pp. 836–846, Dec. 2008, doi: 10.1093/poq/nfn045.

J. A. Krosnick, “Response strategies for coping with the cognitive demands of attitude measures in surveys,” Appl. Cogn. Psychol., vol. 5, no. 3, pp. 213–236, May 1991, doi: 10.1002/acp.2350050305.

M. Callegaro, K. L. Manfreda, and V. Vehovar, Web Survey Methodology. 1 Oliver’s Yard, 55 City Road London EC1Y 1SP: SAGE Publications Ltd, 2015. doi: 10.4135/9781529799651.

C. C. Vriesema and H. Gehlbach, “Assessing Survey Satisficing: The Impact of Unmotivated Questionnaire Responding on Data Quality,” Educ. Res., vol. 50, no. 9, pp. 618–627, Dec. 2021, doi: 10.3102/0013189X211040054.

BPS, “Introducing Administrative Data to the Census in Indonesia,” 2022.

APJII, “Laporan Survei Penetrasi & Profil Perilaku Pengguna Internet Indonesia 2019–2020 (Q2),” Asosiasi Penyelenggara Jasa Internet Indonesia, Jakarta, 2020. [Online]. Available: https://survei.apjii.or.id/

C. F. Cannell, L. Oksenberg, and J. M. Converse, “Striving for Response Accuracy: Experiments in New Interviewing Techniques,” J. Mark. Res., vol. 14, no. 3, p. 306, Aug. 1977, doi: 10.2307/3150768.

Y. P. Ongena, “Interviewer and Respondent Interaction in Survey Interviews,” Vrije Universiteit, Netherlands, 2005.

A. R. Artino, Q. R. Youmans, and M. G. Tuck, “Getting the Most Out of Surveys: Optimizing Respondent Motivation,” J. Grad. Med. Educ., vol. 14, no. 6, pp. 629–633, Dec. 2022, doi: 10.4300/JGME-D-22-00722.1.

B. Abu Shawar and E. Atwell, “Chatbots: Are they Really Useful?,” J. Lang. Technol. Comput. Linguist., vol. 22, no. 1, pp. 29–49, Jul. 2007, doi: 10.21248/jlcl.22.2007.88.

M. Jovanovic, M. Baez, and F. Casati, “Chatbots as Conversational Healthcare Services,” IEEE Internet Comput., vol. 25, no. 3, pp. 44–51, May 2021, doi: 10.1109/MIC.2020.3037151.

E. Tallyn, H. Fried, R. Gianni, A. Isard, and C. Speed, “The Ethnobot,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, New York, NY, USA: ACM, Apr. 2018, pp. 1–13. doi: 10.1145/3173574.3174178.

I. Celino and G. Re Calegari, “Submitting surveys via a conversational interface: An evaluation of user acceptance and approach effectiveness,” Int. J. Hum. Comput. Stud., vol. 139, p. 102410, Jul. 2020, doi: 10.1016/j.ijhcs.2020.102410.

L. F. Bouchard, L. Peters, and T. AI, Building LLMs for Production: Enhancing LLM Abilities and Reliability with Prompting, Fine-tuning, and RAG. Towards AI, 2024. [Online]. Available: https://books.google.co.id/books?id=siLP0AEACAAJ