Can LLMs write good survey questions? We tested different models and prompting techniques across four domains to see how well AIgenerated survey items perform.
AI-based tools, and LLMs in particular, are attracting attention because they promise to tackle long-standing challenges in survey design. Crafting clear, reliable, and unbiased questions is labor-intensive and requires expertise that is not always available in applied settings. LLMs, however, can generate item wordings within seconds, highlight potential sources of misunderstanding, and systematically vary formulations. These capabilities create the potential for faster, more iterative, and more evidence-driven questionnaire development. If validated and used with methodological care, LLMs could substantially expand the toolbox for improving measurement quality in both academic and applied Survey research.
Despite growing interest in LLM tools within industry, published research in this area remains sparse, and little is known about the quality and characteristics of survey items generated by LLMs or the factors influencing their performance. This work provides the first in-depth analysis of LLM-based survey item generation and systematically evaluates how different design choices affect item quality. Figure 1 provides an overview of the workflow for this study. Five LLMs, namely GPT-4o, GPT-4omini, GPT-oss-20B, LLaMA 3.1 8B, and LLaMA 3.1 70B, were used to generate survey items on four substantive domains: work, living conditions, national politics, and recent politics. In addition, three different prompting techniques were compared: zero-shot, role, and chain-ofthought prompting. In the zero-shot prompting approach, the model received a clear description of the task without any examples of the desired output. Role prompting instructed to act as an expert survey methodologist, in addition to the clear task instructions without examples, such as in the zero-shot prompt. Finally, chain-of-thought prompting encouraged the model to think in reasoning steps (see Figure 2).
To assess the quality of the generated Survey items, the Survey Quality Predictor 3.0 (SQP; https://sqp.gesis.org) was used, a tool developed by survey methodologists for estimating the quality of attitudinal survey items based on coding their formal and linguistic characteristics (e.g., question length, domain and concept of the question, social desirability, answer scale format). To code these characteristics, an LLM-assisted procedure was used. The analysis allowed us to evaluate not only Overall quality but also around 60 specific survey item characteristics, offering a detailed view of how LLM-generated questions differ.
The analysis addresses one main research question: Can LLMs generate high-quality survey items, and how is their performance influenced by prompting technique and model choice?
The findings show striking differences in Survey item characteristics across the different models and prompting techniques. Introduction texts, question phrasing, and Response scale formats vary greatly across prompting techniques. This variation is more pronounced for the LLaMA models: their characteristics differ substantially depending on model size and prompt used. GPT models, by contrast, mostly generate similar items regardless of model size and prompting techniques. They mainly generate five-category, symmetric scales and never include a “don’t know” option.
Regarding quality of the items, both the choice of model and the prompting technique employed influence the quality of LLM-generated survey items. Closed-source GPT models generally produce more consistent items than Figure 2: Prompting Techniques open-source LLaMA models: items generated using GPT-4o and GPT-4o-mini are of very similar quality, regardless of whether the larger or smaller model, or which prompt was used. Items generated with LLaMa, in contrast, showed greater variation in quality depending on the model size or prompting technique used. Items generated with GPT-oss-20B were excluded from the analysis as the model failed to generate most of the survey items successfully, i.e., that the model did not produce a complete survey item, the text was cut off, or the output contained multiple survey items.
Overall, chain-of-thought prompting yielded the best results. GPT-4o, GPT-4o-mini, and LLaMA 3.1 70B achieved similar item quality when combined with chain-of-thought prompting, although the LLaMA model showed greater variability.
Regarding the four survey domains for which items were generated (work, living conditions, national politics, and recent politics), items related to “work” and “national politics” reached the highest quality levels across models.
We additionally compared the quality of the generated items to the quality of items in the SQP database. The SQP database (https://sqp.gesis.org/user/database) contains multiple survey items with authorized characteristic codings and corresponding quality estimates. Compared to the SQP items, the LLM-generated items perform similarly; the SQP items’ quality is only slightly higher on average, showing that LLM-generated survey items can reach a level of quality similar to that of survey items currently used in practice.
Master’s thesis which won two awards
The results of this study highlight both the potential and the current limitations of LLMs in supporting questionnaire design. The findings show that LLMs can generate promising Survey items, but they also underscore that the results must be used with caution and evaluated critically. By assessing item quality, this study provides a foundation for future research and contributes to a nuanced understanding of AIsupported survey designs. As the LLM Landscape evolves rapidly and new, more powerful models and prompting techniques continue to emerge, it becomes all the more important to continuously investigate and evaluate the potential of LLMs in the field of survey design.
The study is based on a Master’s thesis which won two awards: the 2026 BVM Young Professionals Award and the GOR Master Thesis Award 2026 from the German Society for Online Research (DGOF). The work has since been developed into a scientific publication. The resulting paper has been accepted for presentation at three conferences and is also available online as a preprint: https://osf.io/preprints/socarxiv/fzn7t_v1.
Anna Fuchs
PhD Candidate at the Social Data
Science and AI Lab (SODA) at the LMU in Munich
Anna Fuchs ist PhD Candidate beim Social Data Science and AI Lab (SODA), LMU München. Die Studie basiert auf einer Masterarbeit, die zwei Preise gewonnen hat, den BVM Young Professionals Award 2026 und den GOR Master Thesis Award 2026 der DGOF.
Download Artikel
Swiss Insights News #26-6
Alle SWISS INSIGHTS News finden Sie hier: SWISS INSIGHTS NEWS
