Using large language models to analyze political texts through natural language understanding

Large language models (LLMs) offer scalable alternatives to human experts when analyzing political texts for meaning, using natural language understanding (NLU). Qualitative NLU methods relying on human experts are severely limited by cost and scalability. Statistical text-as-data methods are scalable but rely on strong and often unrealistic assumptions. We propose a systematic, scalable, and replicable method that can extend existing qualitative and quantitative approaches by using LLMs to interpret texts meaningfully rather than as mere data. Our ensemble means of LLM-generated estimates of party positions on six key issue dimensions correlate highly with equivalent mean ratings by country specialists. When applied to coalition policy declarations, LLM estimates align more closely with standard models of government formation than hand-coded estimates. We conclude with a discussion of the profound implications of modern LLMs for political text analysis.

Duke Scholars

Author Scott de Marchi Political Science