A1410
Title: Thematic coding meets AI
Authors: Mireya Diaz - Case Western Reserve University (United States) [presenting]
Abstract: The performance and utility of different AI agents for coding and theme development as part of the qualitative analytic enterprise were examined. Truthfulness and concordance of direct prompting for thematic coding with and without NVivo codebook generation were assessed across three AI agents (ChatGPT 4.1, ChatGPT 5.2, and Claude-Sonnet 4.6) using three meta-analyses with 14 to 17 studies each. Each agent extracted six to seven themes and two to three subthemes for all meta-analyses. Agreement among the agents was highly variable at the theme level, ranging between 33\% (shared three out of nine themes) to 78\% (shared seven out of nine themes). Greater variability and thus less concordance occurred at the subtheme and code levels, with Claude-Sonnet providing more granular codes. Single information deemed relevant by the agents prevails over the principle of repetition to derive themes. Combining thematic coding results from different agents may provide a richer set of codes and themes than using a single agent. AI agents provide a wide suite of analyses within thematic coding including generation of codebooks for specialized software and code co-occurrence matrices. Thematic coding has great potential when conducting literature synthesis with AI.
