Thinking Academic Evaluation with an AI Impact Agent. First experiences with AI support in the review of research impact case studies

Wróblewska, Marta; Warzecha, Adam

doi:10.4467/20843860PK.26.021.24050

Thinking Academic Evaluation with an AI Impact Agent. First experiences with AI support in the review of research impact case studies

StatusVoR

Dublin Core metadata

Authors

Wróblewska, Marta

Warzecha, Adam

Date

2026-11-02

Journal title

Przegląd Kulturoznawczy

Issue

2 (68)

Pages

347-368

DOI

10.4467/20843860PK.26.021.24050

ISSN

1895-975X

Weblink

https://ejournals.eu/czasopismo/przeglad-kulturoznawczy/artykul/thinking-academic-evaluation-with-an-ai-impact-agent-first-experiences-with-ai-support-in-the-review-of-research-impact-case-studies

Access date

2026-11-02

Abstract EN

This paper examines the implications of using artificial intelligence (AI) to support the review and rating of research impact statements in the Polish Evaluation of Quality of Scientific Activity. Under the “third criterion,” impact is assessed through case-study style descriptions that link research to effects beyond academia. These are reviewed under a highly formalised workflow which reduces room for nuance while increasing the burden of compliance work for authors, managers, and evaluators. To explore what AI can and cannot do in this setting, we developed an instruction-driven evaluation agent within ChatGPT and iteratively refined its prompt to follow the evaluation logic, enforce conservative evidence handling, and generate document-grounded justifications. In a pilot involving a sample of 31 impact case studies, one author completed independent human assessments, while the agent generated scores and justifications against the same operationalised criteria. We compare the agent’s outputs with the human assessment, focusing on where the agent adds value and where it fails. The agent performed best on auditable, document-bound tasks. Failure modes included rigid, keyword-driven interpretations in judgement-intensive components, a factual error (a date) and inconsistent disclosure of file-reading limitations. We argue these patterns reflect the documentary, compliance-oriented logic of the evaluation system: LLM agents can strengthen consistency and auditability, yet risk reproducing formalism and under-rewarding contextual judgement unless embedded in a human-in-the-loop workflow. We conclude by positioning LLM-based agents as augmentation rather than replacement in evaluation: useful as conservative compliance and consistency auditors but requiring human oversight where interpretation and contextual judgement are central.