Thinking Academic Evaluation with an AI Impact Agent. First experiences with AI support in the review of research impact case studies

StatusVoR
Simple item page
dc.abstract.enThis paper examines the implications of using artificial intelligence (AI) to support the review and rating of research impact statements in the Polish Evaluation of Quality of Scientific Activity. Under the “third criterion,” impact is assessed through case-study style descriptions that link research to effects beyond academia. These are reviewed under a highly formalised workflow which reduces room for nuance while increasing the burden of compliance work for authors, managers, and evaluators. To explore what AI can and cannot do in this setting, we developed an instruction-driven evaluation agent within ChatGPT and iteratively refined its prompt to follow the evaluation logic, enforce conservative evidence handling, and generate document-grounded justifications. In a pilot involving a sample of 31 impact case studies, one author completed independent human assessments, while the agent generated scores and justifications against the same operationalised criteria. We compare the agent’s outputs with the human assessment, focusing on where the agent adds value and where it fails. The agent performed best on auditable, document-bound tasks. Failure modes included rigid, keyword-driven interpretations in judgement-intensive components, a factual error (a date) and inconsistent disclosure of file-reading limitations. We argue these patterns reflect the documentary, compliance-oriented logic of the evaluation system: LLM agents can strengthen consistency and auditability, yet risk reproducing formalism and under-rewarding contextual judgement unless embedded in a human-in-the-loop workflow. We conclude by positioning LLM-based agents as augmentation rather than replacement in evaluation: useful as conservative compliance and consistency auditors but requiring human oversight where interpretation and contextual judgement are central.
dc.affiliationINH
dc.affiliationWydział Nauk Humanistycznych w Warszawie
dc.affiliationInstytut Nauk Humanistycznych
dc.contributor.authorWróblewska, Marta
dc.contributor.authorWarzecha, Adam
dc.date.access2026-11-02
dc.date.accessioned2026-11-09T09:39:57Z
dc.date.available2026-11-09T09:39:57Z
dc.date.created2026
dc.date.issued2026-11-02
dc.description.abstract<jats:p>This paper examines the implications of using artificial intelligence (AI) to support the review and rating of research impact statements in the Polish Evaluation of Quality of Scientific Activity. Under the “third criterion,” impact is assessed through case-study style descriptions that link research to effects beyond academia. These are reviewed under a highly formalised workflow which reduces room for nuance while increasing the burden of compliance work for authors, managers, and evaluators. To explore what AI can and cannot do in this setting, we developed an instruction-driven evaluation agent within ChatGPT and iteratively refined its prompt to follow the evaluation logic, enforce conservative evidence handling, and generate document-grounded justifications. In a pilot involving a sample of 31 impact case studies, one author completed independent human assessments, while the agent generated scores and justifications against the same operationalised criteria. We compare the agent’s outputs with the human assessment, focusing on where the agent adds value and where it fails. The agent performed best on auditable, document-bound tasks. Failure modes included rigid, keyword-driven interpretations in judgement-intensive components, a factual error (a date) and inconsistent disclosure of file-reading limitations. We argue these patterns reflect the documentary, compliance-oriented logic of the evaluation system: LLM agents can strengthen consistency and auditability, yet risk reproducing formalism and under-rewarding contextual judgement unless embedded in a human-in-the-loop workflow. We conclude by positioning LLM-based agents as augmentation rather than replacement in evaluation: useful as conservative compliance and consistency auditors but requiring human oversight where interpretation and contextual judgement are central.</jats:p>
dc.description.accesstimeat_publication
dc.description.issue2 (68)
dc.description.physical347-368
dc.description.sdgPeaceJusticeAndStrongInstitutions
dc.description.sdgIndustryInnovationAndInfrastructure
dc.description.sdgDecentWorkAndEconomicGrowth
dc.description.sdgQualityEducation
dc.description.versionfinal_published
dc.identifier.doi10.4467/20843860PK.26.021.24050
dc.identifier.eissn2084-3860
dc.identifier.issn1895-975X
dc.identifier.urihttps://share.swps.edu.pl/handle/swps/2736
dc.identifier.weblinkhttps://ejournals.eu/czasopismo/przeglad-kulturoznawczy/artykul/thinking-academic-evaluation-with-an-ai-impact-agent-first-experiences-with-ai-support-in-the-review-of-research-impact-case-studies
dc.languageen
dc.pbn.affiliationliteraturoznawstwo
dc.pbn.affiliationjęzykoznawstwo
dc.rightsCC-BY
dc.rights.questionYes_rights
dc.share.articleOPEN_JOURNAL
dc.subject.enresearch impact
dc.subject.enresearch impact evaluation
dc.subject.enacademic evaluation
dc.subject.enlarge language models
dc.subject.enChatGPT
dc.subject.enPOL-on
dc.swps.sciencecloudsend
dc.titleThinking Academic Evaluation with an AI Impact Agent. First experiences with AI support in the review of research impact case studies
dc.title.journalPrzegląd Kulturoznawczy
dc.typeJournalArticle
dspace.entity.typeArticle