From Toil to Thought: Designing for Strategic Exploration and Responsible AI in Systematic Literature Reviews
arXiv:2603.05514v1 Announce Type: cross Abstract: Systematic Literature Reviews (SLRs) are fundamental to scientific progress, yet the process is hindered by a fragmented tool ecosystem that imposes a high cognitive load. This friction suppresses the iterative, exploratory nature of scholarly work. To investigate these challenges, we conducted an exploratory design study with 20 experienced researchers. This study identified key friction points: 1) the high cognitive load of managing iterative query refinement across multiple databases, 2) the overwhelming scale and pace of publication of modern literature, and 3) the tension between automation and scholarly agency. Informed by these findings, we developed ARC, a design probe that operationalizes solutions for multi-database integration, transparent iterative search, and verifiable AI-assisted screening. A comparative user study with 8 researchers suggests that an integrated environment facilitates a transition in scholarly work, mo
arXiv:2603.05514v1 Announce Type: cross Abstract: Systematic Literature Reviews (SLRs) are fundamental to scientific progress, yet the process is hindered by a fragmented tool ecosystem that imposes a high cognitive load. This friction suppresses the iterative, exploratory nature of scholarly work. To investigate these challenges, we conducted an exploratory design study with 20 experienced researchers. This study identified key friction points: 1) the high cognitive load of managing iterative query refinement across multiple databases, 2) the overwhelming scale and pace of publication of modern literature, and 3) the tension between automation and scholarly agency. Informed by these findings, we developed ARC, a design probe that operationalizes solutions for multi-database integration, transparent iterative search, and verifiable AI-assisted screening. A comparative user study with 8 researchers suggests that an integrated environment facilitates a transition in scholarly work, moving researchers from managing administrative overhead to engaging in strategic exploration. By utilizing external representations to scaffold strategic exploration and transparent AI reasoning, our system supports verifiable judgment, aiming to augment expert contributions from initial creation through long-term maintenance of knowledge synthesis.
Executive Summary
The article discusses the challenges faced by researchers in conducting Systematic Literature Reviews (SLRs) due to a fragmented tool ecosystem, imposing a high cognitive load. An exploratory design study identified key friction points, and a design probe called ARC was developed to address these issues. ARC integrates multiple databases, facilitates transparent iterative search, and provides verifiable AI-assisted screening, enabling researchers to transition from administrative tasks to strategic exploration.
Key Points
- ▸ The current tool ecosystem for SLRs imposes a high cognitive load on researchers
- ▸ ARC design probe addresses key friction points, including multi-database integration and AI-assisted screening
- ▸ The system aims to support verifiable judgment and augment expert contributions in knowledge synthesis
Merits
Integrated Environment
ARC provides an integrated environment that facilitates a transition in scholarly work, moving researchers from managing administrative overhead to engaging in strategic exploration
Transparent AI Reasoning
The system utilizes external representations to scaffold strategic exploration and transparent AI reasoning, supporting verifiable judgment
Demerits
Limited User Study
The comparative user study was conducted with only 8 researchers, which may not be representative of the broader research community
Dependence on AI
The system's reliance on AI-assisted screening may raise concerns about bias and accuracy
Expert Commentary
The article presents a significant contribution to the field of SLRs, highlighting the need for integrated environments that support strategic exploration and transparent AI reasoning. The development of ARC demonstrates the potential for design probes to address key friction points in the research process. However, further research is needed to fully realize the potential of such systems and to address the limitations and concerns raised by the use of AI in research. Ultimately, the success of ARC and similar systems will depend on their ability to balance the benefits of automation with the need for human judgment and agency.
Recommendations
- ✓ Further research is needed to test the efficacy of ARC with a larger and more diverse group of researchers
- ✓ The development of standards and guidelines for AI-assisted research should be a priority to ensure accountability and transparency