The State of Replication at IEEE ISMAR and IEEE VR: A Scoping Literature Review (2010-2024) and Online Survey
Mohammed Safayet Arefin, Verena Biener, S M Rashidul Hasan Nijhum, Florian Weidner, Jens Grubert, and J. Edward Swan II. The State of Replication at IEEE ISMAR and IEEE VR: A Scoping Literature Review (2010-2024) and Online Survey. In IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, October 2025. in press.
Download
(unavailable)
Abstract
A replication attempts to confirm outcomes of earlier research, which is critical in validating and generalizing scientific findings. Yet, its prevalence and practices remain underexplored in Augmented and Virtual Reality (AR/VR) research. To address this, we present a scoping literature review of replication studies within IEEE ISMAR and IEEE VR, spanning 15 years from 2010 to 2024. Our analysis revealed that replication in AR/VR research is rare. Of 2167 total papers reviewed, less than 4% were identified as replication studies. Of these, conceptual replications were the predominant type (57%). Most of those were studies in VR (67%) and within-subject designs (66%). Complementing the literature survey results, we conducted an online survey with 61 participants about their experiences with replication studies and found that 39% of them had conducted a replication study. However, limited resources and external motivation hamper the execution of replication studies among these AR/VR researchers. Combining the findings from our literature review and online survey, we discuss the current state of replication research and the factors contributing to its infrequency. We provide recommendations to improve AR/VR research replication practices, focusing on research culture and reporting, and discuss ongoing challenges.
BibTeX
@InProceedings{ISMAR25,
author = {Mohammed Safayet Arefin and Verena Biener and
S M Rashidul Hasan Nijhum and Florian Weidner and
Jens Grubert and J. Edward {Swan~II}},
title = {The State of Replication at IEEE ISMAR and IEEE VR: A Scoping
Literature Review (2010-2024) and Online Survey},
booktitle = {IEEE International Symposium on Mixed and Augmented Reality
(ISMAR)},
year = 2025,
location = {Daejeon, South Korea},
publisher = {IEEE},
date = {October 8--12},
month = {October},
note = {in press.},
abstract = {
A replication attempts to confirm outcomes of earlier research, which is
critical in validating and generalizing scientific findings. Yet, its prevalence
and practices remain underexplored in Augmented and Virtual Reality (AR/VR)
research. To address this, we present a scoping literature review of replication
studies within IEEE ISMAR and IEEE VR, spanning 15 years from 2010 to 2024. Our
analysis revealed that replication in AR/VR research is rare. Of 2167 total
papers reviewed, less than 4% were identified as replication studies. Of these,
conceptual replications were the predominant type (57%). Most of those were
studies in VR (67%) and within-subject designs (66%). Complementing the
literature survey results, we conducted an online survey with 61 participants
about their experiences with replication studies and found that 39% of them had
conducted a replication study. However, limited resources and external
motivation hamper the execution of replication studies among these AR/VR
researchers. Combining the findings from our literature review and online
survey, we discuss the current state of replication research and the factors
contributing to its infrequency. We provide recommendations to improve AR/VR
research replication practices, focusing on research culture and reporting, and
discuss ongoing challenges.
},
}