Detecting Study Abroad Effects with an Automated Speaking Test: An Exploratory Comparison-Group Study
pdf
html

Keywords

short-term study abroad, automated speaking assessment, oral fluency, retest effect, Japanese EFL

How to Cite

Urano, K. (2026). Detecting Study Abroad Effects with an Automated Speaking Test: An Exploratory Comparison-Group Study. Brief Reports in Language Research, 1(1). https://doi.org/10.65553/BRLR.260104

Abstract

Short-term study abroad programs are expected to demonstrate language gains, but such gains are small and hard to attribute to the program. This exploratory study used an automated speaking test, AISATS, and a comparison group to evaluate a three-week, speaking-focused program in the United States. Twelve Japanese university students who joined the program and nine who stayed in Japan took the test before and after it. Group × Time interactions for the total score and five component scores were estimated as standardized differences in gains (Cohen’s d) and checked with mixed ANOVAs, analyses of covariance, and an alternative post-test score scale. Both groups gained similarly on the total score, largely because Task Achievement rose by more than one scale point in both groups, a change that is unlikely to reflect the program. With Task Achievement set aside, Fluency was the only component that favored the study abroad group (d = 0.87, 95% CI [−0.04, 1.77]; adjusted d = 0.57), although no interaction was statistically significant. An automated fluency score may be sensitive to a short, speaking-focused stay abroad, provided that component scores are read separately and against a comparison group.

pdf
html

References

Bernstein, J., Van Moere, A., & Cheng, J. (2010). Validating automated speaking tests. Language Testing, 27(3), 355–377. https://doi.org/10.1177/0265532210364404

Bui, G. (2026a). Task-based language teaching: Principles and common misunderstandings. Conceptual Foundations in Language Research, 1, 260101. https://doi.org/10.65553/CFLR.260101

Bui, G. (2026b). Genuine vs. virtual social presence in AI-assisted task-based interaction: An exploratory study on willingness to communicate and L2 performance across task complexity. Brief Reports in Language Research, 1, 260103. https://doi.org/10.65553/BRLR.260103

Hausknecht, J. P., Halpert, J. A., Di Paolo, N. T., & Moriarty Gerrard, M. O. (2007). Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology, 92(2), 373–385. https://doi.org/10.1037/0021-9010.92.2.373

Hirai, A. (2018). The effects of study abroad duration and predeparture proficiency on the L2 proficiency of Japanese university students: A meta-analysis approach. JLTA Journal, 21, 102–123. https://doi.org/10.20622/jltajournal.21.0_102

Koizumi, R., & Nagamine, Y. (2025). Gakushū shien ni muketa AI saiten-gata supīkingu tesuto no katsuyō [Utilizing an AI-scored speaking test for learning support]. JLTA Journal, 28, 124–143. https://doi.org/10.20622/jltajournal.28.0_124

Llanes, À., & Muñoz, C. (2009). A short stay abroad: Does it make a difference? System, 37(3), 353–365. https://doi.org/10.1016/j.system.2009.03.001

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). American Council on Education/Macmillan.

Moore, I., Torgerson, C., & Beckmann, N. (2021). Systematic review measuring the efficacy of study abroad in undergraduate language learners on linguistic proficiency gains. Review of Education, 9, Article e3306. https://doi.org/10.1002/rev3.3306

Mora, J. C., & Valls-Ferrer, M. (2012). Oral fluency, accuracy, and complexity in formal instruction and study abroad learning contexts. TESOL Quarterly, 46(4), 610–641. https://doi.org/10.1002/tesq.34

Plonsky, L., & Oswald, F. L. (2014). How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64(4), 878–912. https://doi.org/10.1111/lang.12079

Potential Plus K.K. (2025). AISATS: AI Skill Assessment and Training System [Automated speaking test]. https://potentialplus.co/ja/aisats/

R Core Team. (2025). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/

Saito, Y. (2025). Effects of a short-term study abroad program on students’ English proficiency and foreign language anxiety. Eigo Eibei Bungaku, 65, 133–153. https://chuo-u.repo.nii.ac.jp/records/2002166

Segalowitz, N. (2010). Cognitive bases of second language fluency. Routledge. https://doi.org/10.4324/9780203851357

Segalowitz, N., & Freed, B. F. (2004). Context, contact, and cognition in oral fluency acquisition: Learning Spanish in at home and study abroad contexts. Studies in Second Language Acquisition, 26(2), 173–199. https://doi.org/10.1017/S0272263104262027

Sekiya, Y., Park, S., & Tsuji, R. (2018). Effects of short-term study abroad programs. Studies in Linguistics and Language Teaching, 29, 161–180. https://doi.org/10.69236/0000001600

Tavakoli, P., & Skehan, P. (2005). Strategic planning, task structure, and performance testing. In R. Ellis (Ed.), Planning and task performance in a second language (pp. 239–273). John Benjamins. https://doi.org/10.1075/lllt.11.15tav

Van Moere, A. (2012). A psycholinguistic approach to oral language assessment. Language Testing, 29(3), 325–344. https://doi.org/10.1177/0265532211424478

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Copyright (c) 2026 Ken Urano (Author)

Views: 261Downloads: 80

Downloads

Download data is not yet available.
Creative Commons License All articles published in Brief Reports in Language Research are open access and distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are properly credited. Copyright is retained by the author(s).