Abstract
Short-term study abroad programs are expected to demonstrate language gains, but such gains are small and hard to attribute to the program. This exploratory study used an automated speaking test, AISATS, and a comparison group to evaluate a three-week, speaking-focused program in the United States. Twelve Japanese university students who joined the program and nine who stayed in Japan took the test before and after it. Group × Time interactions for the total score and five component scores were estimated as standardized differences in gains (Cohen’s d) and checked with mixed ANOVAs, analyses of covariance, and an alternative post-test score scale. Both groups gained similarly on the total score, largely because Task Achievement rose by more than one scale point in both groups, a change that is unlikely to reflect the program. With Task Achievement set aside, Fluency was the only component that favored the study abroad group (d = 0.87, 95% CI [−0.04, 1.77]; adjusted d = 0.57), although no interaction was statistically significant. An automated fluency score may be sensitive to a short, speaking-focused stay abroad, provided that component scores are read separately and against a comparison group.
References
Bernstein, J., Van Moere, A., & Cheng, J. (2010). Validating automated speaking tests. Language Testing, 27(3), 355–377. https://doi.org/10.1177/0265532210364404
Bui, G. (2026a). Task-based language teaching: Principles and common misunderstandings. Conceptual Foundations in Language Research, 1, 260101. https://doi.org/10.65553/CFLR.260101
Bui, G. (2026b). Genuine vs. virtual social presence in AI-assisted task-based interaction: An exploratory study on willingness to communicate and L2 performance across task complexity. Brief Reports in Language Research, 1, 260103. https://doi.org/10.65553/BRLR.260103
Hausknecht, J. P., Halpert, J. A., Di Paolo, N. T., & Moriarty Gerrard, M. O. (2007). Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology, 92(2), 373–385. https://doi.org/10.1037/0021-9010.92.2.373
Hirai, A. (2018). The effects of study abroad duration and predeparture proficiency on the L2 proficiency of Japanese university students: A meta-analysis approach. JLTA Journal, 21, 102–123. https://doi.org/10.20622/jltajournal.21.0_102
Koizumi, R., & Nagamine, Y. (2025). Gakushū shien ni muketa AI saiten-gata supīkingu tesuto no katsuyō [Utilizing an AI-scored speaking test for learning support]. JLTA Journal, 28, 124–143. https://doi.org/10.20622/jltajournal.28.0_124
Llanes, À., & Muñoz, C. (2009). A short stay abroad: Does it make a difference? System, 37(3), 353–365. https://doi.org/10.1016/j.system.2009.03.001
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). American Council on Education/Macmillan.
Moore, I., Torgerson, C., & Beckmann, N. (2021). Systematic review measuring the efficacy of study abroad in undergraduate language learners on linguistic proficiency gains. Review of Education, 9, Article e3306. https://doi.org/10.1002/rev3.3306
Mora, J. C., & Valls-Ferrer, M. (2012). Oral fluency, accuracy, and complexity in formal instruction and study abroad learning contexts. TESOL Quarterly, 46(4), 610–641. https://doi.org/10.1002/tesq.34
Plonsky, L., & Oswald, F. L. (2014). How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64(4), 878–912. https://doi.org/10.1111/lang.12079
Potential Plus K.K. (2025). AISATS: AI Skill Assessment and Training System [Automated speaking test]. https://potentialplus.co/ja/aisats/
R Core Team. (2025). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/
Saito, Y. (2025). Effects of a short-term study abroad program on students’ English proficiency and foreign language anxiety. Eigo Eibei Bungaku, 65, 133–153. https://chuo-u.repo.nii.ac.jp/records/2002166
Segalowitz, N. (2010). Cognitive bases of second language fluency. Routledge. https://doi.org/10.4324/9780203851357
Segalowitz, N., & Freed, B. F. (2004). Context, contact, and cognition in oral fluency acquisition: Learning Spanish in at home and study abroad contexts. Studies in Second Language Acquisition, 26(2), 173–199. https://doi.org/10.1017/S0272263104262027
Sekiya, Y., Park, S., & Tsuji, R. (2018). Effects of short-term study abroad programs. Studies in Linguistics and Language Teaching, 29, 161–180. https://doi.org/10.69236/0000001600
Tavakoli, P., & Skehan, P. (2005). Strategic planning, task structure, and performance testing. In R. Ellis (Ed.), Planning and task performance in a second language (pp. 239–273). John Benjamins. https://doi.org/10.1075/lllt.11.15tav
Van Moere, A. (2012). A psycholinguistic approach to oral language assessment. Language Testing, 29(3), 325–344. https://doi.org/10.1177/0265532211424478

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright (c) 2026 Ken Urano (Author)