Brief Reports in Language Research
Brief Reports in Language
Research, 1, 260104 (2026)
https://doi.org/10.65553/BRLR.260104
Detecting
Study Abroad Effects with an Automated Speaking Test: An Exploratory
Comparison-Group Study
Ken Urano*
Faculty of Business Administration, Hokkai-Gakuen University, Sapporo, Japan
Abstract
Short-term
study abroad programs are expected to demonstrate language gains, but such
gains are small and hard to attribute to the program. This exploratory study
used an automated speaking test, AISATS (AI Skill Assessment and Training
System), and a comparison group to evaluate a three-week, speaking-focused
program in the United States. Twelve Japanese university students who joined
the program and nine who stayed in Japan took the test before and after it.
Group × Time interactions for the total
score and five component scores were estimated as standardized differences in
gains (Cohen’s d) and checked with mixed ANOVAs, analyses of
covariance, and re-analyses using post-test scores produced under the test
provider’s updated rubric. Both groups gained similarly on
the total score, largely because Task Achievement rose by more than one point
in both groups, a change that is unlikely to reflect the program. With Task
Achievement set aside, Fluency was the only component that favored
the study abroad group (d = 0.87, 95% CI [-0.04, 1.77];
adjusted d
= 0.57), although no interaction was statistically
significant. Automated fluency scores may be sensitive to a short,
speaking-focused stay abroad, provided that component
scores are read separately and against a comparison group.
Keywords: short-term study
abroad, automated speaking assessment, oral fluency, retest effect, Japanese
EFL
Universities that run
overseas programs are increasingly expected to show that those programs achieve
their language aims. Because a program runs every year with a new cohort, its
evaluation has to be repeatable and easy to
administer. A convincing account of a program’s effect must also separate it from the other changes that learners
undergo over the same period. Both requirements are hard to meet for short-term
programs, whose gains are usually modest. In a meta-analysis of 31 studies of
Japanese university learners, Hirai (2018) found that gains from stays of
one month or less were much smaller than those from longer stays. Detecting a
small gain and attributing it to the program requires an instrument sensitive
enough to register the change and a design that can separate program effects
from other developments.
Earlier studies that
applied objective speaking measures to Japanese short-term programs share one
limitation. Saito (2025) reported a rise in Versant speaking scores after a one-month program
in Australia, and Sekiya et al. (2018) found large gains on a rated
group-oral test after one-month programs in the United States, most of which
had faded ten months later. Neither study included a comparison group, so the
gains cannot be distinguished from changes that would have occurred without the
program, concurrent coursework, or familiarity with the test. When the expected
effect is small, the parameter of interest is the Group × Time interaction,
that is, how much more one group changes than the other over the same interval.
Human-rated speaking
tests, such as oral proficiency interviews and paired interviews, require
trained raters and individual scheduling, are costly to run for every student
each year, and are open to variation in rater severity and fatigue, which makes
annual repeated use impractical for most programs. Automated speaking tests,
which elicit and score spoken responses without a human rater, are well suited
to repeated evaluation. Their composite scores correlate highly with oral
proficiency interview scores (Bernstein et al., 2010), and some of them, such as
Versant, define speaking as the ability to process and produce speech
efficiently in real time (Van Moere, 2012). Repeated administration, however, invites retest
effects. Hausknecht et al. (2007) found an average
gain of about a quarter of a standard deviation when cognitive ability tests
were retaken, and a test taker who has become familiar with an on-screen
interviewer may respond more readily and at greater length the second time. A
change in the scoring system between administrations is a further possible
source of change unrelated to learning. In Messick’s (1989) terms, such changes
are construct-irrelevant variance. A comparison group that takes the same test
over the same months absorbs them, leaving the interaction to capture the
change attributable to the program.
Whether the composite
or the components are read also matters. Validation studies usually report
results for the composite, and the scoring algorithms are proprietary. If a
construct-irrelevant change falls on a component that carries weight in the
composite, the composite can move for reasons unrelated to the program and hide
a smaller group difference in another component. A component-level analysis is
therefore needed in addition to the comparison group.
Among the components
that such tests report, fluency has the clearest link to intensive second
language use. Automated fluency scores are typically based on speech rate and
pausing, which correspond to the speed and breakdown dimensions of utterance
fluency (Segalowitz, 2010; Tavakoli & Skehan, 2005). These are the
aspects of speaking that study abroad has most consistently been found to
develop. Learners who spent a semester or three months abroad came to speak
faster and with fewer pauses (Mora & Valls-Ferrer, 2012; Segalowitz & Freed, 2004), whereas accuracy
improved only moderately and complexity did not change in the former study, and
a systematic review identified fluency gains as one of the more consistent
findings in this literature (Moore et al., 2021). Llanes and Muñoz (2009) reported measurable fluency and
accuracy gains after only three to four weeks, although their 24 participants,
aged 13 to 22, had no comparison group. A short program that places heavy
demands on speaking might therefore leave its clearest trace on fluency rather
than on knowledge-based components such as vocabulary and grammar.
The present study analyzes automated speaking test data for one cohort of a
three-week program, evaluated against a comparison group, a combination that
has seldom been reported for short-term programs in the Japanese context. The
data were collected for the program’s routine evaluation
rather than for a pre-planned study, and the sample is small, so the analysis
is exploratory. The expectation that Fluency would be the most
program-sensitive component was formed before the analysis, but it was not
preregistered, and no confirmatory test is claimed. The two research questions
below ask, together, whether component scores can detect short-term gains that
the composite conceals:
RQ1: Did the Study Abroad group
improve more than the Comparison group on the total score of the automated
speaking test?
RQ2: Did any component score, and in particular Fluency, detect a larger gain for the Study
Abroad group than the total score did?
Methods
Participants
and Program
The participants were 21
undergraduate business students at a private university in Japan, aged 19 to
21. Twelve (three men and nine women) took part in a three-week overseas
program and formed the Study Abroad group. The other nine (six men and three women)
stayed in Japan and formed the Comparison group. Students chose whether to join
the program, so the groups were not randomly formed. No independent proficiency
measure was available, and baseline equivalence was examined with the pre-test
scores. In the first term (April to July), six of the 12 Study Abroad students
and all nine Comparison students took the same elective task-based English
course, which focused on oral skills. In line
with task-based language teaching principles, the course prioritized meaningful
oral communication and learner-driven language use over isolated structural
drills (Bui, 2026a). No university English
classes were held during the summer break.
The study-abroad program ran from
late August to mid-September at a university in the United States. On weekday
afternoons, the students attended an intensive communicative English course in
small classes built around speaking, discussion, and presentations. They also
visited two local companies, where they gave prepared presentations in English
to business professionals and received feedback. They lived with host families
throughout the stay.
Instrument
Spoken proficiency was measured
with AISATS (Potential Plus, 2025), an automated speaking test that combines the Speechace scoring engine with a browser-based platform in
which an on-screen AI interviewer delivers the test in about 10 to 15 minutes.
The form used here contained two read-aloud items, one picture description, and
two opinion questions. The test reports a total score and five component scores
on a 9-point scale: Pronunciation, Fluency, Vocabulary, Grammar, and Task
Achievement, the last of which concerns the relevance and completeness of
responses. Fluency is computed from speaking rate and the number of pauses. The
algorithms for the other components are not publicly documented. Koizumi and
Nagamine (2025) summarized the
published validity information for the test: scores agreed with experienced
IELTS examiners within 0.5 points for 77% of responses, and the test–retest correlation was
.80. The report does not state whether these findings concern the total score
or the components, or which population and tasks they came from.
Between the two administrations,
the vendor changed the way scores were reported. Pre-test scores were reported
on an IELTS-referenced rubric in steps of 0.5 points (total scores also
included decimals). Post-test performances were scored on a new Speechace-referenced rubric in steps of 0.1 points, and the
vendor also supplied post-test scores converted to the original rubric by an
undisclosed procedure. The main analyses use the converted scores, and the Speechace-referenced scores serve as a sensitivity check.
The two versions of the post-test scores were closely related (r = .91 to .99 across the six
measures).
Procedure
Both groups took the test in May
and again in October as part of the program’s routine evaluation, so the
interval between tests was considerably longer than the program itself. On each
occasion, the students took the test individually within a designated period,
in principle on a computer with a headset and without a proctor. All
participants gave written informed consent for the use of their anonymized
scores in research and publication. They were informed that participation was
voluntary and would not affect their grades.
Analysis
The main parameter was the Group
× Time interaction for each measure, expressed as Cohen’s d: the difference
between the groups’ mean
gains divided by the square root of the unweighted mean of their gain-score
variances, with approximate 95% confidence intervals (normal approximation,
with no small-sample correction). Positive values favor
the Study Abroad group. Effect sizes were interpreted in
light of their confidence intervals rather than against fixed benchmarks
(Plonsky
& Oswald, 2014). With this sample, significance tests have little
power, so the estimates and intervals were treated as the primary result.
Because the analysis is exploratory and covers six measures, no correction for
multiple comparisons was applied, and the components are read as a pattern of
effect sizes.
Three supplementary analyses were
conducted. A 2 ×
2 mixed ANOVA was run for each measure, and within-group change was tested with
paired t tests. Because gain
scores can reflect regression toward the mean, an ANCOVA with the pre-test
score as covariate was also run, and the adjusted group difference was divided
by the pooled post-test standard deviation to give an adjusted d. All analyses were repeated with
the Speechace-referenced post-test scores, that is,
the scores produced under the provider’s updated rubric before the conversion described
above. Analyses were conducted in R (R Core Team, 2025).
Results
No baseline difference was
statistically significant (Welch tests, all ps ≥ .27), but the Study Abroad group
began somewhat lower on Fluency (d = −0.48) and Pronunciation (d = −0.40) and higher on Grammar (d = 0.41). Table 1 reports the
pre- and post-test means, and Table 2 the interaction effect sizes.
Table 1.
Pre- and Post-Test Means (and Standard Deviations)
by Group
|
Measure |
SA Pre |
SA Post |
SA Gain |
CG Pre |
CG Post |
CG Gain |
|
Total |
4.72 (0.63) |
5.21 (0.40) |
+0.48 (0.61) |
4.90 (0.45) |
5.44 (0.46) |
+0.54 (0.40) |
|
Pronunciation |
5.29 (0.58) |
5.54 (0.33) |
+0.25 (0.50) |
5.50 (0.43) |
5.61 (0.22) |
+0.11 (0.42) |
|
Fluency |
5.12 (0.48) |
5.38 (0.31) |
+0.25 (0.40) |
5.33 (0.35) |
5.28 (0.36) |
−0.06 (0.30) |
|
Vocabulary |
5.46 (0.66) |
5.25 (0.34) |
−0.21 (0.66) |
5.39 (0.49) |
5.22 (0.51) |
−0.17 (0.79) |
|
Grammar |
5.58 (0.36) |
5.58 (0.36) |
0.00 (0.43) |
5.44 (0.30) |
5.67 (0.35) |
+0.22 (0.36) |
|
Task Achievement |
2.71 (0.81) |
4.00 (0.85) |
+1.29 (0.84) |
2.83 (1.00) |
4.44 (1.04) |
+1.61 (0.93) |
Note. SA = Study Abroad (n = 12); CG = Comparison (n = 9). Post-test scores are those converted by the test provider to the
pre-test rubric (see Methods). Gains are computed from unrounded means;
standard deviations of gains are in parentheses.
Both groups gained almost the same amount on the total score, and both gains were significant (Study Abroad: t(11) = 2.74, p = .019; Comparison: t(8) = 4.11, p = .003). The main effect of time was large, F(1, 19) = 19.26, p < .001, η2G = .224, whereas the interaction was negligible (d = −0.12). On the total score alone, the program would appear to have had no effect.
The shared gain came mostly from Task Achievement, which rose by more than one scale point in both groups, far more than any other component. Its time effect was F(1, 19) = 56.32, p < .001, η2G = .404, and its interaction slightly favored the Comparison group (d = −0.36).
Table 2.
Group × Time Interaction by
Measure
|
Measure |
d [95% CI] |
F(1, 19) |
p |
Adjusted d |
p |
d
(Speechace) |
|
Total |
−0.12 [−0.98, 0.75] |
0.07 |
.797 |
−0.41 |
.326 |
−0.09 |
|
Pronunciation |
0.30 [−0.57, 1.17] |
0.46 |
.508 |
−0.05 |
.898 |
0.34 |
|
Fluency |
0.87 [−0.04, 1.77] |
3.69 |
.070 |
0.57 |
.149 |
0.73 |
|
Vocabulary |
−0.06 [−0.92, 0.81] |
0.02 |
.896 |
0.06 |
.889 |
−0.16 |
|
Grammar |
−0.56 [−1.44, 0.32] |
1.58 |
.224 |
−0.37 |
.409 |
−0.72 |
|
Task Achievement |
−0.36 [−1.23, 0.51] |
0.68 |
.419 |
−0.40 |
.311 |
−0.38 |
Note. d =
difference in mean gains divided by the SD of the gain scores (the square root
of the unweighted mean of the groups’ gain-score variances); positive values favor the Study Abroad group. F and the first p
are for the interaction term of the mixed ANOVA. Adjusted d and the second p
are from the ANCOVA. d
(Speechace) = gain-score d with Speechace-referenced post-test scores.
Once Task Achievement is set aside, a component-specific pattern emerges (Figure 1). Fluency showed the largest interaction in favor of the Study Abroad group, d = 0.87, 95% CI [−0.04, 1.77]. The Study Abroad group rose by 0.25 points, t(11) = 2.17, p = .053, while the Comparison group changed little, t(8) = −0.55, p = .594. Six of the 12 Study Abroad students gained on Fluency and one declined, whereas one of the nine Comparison students gained and two declined, so the difference was not produced by one or two exceptional learners. Pronunciation moved in the same direction but less strongly. Grammar favored the Comparison group (d = −0.56), and Vocabulary declined slightly in both groups. No interaction reached significance, and every confidence interval included zero.
Note. Filled
circles show gain-score d with
approximate 95% confidence intervals; open triangles show ANCOVA-adjusted d.
Figure 1. Group × Time
Interaction Effect Sizes by Measure
Adjusting for pre-test scores reduced the Fluency difference but did not reverse it (adjusted difference = 0.19 points, 95% CI [−0.07, 0.46], adjusted d = 0.57). The Grammar difference was likewise reduced, and the adjusted Pronunciation difference fell to nearly zero. The same pattern appeared with the Speechace-referenced scores: Task Achievement again rose by more than one point in both groups, and Fluency again showed the largest interaction in favor of the Study Abroad group, d = 0.73, 95% CI [−0.16, 1.62]. Finally, post-test Fluency and Grammar (converted scores) were positively correlated across participants, r = .61, 95% CI [.25, .83], which gives no sign that fluency was gained at the expense of accuracy. Because only half of the Study Abroad students had taken the first-term oral-skills course, the group was also split by participation (six students each). Fluency gains were +0.42 for those who had taken the course and +0.08 for those who had not, against −0.06 in the Comparison group (d = 1.39 and 0.41, respectively), whereas Grammar gains were 0.00 in both subgroups, against +0.22 in the Comparison group. Because every Comparison student had taken the course, participation alone cannot account for the Fluency difference, but the subgroups are too small for inference.
Discussion and conclusion
This exploratory study evaluated
a three-week, speaking-focused overseas program with an automated speaking test
and a comparison group. Two findings stand out. The first is a large Task
Achievement gain shared by both groups. The Comparison group, which received no
English instruction during the summer, gained slightly more than the Study
Abroad group, whereas a program effect would predict the reverse. This change
therefore tells us nothing about the program. Two explanations are available,
and the present data cannot distinguish between
them. Students who had already
encountered the on-screen interviewer may have responded more readily and at
greater length the second time (Hausknecht et al., 2007), and a component
that judges the completeness of responses may be particularly sensitive to
response length. Indeed, recent
exploratory evidence suggests that interacting with an AI interlocutor rather
than a human partner can reduce face-threat and communicative anxiety,
fostering greater willingness to elaborate in East Asian EFL contexts (Bui, 2026b). Alternatively, the
change in the scoring rubric between the administrations may have shifted the
Task Achievement scale, although the rise was equally large before conversion.
Under either explanation the change is construct-irrelevant in Messick’s (1989) sense. Its practical consequence is that one component dominated the
composite. An evaluation based only on the total score would have concluded
that the program had no measurable effect, and a single-group evaluation would
have concluded that the participants had improved substantially. The comparison
group exposed the second error, and the component-level analysis exposed the
first.
The second finding is that Fluency was the only component that separated the groups in the predicted direction. The difference became smaller after adjustment for baseline scores but kept its direction, and it appeared on both post-test score scales. It is consistent with research in which oral fluency is among the most consistently reported gains from study abroad (Mora & Valls-Ferrer, 2012; Moore et al., 2021; Segalowitz & Freed, 2004), and it extends that contrast to a three-week program measured automatically (cf. Llanes & Muñoz, 2009). Theoretically, the pattern fits the view that sustained real-time use of a second language first affects the speed and smoothness of delivery, which a score based on speech rate and pausing captures, whereas knowledge-based components such as Vocabulary and Grammar are less likely to change in three weeks, although accuracy gains have been reported after stays that short (Llanes & Muñoz, 2009). The program kept speaking in constant demand through classes, prepared presentations, and homestay interaction, and the summer, when only the Study Abroad group received English instruction, was the main systematic difference between the groups. Two qualifications apply. Part of the gain-score difference probably reflects regression toward the mean, because the Study Abroad group began lower on Fluency. The Fluency score also aggregates read-aloud and free-response items, although fluency gains are expected mainly in unplanned speech (Tavakoli & Skehan, 2005), so it may dilute a program-related change or register changes in reading speed. The source of the Comparison group’s small Grammar gain is unclear, since the shared first-term course focused on oral skills. The Fluency gain was concentrated among Study Abroad students who had also taken that course (+0.42 versus +0.08), so it may reflect the course and the program together.
Three practical points follow for evaluators who use automated speaking tests in pre–post designs. Component scores should be inspected routinely, because the composite can conceal both a construct-irrelevant change and a component-specific difference. Components that may be sensitive to response length are vulnerable to retest effects, so students should be familiarized with the test format before the baseline administration, and a comparison group should be included wherever possible. The scoring system of a commercial test can also change during an evaluation, so evaluators should record which version and score scale were used at each administration and request documentation of any conversion.
The limitations are substantial, and the results should be
regarded as preliminary. The sample was small, the confidence intervals were
wide, and no interaction was statistically significant, although a design with
12 and 9 participants has little power to detect effects of this size. Group
membership was self-selected, and the groups differed in gender composition and
in baseline Fluency. The testing interval extended well beyond the program and
overlapped the first term, in which all of the
Comparison students but only half of the Study Abroad students took the shared
oral-skills course. The test was taken without a proctor, the equivalence of
the pre- and post-test scales cannot be verified, and the validity of the
component scores for this population has not been established. Future work
should validate the component scores against trained human raters, record the
audio so that fluency can be measured separately for read-aloud and
free-response items, and repeat the design with later cohorts.
In conclusion, an automated fluency score showed a difference favoring the program group that is in line with what a speaking-focused stay abroad would be expected to develop, but that awaits confirmation. Automated speaking tests can support the routine evaluation of short-term programs, provided that component scores are inspected separately, that a comparison group is used to separate program effects from changes common to all test takers, and that changes in the scoring system are documented.
Declaration of AI Use
The author used Claude
(Anthropic) for language editing and for assistance in writing the R scripts.
The author reviewed and verified all text, code, and output and takes full
responsibility for the content.
Bernstein, J., Van Moere, A., & Cheng, J. (2010).
Validating automated speaking tests. Language
Testing, 27(3), 355–377. https://doi.org/10.1177/0265532210364404
Bui, G. (2026a). Task-based language teaching: Principles
and common misunderstandings. Conceptual Foundations in Language Research,
1, 260101. https://doi.org/10.65553/CFLR.260101
Bui, G. (2026b).
Genuine vs. virtual social presence in AI-assisted task-based interaction: An
exploratory study on willingness to communicate and L2 performance across task
complexity. Brief Reports in Language Research, 1, 260103. https://doi.org/10.65553/BRLR.260103
Hausknecht, J. P.,
Halpert, J. A., Di Paolo, N. T., & Moriarty Gerrard, M. O. (2007).
Retesting in selection: A meta-analysis of coaching and practice effects for
tests of cognitive ability. Journal of
Applied Psychology, 92(2),
373–385. https://doi.org/10.1037/0021-9010.92.2.373
Hirai, A. (2018). The effects of study abroad duration
and predeparture proficiency on the L2 proficiency of Japanese university
students: A meta-analysis approach. JLTA
Journal, 21, 102–123. https://doi.org/10.20622/jltajournal.21.0_102
Koizumi, R., & Nagamine, Y. (2025). Gakushū shien ni muketa AI saiten-gata
supīkingu tesuto no katsuyō [Utilizing an AI-scored speaking test for learning
support]. JLTA Journal, 28, 124–143. https://doi.org/10.20622/jltajournal.28.0_124
Llanes, À., & Muñoz, C. (2009). A short stay abroad:
Does it make a difference? System, 37(3), 353–365. https://doi.org/10.1016/j.system.2009.03.001
Moore, I., Torgerson, C., & Beckmann, N. (2021).
Systematic review measuring the efficacy of study abroad in undergraduate
language learners on linguistic proficiency gains. Review of Education, 9,
Article e3306. https://doi.org/10.1002/rev3.3306
Mora, J. C., & Valls-Ferrer, M. (2012). Oral fluency,
accuracy, and complexity in formal instruction and study abroad learning
contexts. TESOL Quarterly, 46(4), 610–641. https://doi.org/10.1002/tesq.34
Plonsky, L., & Oswald, F. L. (2014). How big is
“big”? Interpreting effect sizes in L2 research. Language Learning, 64(4),
878–912. https://doi.org/10.1111/lang.12079
Potential Plus K.K. (2025). AISATS: AI Skill Assessment and Training System [Automated speaking
test]. https://potentialplus.co/ja/aisats/
R Core Team. (2025). R:
A language and environment for statistical computing [Computer software]. R
Foundation for Statistical Computing. https://www.R-project.org/
Saito, Y. (2025). Effects of a short-term study abroad
program on students’ English proficiency and foreign language anxiety. Eigo Eibei Bungaku, 65, 133–153. https://chuo-u.repo.nii.ac.jp/records/2002166
Segalowitz, N. (2010). Cognitive
bases of second language fluency. Routledge. https://doi.org/10.4324/9780203851357
Segalowitz, N., & Freed, B. F. (2004). Context, contact, and
cognition in oral fluency acquisition: Learning Spanish in at home and study
abroad contexts. Studies in Second
Language Acquisition, 26(2),
173–199. https://doi.org/10.1017/S0272263104262027
Sekiya, Y., Park, S., & Tsuji, R. (2018). Effects of
short-term study abroad programs. Studies
in Linguistics and Language Teaching, 29,
161–180. https://doi.org/10.69236/0000001600
Tavakoli, P., & Skehan, P. (2005). Strategic
planning, task structure, and performance testing. In R. Ellis (Ed.), Planning and task performance in a second
language (pp. 239–273). John Benjamins. https://doi.org/10.1075/lllt.11.15tav
Van Moere, A. (2012). A psycholinguistic approach to oral
language assessment. Language Testing,
29(3), 325–344. https://doi.org/10.1177/0265532211424478
Author
bio
Ken Urano
is a Professor in the Faculty of Business Administration at Hokkai-Gakuen University, Japan. His research interests include
second language acquisition, task-based language teaching, English for specific
purposes, and computer-assisted language learning. His recent work examines the
effects of short-term study abroad programs.
Email: urano@hgu.jp
ORCID: https://orcid.org/0000-0003-2550-5227