Keywords
How to Cite
Abstract
This study tests whether equivalent instructions alter the linguistic features of writing generated at Common European Framework of Reference for Languages (CEFR) levels. Four instruction-tuned language models each generated responses at four CEFR levels across ten writing tasks, five equivalent instruction wordings, and five repetitions per condition (N = 4,000 texts). Twenty prespecified linguistic measures described vocabulary, grammar, discourse, and stance in each generated text. Requested level accounted for 27.8% of variation in these measures, whereas wording accounted for 0.1%; dispersion did not differ significantly across wordings. Classifier accuracy changed by at most 0.6 percentage points for an unseen wording, compared with an 11–16-point loss for an unseen task. Task effects remained large after adjustment for response length. Within these five paraphrases, task choice mattered more than instruction wording, supporting wording robustness under the tested conditions without establishing that generated texts validly represent human proficiency.
References
Aher, G. V., Arriaga, R. I., & Kalai, A. T. (2023). Using large language models to simulate multiple humans and replicate human subject studies. In Proceedings of the 40th International Conference on Machine Learning (Vol. 202, pp. 337–371). PMLR. https://proceedings.mlr.press/v202/aher23a.html
Alexopoulou, T., Michel, M., Murakami, A., & Meurers, D. (2017). Task effects on linguistic complexity and accuracy: A large-scale learner corpus analysis employing natural language processing techniques. Language Learning, 67(S1), 180–208. https://doi.org/10.1111/lang.12232
Anderson, M. J. (2001). A new method for non-parametric multivariate analysis of variance. Austral Ecology, 26(1), 32–46. https://doi.org/10.1111/j.1442-9993.2001.01070.pp.x
Anderson, M. J. (2006). Distance-based tests for homogeneity of multivariate dispersions. Biometrics, 62(1), 245–253. https://doi.org/10.1111/j.1541-0420.2005.00440.x
Anthropic. (2026). Claude Sonnet 5.5 [Large language model]. https://www.anthropic.com
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337–351. https://doi.org/10.1017/pan.2023.2
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024). Synthetic replacements for human survey data? The perils of large language models. Political Analysis, 32(4), 401–416. https://doi.org/10.1017/pan.2024.5
Carlo, M., & Takeuchi, O. (2026a). Do large language models encode second-language writing proficiency? A CALF-based perspective. Digital Studies in Language and Literature, 3(1), 97–142. https://doi.org/10.1515/dsll-2025-0019
Carlo, M., & Takeuchi, O. (2026b). Form without function? Register sensitivity in CEFR-conditioned LLM and second-language learner writing. International Journal of Applied Linguistics. Advance online publication. https://doi.org/10.1111/ijal.70284
Carlo, M., & Takeuchi, O. (2026c). Proficiency without interaction: Comparing stance and engagement in LLM-generated and human CEFR writing. International Journal of Applied Linguistics. Advance online publication. https://doi.org/10.1111/ijal.70353
Council of Europe. (2020). Common European Framework of Reference for Languages: Learning, teaching, assessment – Companion volume. Council of Europe Publishing. https://rm.coe.int/16809ea0d4
Explosion. (n.d.). English trained pipelines [Computer software documentation]. https://spacy.io/models/en
Freedman, D., & Lane, D. (1983). A nonstochastic interpretation of reported significance levels. Journal of Business & Economic Statistics, 1(4), 292–298. https://doi.org/10.1080/07350015.1983.10509354
Google. (2026). Gemma 4 26B-A4B instruction-tuned QAT Q4_0 GGUF [Large language model]. Hugging Face. https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-gguf
Hawkins, J. A., & Buttery, P. (2010). Criterial features in learner corpora: Theory and illustrations. English Profile Journal, 1(1), Article e5. https://doi.org/10.1017/S2041536210000103
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). The curious case of neural text degeneration. In International Conference on Learning Representations. https://openreview.net/forum?id=rygGQyrFvH
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000
Kerz, E., Wiechmann, D., Qiao, Y., Tseng, E., & Ströbel, M. (2021). Automated classification of written proficiency levels on the CEFR-scale through complexity contours and RNNs. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications (pp. 199–209). Association for Computational Linguistics. https://aclanthology.org/2021.bea-1.21/
Khushik, G. A., & Huhta, A. (2020). Investigating syntactic complexity in EFL learners’ writing across Common European Framework of Reference levels A1, A2, and B1. Applied Linguistics, 41(4), 506–532. https://doi.org/10.1093/applin/amy064
Khushik, G. A., & Huhta, A. (2022). Syntactic complexity in Finnish-background EFL learners’ writing at CEFR levels A1–B2. European Journal of Applied Linguistics, 10(1), 142–184. https://doi.org/10.1515/eujal-2021-0011
Lee, Y.-J., Parrila, R., & Colenbrander, D. (2026). Adopt, adapt, or reject? Analysing student-AI interaction in university curriculum-aligned writing tasks. Applied Language Sciences, 2, 260102. https://doi.org/10.65553/ALS.260102
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022). Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8086–8098). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.556
Malik, A., Mayhew, S., Piech, C., & Bicknell, K. (2024). From Tarzan to Tolkien: Controlling the language proficiency level of LLMs for content generation. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 15670–15693). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-acl.926
Messick, S. (1994). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning (Research Report No. RR-94-45). Educational Testing Service. https://doi.org/10.1002/j.2333-8504.1994.tb01618.x
Mistral AI. (2025). Ministral 3 14B Instruct 2512 GGUF [Large language model]. Hugging Face. https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512-GGUF
Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., & Stanovsky, G. (2024). State of what art? A call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12, 933–949. https://doi.org/10.1162/tacl_a_00681
Nicholls, D., Caines, A., & Buttery, P. (2024). The Write & Improve Corpus 2024: Error-annotated and CEFR-labelled essays by learners of English. Cambridge University Press & Assessment. https://doi.org/10.17863/CAM.112997
Qwen Team. (2026). Qwen3.5-9B [Large language model]. Hugging Face. https://huggingface.co/Qwen/Qwen3.5-9B
Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2024). Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In International Conference on Learning Representations. https://openreview.net/forum?id=RIu5lyNXjT
Speer, R. (2023). wordfreq (Version 3.1.1) [Computer software]. Python Package Index. https://pypi.org/project/wordfreq/3.1.1/
Vajjala, S., & Rama, T. (2018). Experiments with universal CEFR classification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications (pp. 147–153). Association for Computational Linguistics. https://doi.org/10.18653/v1/W18-0515
Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002
Watson Todd, R. (2026). Using generative AI to revise articles: How much is too much? Applied Language Sciences, 2, 260108. https://doi.org/10.65553/ALS.260108
Webson, A., & Pavlick, E. (2022). Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 2300–2344). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.naacl-main.167
Wisniewski, K. (2017). Empirical learner language and the levels of the Common European Framework of Reference. Language Learning, 67(S1), 232–253. https://doi.org/10.1111/lang.12223
Yamashita, T. (2025). Examining English language learners’ longitudinal development of syntactic complexity across five CEFR levels with a robust measurement design: A mixed-methods approach. Applied Linguistics. Advance online publication. https://doi.org/10.1093/applin/amaf074
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright (c) 2026 Michele Carlo (Author)