Dictionaries, DeepL, and ChatGPT: Comparing Language Tool Effects on L2 English Speaking and Learner Perceptions

Abstract

In recent years, language learners have increasingly relied on digital tools such as machine translation (MT) and generative artificial intelligence (AI) tools for language support. While the effects of MT tools like DeepL on second language (L2) writing have been relatively well-documented, fewer studies have examined their influence on L2 speaking performance. Research on the impact of generative AI tools such as ChatGPT in this context is even more limited. This multi-method experimental study aims to assess how the use of MT and generative AI tools affects L2 English speaking performance, as well as learners’ experiences preparing oral presentations. 30 university language students were divided into three groups and assigned either dictionaries, DeepL, or ChatGPT for language support when preparing a group presentation. The presentations were recorded, transcribed, and analysed for fluency and lexical complexity. Additionally, participants completed a two-part survey: one part reflecting on their assigned tool during the preparation process, and one evaluating their general experience with all three tools. Results showed that MT users outperformed other groups in fluency-related measures, while AI users demonstrated significantly greater lexical diversity and density than the MT group (p = 0.04). Lecturer evaluations significantly favoured presentations prepared with MT and AI tools over those prepared with dictionaries (p = 0.02). Additionally, participants using notes during the delivery demonstrated a significantly higher lexical diversity (p = 0.025). Survey responses indicated that perceived enjoyment and usefulness varied between groups, with generative AI tools generally rated more positively for idea generation but less so when it came to practising the presentation. Overall, participants favoured MT tools for preparing both oral and written tasks. These findings suggest that while MT tools currently offer the most consistent benefits for L2 speaking performance, generative AI tools hold promise, particularly for idea development and lexical variation.