Computational Linguistics in the Netherlands Journal
The Computational Linguistics in the Netherlands Journal (CLIN Journal) provides an international forum for the electronic publication of high-quality scholarly articles in all areas of computational linguistics, language and speech technology. All published papers are open access and freely available online. The focus of the Journal is on computational linguistics research on Dutch and its variants. CLIN Journal is linked to the yearly CLIN conference and accepts submissions of full papers based on research presented at the conference. These papers are rigorously reviewed by members of the editorial board and additional expert reviewers, and when accepted are published in a volume of the journal with as guest editor the organizer(s) of the corresponding CLIN conference. ISSN: 2211-4009.
Copyright (c) 2026 Computational Linguistics in the Netherlands Journal
Fine-tuning with Uniform Information Density-based regularization for Dutch language modelling
The uniform information density (UID) hypothesis states that the information within utterances of communication should be evenly distributed for optimal communication. As human beings have the natural tendency to have an even information density within their communication, for lange language models (LLMs) the training elements that impact their information density is still an area of investigation. Previous research has indicated that modifying the (pre)training loss function with regularizers based on information-theoretic principles has had a favorable impact on the general perplexity and information density of generated responses of LLMs. This study investigates the effects of fine-tuning a Dutch pre-trained GPT-2 model using these regularizers on the perplexity and information density of generated responses.
From Orthography to Semantics: Large-Scale Unsupervised Textual Similarity Detection in Historical Greek
Computational methods for detecting textual similarity provide a powerful lens for exploring linguistic patterns, formulaic language, and textual transmission in historical corpora. In this paper, we investigate what insights become possible when such similarity measures are applied across a vast corpus of Greek texts from Antiquity to the Byzantine period. We propose two methods that enable analysis at this scale: orthographic similarity using MinHash-LSH and semantic similarity using transformer-based sentence embeddings. We first validate both approaches on the Database of Byzantine Book Epigrams, which serves as a gold standard for assessing performance, before applying them to a much larger and more heterogeneous corpus. Scaling MinHash-LSH reveals repeated formulae across textual traditions, while clustering semantic embeddings uncovers conceptual and thematic relationships between texts, highlighting recurring motifs and ideas despite orthographic variation. Our findings illustrate how unsupervised methods suited to high-volume data uncover structures and relationships that targeted studies may overlook.
Tailoring LLM-generated image captions to user needs
One of the original motivations for the development of image captioning systems is to make visual content accessible for people who are blind or visually impaired. What seemed like a huge challenge fifteen years ago, has now made it into consumer products: large language models such as ChatGPT are seemingly able to describe images in fluent natural language. But it is still unclear to what extent the generated descriptions actually match user needs. This study investigates the quality of LLM-generated image descriptions in the context of Dutch news articles. We operationalise output quality based on earlier user studies and existing image description guidelines, and present an extensive evaluation protocol that may be used in future research to assess the quality of automatically generated image descriptions.
Exploring Cultural Variations in Moral Judgments with Large Language Models
Large Language Models (LLMs) have shown strong performance across many tasks, but their ability to capture culturally diverse moral values remains unclear. In this paper, we examine whether LLMs mirror variations in moral attitudes reported by the World Values Survey (WVS) and the Pew Research Center’s Global Attitudes Survey (PEW). We compare smaller monolingual and multilingual models (GPT-2, OPT, BLOOMZ, and Qwen) with recent instruction-tuned mod- els (GPT-4o, GPT-4o-mini, Gemma-2-9b-it, and Llama-3.3-70B-Instruct). Using log-probability- based moral justifiability scores, we correlate each model’s outputs with survey data covering a broad set of ethical topics. Our results show that many earlier or smaller models often produce near-zero or negative correlations with human judgments. In contrast, advanced instruction-tuned models achieve substantially higher positive correlations, suggesting they better reflect real-world moral attitudes. We provide a detailed regional analysis revealing that models align better with Western, Educated, Industrialized, Rich, and Democratic (W.E.I.R.D.) nations than with other regions. While scaling model size and using instruction tuning improves alignment with cross-cultural moral norms, challenges remain for certain topics and regions. We discuss these findings in relation to bias analysis, training data diversity, information retrieval implications, and strategies for improving the cultural sensitivity of LLMs.
Analyzing Cancer Patients' Experiences with Embedding-based Topic Modeling and LLMs
This study investigates the use of neural topic modeling and LLMs to uncover meaningful themes from patient storytelling data, with the goal of offering insights that could contribute to more patient-oriented healthcare practices. We analyze a collection of transcribed interviews with cancer patients (132,722 words in 13 interviews). We first evaluate BERTopic and Top2Vec for the purpose of individual interview summarization, by using similar preprocessing, chunking, and clustering configurations to ensure a fair comparison on Keyword Extraction. LLMs (GPT4) are then used for next step topic labeling. Their outputs for a single interview (I0) are rated through a small-scale human evaluation, focusing on coherence, clarity, and relevance. Based on the preliminary results and evaluation, BERTopic shows stronger performance and is selected for further experimentation using three clinically oriented embedding models. We then analyzed the full interview collection with the best model setting. Results show that domain-specific embeddings improved topic precision and interpretability, with BioClinicalBERT producing the most consistent results across transcripts. The global analysis of the full dataset of 13 interviews, using the BioClinicalBERT embedding model, reveals the most dominant topics throughout all 13 interviews, namely "Coordination and Communication in Cancer Care Management" and "Patient Decision-Making in Cancer Treatment Journey''. Although the interviews are machine translations from Dutch to English, and clinical professionals are not involved in this evaluation, the findings suggest that neural topic modeling, particularly BERTopic, can help provide useful feedback to clinicians from patient interviews. This pipeline could support more efficient document navigation and strengthen the role of patients' voices in healthcare workflows. Affiliated resources created in this work will be shared publicly at https://github.com/4dpicture/TM4health including codes on preprocessing and stopword list prepared.
Is She Even Relevant? When BERT Ignores Explicit Gender Cues
Gender bias in large language models has primarily been investigated for English, while languages with grammatical or morphological gender remain comparatively understudied. This paper investigates how and when gender information emerges in a Dutch BERT model trained from scratch, offering one of the first checkpoint-level analyses of bias formation in a Transformer architecture for a language combining overt morphological gender marking and generic forms. By extracting contextual embeddings throughout training, we construct dynamic gender subspaces using linear SVMs to trace when gender becomes linearly encoded and how this encoding evolves over time. Contextual embeddings are often assumed to integrate contextual cues robustly, allowing models to adjust the representation of a word depending on its more local usage. We therefore test whether explicit gender cues in controlled sentence templates (e.g., Zij is een loodgieter (‘She is a plumber’)) can override learned statistical associations (plumber → male). Our findings challenge this assumption: although gender becomes clearly linearly separable around epoch 20 and is distributed across multiple embedding dimensions, the model struggles to update its internal gender representation in light of explicit contextual cues in short sentence templates. Stereotypical gender–profession pairings are predicted far more accurately than anti-stereotypical ones, and generic forms in Dutch systematically default to a male interpretation, even when the context explicitly denotes a female referent. Together, our results seem to indicate that contextualization in the representations learned by our Dutch BERT model is not sufficiently dynamic along the probed gender direction: explicit gender cues are frequently ignored in anti-stereotypical contexts, resulting in persistent male-default behaviour. These findings underscore once more the challenges for contextual modeling and gender (bias) in multilingual NLP.
The CLARIN Knowledge Infrastructure for Linguistic and Language Technology Research
While CLARIN is widely recognized for its technical infrastructure of language resources, tools, and services, its Knowledge Infrastructure is a vital and complementary pillar that supports researchers, educators, and developers in effectively using, sharing, and extending language data and technologies. The CLARIN Knowledge Infrastructure promotes expertise exchange and best practices, and provides training and support across the research community. We present its main components and their relevance for the computational linguistics and digital humanities communities, with a focus on the Dutch and Flemish research landscape. We contextualize this by describing the knowledge infrastructure components in similar research infrastructures for other scientific domains.
Dictionaries, DeepL, and ChatGPT: Comparing Language Tool Effects on L2 English Speaking and Learner Perceptions
In recent years, language learners have increasingly relied on digital tools such as machine translation (MT) and generative artificial intelligence (AI) tools for language support. While the effects of MT tools like DeepL on second language (L2) writing have been relatively well-documented, fewer studies have examined their influence on L2 speaking performance. Research on the impact of generative AI tools such as ChatGPT in this context is even more limited. This multi-method experimental study aims to assess how the use of MT and generative AI tools affects L2 English speaking performance, as well as learners’ experiences preparing oral presentations. 30 university language students were divided into three groups and assigned either dictionaries, DeepL, or ChatGPT for language support when preparing a group presentation. The presentations were recorded, transcribed, and analysed for fluency and lexical complexity. Additionally, participants completed a two-part survey: one part reflecting on their assigned tool during the preparation process, and one evaluating their general experience with all three tools. Results showed that MT users outperformed other groups in fluency-related measures, while AI users demonstrated significantly greater lexical diversity and density than the MT group (p = 0.04). Lecturer evaluations significantly favoured presentations prepared with MT and AI tools over those prepared with dictionaries (p = 0.02). Additionally, participants using notes during the delivery demonstrated a significantly higher lexical diversity (p = 0.025). Survey responses indicated that perceived enjoyment and usefulness varied between groups, with generative AI tools generally rated more positively for idea generation but less so when it came to practising the presentation. Overall, participants favoured MT tools for preparing both oral and written tasks. These findings suggest that while MT tools currently offer the most consistent benefits for L2 speaking performance, generative AI tools hold promise, particularly for idea development and lexical variation.
Fuzzy Semantic Retrieval Strategies for Automated Short-Answer Grading with Large Language Models in Language Learning
Automated assessment of short-answer exercises in language learning faces a fundamental challenge: multiple admissibility means that numerous distinct responses can be equally correct, rendering rule-based evaluation inadequate. This paper investigates how large language models (LLMs) can address this challenge through retrieval-augmented generation (RAG) for binary classification, determining whether a student's response to a given exercise is correct or incorrect. Across 306 experiments spanning nine grammar topics in English, Spanish, and Dutch (1,185 authentic student responses), ten retrieval approaches are evaluated, extending prior work on baselines and exercise-level matching with novel strategies: sentence-level matching, random selection, and semantic similarity methods adapted from fuzzy matching in translation memory systems. Two central findings emerge. First, RAG with semantic similarity proves effective for identifying relevant examples: when optimised per topic, it achieves 89.4% classification accuracy and recall up to 4.3 percentage points higher than rule-based exercise-level matching. Second, an accuracy-recall trade-off governs configuration choice: single-example configurations maximise accuracy (87.8%), while higher shot counts maximise recall (93.0%). These results establish new performance benchmarks for LLM-based short-answer grading in second language acquisition, with actionable guidance: student-facing applications should use low shot counts to optimise accuracy, while teacher-facing systems benefit from higher shot counts to ensure comprehensive error detection.
Transcribing Children’s Speech: ASR Performance and Obtaining Reliable Orthographic Transcriptions
Automatic speech recognition (ASR) has the potential to substantially reduce manual annotation effort in child speech research by generating automatic transcriptions. However, obtaining reliably high-quality ASR transcriptions for child speech remains challenging in low-resource languages due to limited child-specific pre-trained models and highly diverse noise conditions. This study investigates the effectiveness of state-of-the-art ASR models on child speech through two research questions, by evaluating nine ASR models from three model families (Whisper, NVIDIA, and wav2vec2) on two Dutch child speech datasets, JASMIN and DART. Research question 1 examines the performance of ASR-models applied to child speech. The fine-tuned Whisper-medium model achieves the best overall performance, with a WER of 5.54% on JASMIN and 70.37% on DART, showing that the noisy DART data are clearly more challenging. Research question 2 examines to what extent it is possible to select a subset for which reliable orthographic transcriptions can be obtained automatically, without the need for manual verification. We use an utterance-level selection method that compares ASR output with the original read prompt to identify correctly pronounced recordings. Using the proposed selection method, 42.0% [for JASMIN] and 18.1% [for DART] of the utterances can be automatically identified as correctly pronounced with high confidence, resulting in very low error rates on an utterance level (precisions of 98.3% and higher) and reducing the need for manual verification.
Parsing the Dutch C-CLAMP: Unlocking 150 years of written Dutch for syntactic analysis
We present the parsed Gothenburg edition of the Dutch Corpus of Contemporary and Late Modern Periodicals (Dutch C-CLAMP) and the Dutch Verb Construction database derived from this parsed corpus. The Dutch C-CLAMP is a diachronic corpus of Dutch-language periodicals, with material from the 19th and 20th century from Belgium and The Netherlands. Both the parsed corpus and the Dutch Verb Construction database will be made available to other researchers. In the paper we discusse the creation of the parsed corpus and offer a quantitative overview of the result. In the second half of the paper we introduce the Dutch Verb Construction database, a research database to support large scale diachronic investigation of verb constructions, and discuss its extractions and present an evaluation of the database against manually annotated data. We end the paper with a small case study on verb order, exemplifying one type of research the database facilitates.
Evaluating Dutch Speakers and Large Language Models on Standard Dutch: a grammatical Challenge Set based on the Algemene Nederlandse Spraakkunst
This study evaluates the linguistic knowledge of Dutch Large Language Models (LLMs) by introducing a novel challenge set based on the Algemene Nederlandse Spraakkunst (ANS). The ANS is a comprehensive resource of Dutch prescriptive grammar created by linguists. We collect acceptability judgements of Dutch native speakers on our dataset, validating its usability while observing varying degrees of grammatical acceptability on specific syntactic phenomena. We evaluate both transformer-encoder and transformer-decoder Dutch LLMs on this dataset, and we compare their performance against the standard rules of Dutch in our dataset and the speaker ratings. We find that transformer-encoder models exhibit almost perfect accuracy on our dataset, yet sensitivities for specific sentences differ between models and humans, partially due to mismatches between the reference grammar and actual use of Dutch.
Copyright (c) 2025 Computational Linguistics in the Netherlands Journal
Intrinsic evaluation of Mono- and Multilingual Dutch Language Models
Through transfer learning, multilingual language models can produce good results on extrinsic, downstream NLP tasks in low-resource languages despite a lack of abundant training data. In most cases, however, monolingual models still perform better. Using the Dutch SimLex-999 dataset, we intrinsically evaluate several pre-trained monolingual stacked encoder LLMs for Dutch and compare them to several multilingual models that support Dutch, including two with parallel architectures (BERTje and mBERT). We also try to improve these models’ semantic representations by tuning the multilingual models on additional Dutch data. Furthermore, we explore the effect of tuning these models on written versus transcribed spoken data. While we can improve multilingual model performance through fine-tuning, we find that significant amounts of fine-tuning data and compute are required to outscore monolingual models on the intrinsic evaluation metric.
Evaluating Humor Generation in an Improvisational Comedy Setting
While computational humor generation has long been considered a challenging task, recent large language models have significantly improved the quality of generated jokes. Evaluating humor quality is usually difficult, as not only is the exact quality subjective, but delivery also plays a role. Another disparity in evaluation standards between human and computer-generated humor is the difference in writing time between the two. In this study, we evaluate and compare the quality of humor generated by GPT-4 with human-written jokes in an improvisational comedy setting in Dutch. In a live performance setting on national TV, nine different audience suggestions were used across three improvisational comedy games. Three professional comedians each performed their own improvised joke and an AI-generated joke per round, resulting in a total of 54 jokes. The AI-generated jokes were selected in real time from candidate outputs generated by GPT-4 using a few-shot chain-of-thought prompt specific to each game and audience suggestion. An audience of 40 people then rated all jokes on a 4-point scale, resulting in 2,160 ratings. This allows us to compare the difference in quality between AI and human-created jokes delivered by the same comedian for the same audience suggestion. Our results show that audience members preferred human-created jokes 34.6% of the time, AI-generated jokes 29.7% of the time, and rated them equally in 35.7% of cases. Human-created jokes also received a slightly higher average rating (2.67 vs. 2.59), although GPT-4 occasionally produced standout jokes that received high “best joke” votes. These findings suggest that while human improvisation retains a narrow edge in consistency, current large language models can produce competitive humor under real-time constraints.
Fietje: An open, efficient LLM for Dutch
This paper introduces Fietje, a family of small language models (SLMs) specifically designed for the Dutch language. The model is based on Phi 2, an English-centric model of 2.7 billion parameters. Fietje demonstrated competitive results with larger language models upon its release. A core emphasis of this work is transparency and reproducibility: Fietje is fully open-source, with model weights, datasets, training, and evaluation code all publicly accessible. The paper discusses the performance of Fietje and many other models on an extensive evaluation suite of benchmarks on reasoning, sentiment analysis, world knowledge, linguistic acceptability and word sense disambiguation. Evaluation results illustrate the rapid progress in the field of large language models (LLMs), where recent small models outperform older, larger models that were fine-tuned for Dutch. This trend signals an exciting future for Dutch language processing, suggesting that even compact LLMs are becoming increasingly capable. Furthermore, ongoing and future efforts to adapt LLMs to Dutch are poised to enhance these models even further, broadening their applicability and accessibility. Fietje is only an intermediate step in improving accessibility to language technology for users of the Dutch language.
The Riddle Experiment: two groups are trying to solve a Black Story behind a screen, only one group is alive
Investigating the cognitive abilities of large language models (LLMs) can inform theories about both artificial and human intelligence and highlight areas where AI may complement human cognition. This study explores GPT-4’s logical reasoning abilities by comparing its performance in solving Black Story riddles to that of humans. Black Stories are riddles where players reconstruct a hidden narrative by asking yes-or-no questions to a player who knows the full story. These riddles test logical reasoning, creativity, and inference skills of the solvers in an interactive setting. The study utilized a set of 12 existing Black Stories, with deviations in details included. Each Black Story was tested twice in the human and GPT-4 group to minimize individual differences. The experiment was conducted via text messaging to align the testing set-up for the two groups and eliminate potential non-verbal advantages for the human test group. The primary performance indicator was the number of questions needed to solve the riddle, considering the number of given hints to come to the solution. This measure indicated no significant difference between the groups, where both groups managed to arrive at the correct answer eventually. Though GPT-4 was significantly more verbose in questioning than humans, and qualitative results showed that GPT-4 excelled in precise questioning and creativity, but often fixated too much on details. This led to missing the bigger picture and summarizing solutions prematurely. On the other hand, humans covered broader topics and adapted their focus quickly, but had more difficulty figuring out uncommon details. This research suggests that the performance of GPT-4 and humans in solving Black Stories is not significantly different, despite using alternative approaches to achieve results.
Lexical semantic change detection for Ancient Greek: dataset creation and evaluation of a word-embedding-based technique
We create a benchmark for the evaluation of lexical semantic change detection in Ancient Greek and use it to assess the validity of two metrics of lexical semantic change on diachronic embeddings models. Stopponi et al. (2024b) assessed the viability of lexical semantic change detection for Ancient Greek with word2vec models, using two existing measures. However, only a manual evaluation was conducted since a benchmark for the evaluation of this task for Ancient Greek was still missing. We create such a benchmark by extracting cases of semantic change from close-reading studies in Ancient Greek lexical semantics. We also create a parallel benchmark of semantically stable items and assess the effectiveness of the most relevant of the two metrics in distinguishing semantically changed from semantically stable items. Finally, we qualitatively evaluate the candidates for semantic change detected by filtering words by low vector coherence value and high frequency. The results show that the method is effective at retrieving cases of semantic change, especially when coupled with frequency information, but also reinforce the idea that performing lexical semantic change detection on an ancient language and building a robust evaluation benchmark are particularly challenging tasks. In conclusion, we propose a constructive way to leverage this method as a research companion, by integrating it with the close-reading method.
Enriching Historical Records: An OCR and AI-Driven Approach for Database Integration
This research digitizes and analyzes the Leidse hoogleraren en lectoren 1575-1815 books written between 1983 and 1985, which contain biographic data about professors and curators of Leiden University. It addresses the central question: ’How can we design an automated pipeline that integrates OCR, LLM-based interpretation, and database linking to harmonize data from historical document images with existing high-quality database records?’ We applied OCR techniques, generative AI decoding constraints that structure data extraction, and database linkage methods to process typewritten historical records into a digital format. OCR achieved a Character Error Rate (CER) of 1.08% and a Word Error Rate (WER) of 5.06%, while JSON extraction from OCR text achieved an average accuracy of 63% and, based on annotated OCR, 65%. This indicates that generative AI somewhat corrects low OCR performance. Our record linkage algorithm linked annotated JSON files with 94% accuracy and OCR-derived JSON files with 81%. This study contributes to digital humanities research by offering an automated pipeline for interpreting digitized historical documents, addressing challenges like layout variability and terminology differences, and exploring the applicability and strength of an advanced generative AI model.
Word Sense Discrimination with French Transformer Models
This paper investigates unsupervised Word Sense Discrimination using French monolingual transformer models (viz. FlauBERT and CamemBERT), employing clustering and lexical substitution techniques. To investigate approaches that can benefit lower-resource languages, we explore three approaches: (1) clustering contextual embeddings derived through Principal Component Analysis (PCA); (2) a substitute-based method inspired by Amrami and Goldberg (2018), which leverages sparse vectors of model-predicted substitutes; and (3) an enhanced lexical substitution approach adapted from Zhou (2019), designed specifically for BERT-based models and employing embedding dropout to preserve semantic coherence. The evaluation uses two datasets: a manually annotated gold standard comprising 11 homonymous and polysemous target words, and a noisier, augmented corpus sourced from web crawls. Cluster estimation is performed with the Bayesian Information Criterion (BIC), and clustering is conducted using Gaussian Mixture Models (GMM). The gold standard enables comprehensive evaluation across hard-clustering metrics, addressing the lack of consensus on benchmarking Word Sense Discrimination algorithms. Our results show that FlauBERT consistently outperforms CamemBERT on clean datasets, while CamemBERT demonstrates greater robustness to noise. Incorporating Zhou’s (2019) lexical substitution technique yields state-of-the-art performance, particularly in substitute-based methods, but at the cost of significantly higher computational demands and variability due to embedding dropout. These findings highlight the trade-offs between precision and scalability in applying advanced lexical substitution methods.
Assessing Dutch Syllabification Algorithms and Improving Accuracy by Combining Phonetic and Orthographic Information through Deep Learning
Syllabification describes the task of dividing words into syllables. Due to many rules and exceptions, training an algorithm to perform syllabification with high accuracy remains a challenge. Throughout the last decades, different algorithms have been put forth for Dutch syllabification, yet a comprehensive comparative assessment has not been done. Additionally, deep learning has gained significant popularity within NLP in recent years, yet no modern deep-learning based framework has been developed for Dutch orthographic syllabification. Finally, phonetic and orthographic syllabification algorithms have been examined separately, but not in combination. The aim of the current research was twofold: (a) to examine the performance of existing Dutch syllabification algorithms, and (b) to investigate whether combining phonetic and orthographic information into a single model can increase syllabification performance. To compare the performance of algorithms, four algorithms (Brandt Corstius, Liang, Trogkanis-Elkan (CRF), and a newly conceived deep learning model) were applied to three different datasets (dictionary words, loanwords, pseudowords). The algorithms show varying performance across datasets, with the data-driven algorithms outperforming a knowledge-based algorithm in all but one condition. The new deep-learning methods developed led to increased performance compared to the best found in the literature (99.65% word accuracy, a 0.14% improvement). An analysis of the words for which adding phonetic information improved syllabification performance indicates that these were words in which the orthographic ambiguity could be resolved by information on pronunciation. Future research could examine other areas where phonetic information can benefit orthographic processing. In addition, the newly developed deep learning frameworks can be applied to other languages than Dutch.
Exploring the use of pre-trained ASR models for automatic assessment of children’s oral reading
Dutch children’s reading skills have been declining consistently for many years. Oral reading fluency, a combination of decoding skills and word recognition skills, is a fundamental pre-requisite for one’s reading competence. Children’s oral reading fluency is often tested through oral word reading tasks, which are time-consuming to carry out as teachers have to administer the tests in a one-on-one setting, in which they have to indicate the word reading correctness on-the-fly. One possible way of alleviating this workload is to use automatic speech recognition (ASR) to aid in the assessment process. A key concern is that many ASR models struggle with children’s speech. We explored the performance of two pre-trained ASR models: Wav2Vec2.0-CGN and Faster-Whisper-v2. We had them carry out correctness judgement on an oral word reading task, using data from the Children’s Oral Reading Corpus (CHOREC). This corpus contains oral reading data of word lists from native Dutch-speaking primary school children aged 6-12 from Flanders. We compared the results of the ASR models to those of assessors in CHOREC by using the agreement metrics specificity, recall, accuracy, F1-score, and MCC as agreement metrics. We then used two different methods to improve the baseline results, by post-correcting ASR model correctness judgements using manually defined error categories. We found that allowing a deviation from the prompt by one error category obtained the best results for the overall metrics. Faster-Whisper-v2 (accuracy = .89; F1-score = .58; MCC = .54) outperformed Wav2Vec2.0 (accuracy = .70; F1-score = .39; MCC = .38). The MCC values show that both ASR models had mild agreement with assessors. We expected the accuracy levels for both models to be lower than the lowest assessor inter-rater accuracy level (.86), but Faster-Whisper-v2 performed better than expected (.89). However, one should be careful in interpreting this result, since the high accuracy scores are partially due to the imbalanced dataset. We conclude that the performance of standard pre-trained ASR models is promising, but given the current quality of the procedure caution should be exercised in its use. Future research could aim to improve the performance of the whole procedure by e.g. using methods like fine-tuning and validation, and through collaborative research with teachers.
Using GPT-4 for Conventional Metaphor Detection in English News Texts
Metaphor detection presents a significant challenge in natural language processing (NLP) due to the intrinsic complexity of metaphors. In this work, we apply a prompting approach to evaluate GPT-4’s performance on the conventional metaphor identification task. We specifically investigate the effects of prompt variation, output stability, and the role of n-shot prompting. The results indicate that GPT-4’s performance on the metaphor identification task is consistently low across all tested settings, significantly lagging behind the top-performing BERT model. Based on our findings and error analysis, we propose possible approaches for utilizing LLMs and AI assistants in metaphor detection and analysis.
Unlocking Domain Knowledge: Model Adaptation for Non-Normative Dutch
This study examines the adaptation of transformer models to two non-normative Dutch language variants: early modern Dutch and contemporary social media Dutch. Both share linguistic features that set them apart from standard Dutch, including spelling inconsistencies, semantic shifts and out-of-domain vocabulary. To address this, we explore two domain adaptation techniques to adapt models to these language variants: (1) continued full-model pre-training and (2) training specialized adapters integrated into existing models. We evaluate these adaptation techniques on sentiment and emotion detection in early modern Dutch comedies and farces and on emotion and irony detection in Dutch tweets. Our results show that both adaptation methods significantly improve performance on historical and social media Dutch tasks, with the greatest gains occurring when domain-relevant datasets are used. The effectiveness of model adaptation is task-dependent and sensitive to the selection of pre-training data, emphasizing domain relevance over data quantity for optimizing downstream performance. We hypothesize that contemporary Dutch encoder models already capture informal language but lack historical Dutch exposure, making adaptation more impactful for the latter. Additionally, we compare adapted encoder models to generative decoder models, which are state-of-the-art in many NLP tasks. While generative models fail to match the performance of our adapted models for historical Dutch, fine-tuned generative models outperform adapted models on social media Dutch tasks. This suggests that task-specific fine-tuning remains crucial for effective generative modelling. Finally, we release two pre-training corpora for Dutch encoder adaptation and two novel task-specific datasets for early modern Dutch on Hugging Face.
LLMs as chainsaws: evaluating open-weights generative LLMs for extracting fauna and flora from multilingual travelogues
Named Entity Recognition (NER) is crucial in literary-historical research for tasks such as semantic indexing and entity linking. However, historical texts pose challenges for implementing said tasks due to language variations, OCR errors, and poor performance of off-the-shelf annotation tools. Generative Large Language Models (LLMs) present both novel opportunities and challenges in humanities research. These models, while powerful, raise valid concerns regarding biases, hallucinations, and opacity - making their evaluation for the Digital Humanities (DH) community all the more urgent. In response, we propose our work on the evaluation of 3 quantized open-weights LLMs (mistral-7b-instruct-v0.1, nous-hermes-llama2-13b, Meta-Llama-3-8B-instruct) through GPT4ALL for NER on literary-historical travelogues from the 18th to 20th centuries in English, French, Dutch, and German. All models were assessed both quantitatively and qualitatively across 5 incrementally more complex prompts - revealing common error types such as bias, parsing issues, the addition of redundant information, entity adaptations and hallucinations. We analyse prevalent examples per language, century, prompt and model. Our contributions include a publicly accessible annotated dataset, pioneering insights into LLMs’ performance in literary-historical contexts, and the publication of reusable workflows for utilizing and evaluating LLMs in humanities research.
Location-focused translation of flooding events in news articles
We are interested in the automatic extraction of information on flooding events in the Philippines from local news papers. Given that the majority of existing information extraction tools have been developed for English, this study aims to investigate the feasibility of using open-source machine translation (MT) tools to translate Tagalog news items to English. Extra care should be taken when translating location names, as precise location information is indispensable for ef- fective disaster management. We fine-tuned an open-source multi-lingual MT model for disaster news in Tagalog. We investigated several methods to enhance the model performance on location translation and evaluated the different versions to compare the translation quality of locations using a custom location-focused evaluation metric. To this end, two new Tagalog-English datasets specific to the domain were introduced for the purposes of fine-tuning and evaluation. We tested out fine-tuning on domain specific data and two masking techniques using either general masks or database-look-up of names. Contrary to our expectations, our findings show that the base open-source multi-lingual MT model was already proficient in location translation. Our analysis indicates that fine-tuning on domain-specific data improves overall machine translation quality. Our manual analysis provides insight into specific errors of location translation and the unique effects of the fine-tuning techniques.
Gender Bias and the Role of Context in Human Perception and Machine Translation
This paper investigates human gender bias and its relation to bias in machine translation (MT), focussing on the role of context in gender interpretation. To this end, we measured human implicit gender bias and conducted an annotation study, followed by a linguistic and computational analysis to compare human gender perceptions among themselves and with a machine translation system. We created a dataset of 60 gender-ambiguous sentences and collected annotations to understand human gender perceptions and specifically which trigger words in context lead to this perception. The study shows that, unlike the MT system tested in this study, humans exhibit highly varied perceptions of gender in ambiguous contexts. A linguistic analysis on annotated trigger words reveals that proper nouns, nouns and adjectives frequently affect human gender perception.
From ‘nijntje’ to ‘konijn’: New methods for analyzing stress pattern acquisition
In order to gain more insight into stress pattern acquisition by children, we are in need of large datasets and analysis tools that can analyze those data sets. This paper suggests a new way to analyze PhonBank corpora by automatically loading in the annotated data and plotting it in different manners, focusing on stress pattern acquisition. This way, we can more easily track changes in children’s development through time. This method is applied to three longitudinal and two cross-sectional corpora of Dutch, German, and English. The results show that there are some aspects of previously proposed theories that are in need of further investigation. Most notably, past theories have assumed that stages of stress pattern development were the same across different words, while we found some evidence for stages of development per word. This suggests that further research is needed into how exactly these stages differ for different words, and what determines whether a child learns the correct stress pattern for a word sooner or later. Furthermore, comparing our findings to those of previous investigations which used manual analysis methods provides us with the opportunity to discuss the drawbacks and benefits of our analysis tool.
A Knowledge-Based Language Model: Deducing Grammatical Knowledge in a Multi-Agent Language Acquisition Simulation
This paper presents an initial study performed by the MODOMA system. The MODOMA is a computational multi-agent laboratory environment for unsupervised language acquisition experiments such that acquisition is based on the interaction between two language models, an adult and a child agent. Although this framework employs statistical as well as rule-based procedures, the result of language acquisition is a knowledge-based language model, which can be used to generate and parse new utterances of the target language. This system is fully parametrized and researchers can control all aspects of the experiments while the results of language acquisition, that is, the acquired grammatical knowledge, are explicitly represented and can be consulted. Thus, this system introduces novel possibilities for conducting computational language acquisition experiments. The experiments presented by this paper demonstrate that functional and content categories can be acquired and represented by the daughter agent based on training and test data containing different amounts of exemplars generated by the adult agent. Interestingly, similar patterns, which are well established for human-generated data, are also found for these machine-generated data. As the procedures resulted in the successful acquisition of discrete grammatical categories by the child agent, these experiments substantiate the validity of the MODOMA approach to modelling language acquisition.
On Dialect Typicality and Transitional Dialects
Since the late 19th century, dialectologists have already noticed that dialect areas do not have abrupt borders, but gradual transitions from one zone to another. One could speak of ‘focal’ or ‘core’ vs. ‘transitional’ areas for relatively homogenous areas vs. regions with linguistic variants from different neighbours. Dialectometry, the computational branch of dialectology, offers tools to capture and visualise dialect transitions. These techniques can show the global transition patterns, but not for which features, and in what way are they contributing to the transition. On the other hand, detailed studies on individual (often singleton) features in variationist sociolinguistics offer us insights to the details in the way dialects transition from one area to another. But the limit of close examination is the possible over-generalisation of the pattern of one feature over many other features in the dialects. In addition, it is not possible for humans to analyse every single variable in the data manually. In this paper, a new approach is proposed to explore dialect transitions on the feature level, namely the dialect typicality decay analysis. This novel approach builds on previous approaches in dialectometry (automatic dialect classification and feature extraction), and it explores how transitional (and prototypical) dialects possess characteristic features of a particular dialect group. Two main issues are explored using dialect typicality analysis: 1) what does dialect transition look like based on their top characteristic features? and 2) do transitional dialects favour adopting the most typical features from a certain dialect group?