Gold Standard Charter Corpora: News from and for MGH
Gold standard charter corpora are rare – Nicolas Perraux’s CEMA is still the great promise everybody in the world of digital diplomatics is waiting for. While we look forward to this, Pia Geißel’s list of charter corpora can serve as a starting point for anybody seeking charter collections with human-verified transcriptions. Any addition to the corpus is much appreciated, and all digital diplomatists and medievalists are glad to hear that Monumenta Germaniae Historica (MGH) has made their texts available as open access in plain TEI recently. The Latin Text Archive has already incorporated it. MGH is of high quality, so the openMGH is expected to adhere to the same standards. Many other charter corpora do not meet them: CEMA links lots of scanned charter editions with dirty OCR, and Monasterium.net uses Google OCRed texts. OCR of these, in fact, can include errors that make parts of the text incomprehensible. They can only be called “dirty OCR”, as the usual target for high-quality OCR from printed texts is an error rate with less than one error per thousand characters. Anybody planning to use this dirty OCR would like to have some tools that help to identify OCR errors and improve text quality. Literature promising this is around (Englmair et al. 2022, Lyu et al. 2021, Nguyen 2021).
One proposal is the detection of “unexpected” words by checking the raw OCR texts of charter editions with a large language model. To evaluate this method for the DiDip project, Tamás Kovács developed a small pipeline. It compares the raw OCR texts to a XLM-RoBERTa based model fine-tuned with our current charter corpus (mostly from Monasterium.net) to detect unexpected words. The script then queries the full Wiktionary to check if the respective Latin word exists or is at least mentioned somewhere. We thought, testing it on openMGH would be interesting: MGH is of high quality (we said so): how many non-standard-vocabulary words would show up?
We downloaded the charters available on the openMGH platform and put them through our pipeline. As a result, we got a list of 1372 words, e.g. spreket, ἐπαινετῆ, hedificata, Adikenhusun. Are they interesting from an OCR quality perspective? Not that much, as German words, Greek words, medieval orthography, and proper names can be expected. The easy learning from this is: 1. the pipeline is not good for proper nouns. On this we have to work anyway, as NER in medieval charters is a core task for distant diplomatics (see the promising results by Torres Anguilar 2017, Torres Anguilar et al. 2021, 2022, Chastang et al. 2021). 2. The pipeline also fails with switching languages. For this we’re working on a language detection model (the preliminary versions are available on huggingface.io: langdetect and langdetect_v01). But if we exclude all capitalised words and go through the list of the remainig 721 words manually, there is indivichie, futuromm, advieem. They are suspicious (see the full list of more or less suspicious words attached). Shouldn’t it rather be: individue, futurorum, advicem? Obviously, we have standard patterns of OCR misreadings here: du=chi, ru=m, c=e. Let’s repeat: MGH has high quality. If these words are really OCR errors, we would have a word error rate of 0,033% in the full corpus — not much, maybe one error per ~1500 characters if we take an estimate of two wrong characters per word error. Nevertheless, @MGH, you still might want to rise your standards?
We are looking forward to this!
Georg Vogeler, Tamás Kovács, Patrick Sahle
Chastang, Pierre, Sergio Torres Aguilar, und Xavier Tannier. „A Named Entity Recognition Model for Medieval Latin Charters“. Digital Humanities Quarterly 15, Nr. 4 (2021). http://www.digitalhumanities.org/dhq/vol/15/4/000574/000574.html.
Englmeier, Tobias, Florian Fink, Uwe Springmann, and Klaus U. Schulz. ‘Optimizing the Training of Models for Automated Post-Correction of Arbitrary OCR-Ed Historical Texts’. Journal for Language Technology and Computational Linguistics 35, no. 1 (3 December 2022): 1–27. https://doi.org/10.21248/jlcl.35.2022.232.
Lyu, Lijun, Maria Koutraki, Martin Krickl, and Besnik Fetahu. ‘Neural OCR Post-Hoc Correction of Historical Corpora’. Transactions of the Association for Computational Linguistics 9 (4 May 2021): 479–93. https://doi.org/10.1162/tacl_a_00379.
Nguyen, Thi Tuyet Hai, Adam Jatowt, Mickael Coustaty, and Antoine Doucet. ‘Survey of Post-OCR Processing Approaches’. ACM Computing Surveys 54, no. 6 (July 2021): 1–37. https://doi.org/10.1145/3453476.
Sahle, Patrick, Bernhard Assmann. ‚Digital ist besser‘. Die Monumenta Germaniae Historica mit den dMGH auf dem Weg in die Zukunft – eine Momentaufnahme. Schriften des Instituts für Dokumentologie und Editorik 1. Hg. von den Mitgliedern des Instituts für Dokumentologie und Editorik. Norderstedt 2008. urn:nbn:de:hbz:38-23179
Torres Aguilar, Sergio. „La reconnaissance des entités nommées dans les bases numériques de chartes médiévales en latin: Le cas du Corpus Burgundiae Medii Aevi (Xe–XIIIe siècle)“. Médiévales. Langues, Textes, Histoire, Nr. 73 (2017): 47–65. https://doi.org/10.4000/medievales.8182.
Torres Aguilar, Sergio. „Multilingual Named Entity Recognition for Medieval Charters Using Stacked Embeddings and BERT-Based Models“. In Proceedings of the LREC 2022 Second Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2022), 119–28. Marseille: European Language Resources Association, 2022. https://hal.archives-ouvertes.fr/hal-03703239.
Torres Aguilar, Sergio, und Dominique Stutzmann. „Named Entity Recognition for French medieval charters“. In Proceedings of the Workshop on Natural Language Processing for Digital Humanities, 37–46. Silchar: NLP Association of India (NLPAI), 2021. https://hal.archives-ouvertes.fr/hal-03503055.
indivichie inantea fienda vivificae piacitandi munnitatem capiuntu deeem peniA anneme voeato eandam dienind futuromm advieem invamine nomiae inpe dicionebu adiiciam eedem finnavit inmitanda atinent rocham datii ettam actinent edifitia cuntosque addicata mottam cuntos peblem excadentiam exient eiici medietadem adoamenti ammini evedenciam ottentu onnem podii tonnine dacitam monakale nznik exequtio ndreka indiccien diviina onmi invadiavit attemtata matinam dicia mimificentiam temmenta rikizam confirmoit iamdictae cuntatione attinencia danpna fioda anevaen thiende fiodi niene monificenciam atinencia kniif viif dimidiaet iandicti actemptet nunnccupatam advotiam ocia onoch fatiente individe vivifice aiacet ooacti aecepit vidam pedatica dommicę wanc cumuata cuntasc divorun cuttoni tnctam