Gold Standard Charter Corpora: News from and for MGH

Gold standard charter corpora are rare – Nicolas Perraux’s CEMA is still the great promise everybody in the world of digital diplomatics is waiting for. While we look forward to this, Pia Geißel’s list of charter corpora can serve as a starting point for anybody seeking charter collections with human-verified transcriptions. Any addition to the corpus is much appreciated, and all digital diplomatists and medievalists are glad to hear that Monumenta Germaniae Historica (MGH) has made their texts available as open access in plain TEI recently. The Latin Text Archive has already incorporated it. MGH is of high quality, so the openMGH is expected to adhere to the same standards. Many other charter corpora do not meet them: CEMA links lots of scanned charter editions with dirty OCR, and Monasterium.net uses Google OCRed texts. OCR of these, in fact, can include errors that make parts of the text incomprehensible. They can only be called “dirty OCR”, as the usual target for high-quality OCR from printed texts is an error rate with less than one error per thousand characters. Anybody planning to use this dirty OCR would like to have some tools that help to identify OCR errors and improve text quality. Literature promising this is around (Englmair et al. 2022, Lyu et al. 2021, Nguyen 2021).

One proposal is the detection of “unexpected” words by checking the raw OCR texts of charter editions with a large language model. To evaluate this method for the DiDip project, Tamás Kovács developed a small pipeline. It compares the raw OCR texts to a XLM-RoBERTa based model fine-tuned with our current charter corpus (mostly from Monasterium.net) to detect unexpected words. The script then queries the full Wiktionary to check if the respective Latin word exists or is at least mentioned somewhere. We thought, testing it on openMGH would be interesting: MGH is of high quality (we said so): how many non-standard-vocabulary words would show up?

We downloaded the charters available on the openMGH platform and put them through our pipeline. As a result, we got a list of 1372 words, e.g. spreket, ἐπαινετῆ, hedificata, Adikenhusun. Are they interesting from an OCR quality perspective? Not that much, as German words, Greek words, medieval orthography, and proper names can be expected. The easy learning from this is: 1. the pipeline is not good for proper nouns. On this we have to work anyway, as NER in medieval charters is a core task for distant diplomatics (see the promising results by Torres Anguilar 2017, Torres Anguilar et al. 2021, 2022, Chastang et al. 2021). 2. The pipeline also fails with switching languages. For this we’re working on a language detection model (the preliminary versions are available on huggingface.io: langdetect and langdetect_v01). But if we exclude all capitalised words and go through the list of the remainig 721 words manually, there is indivichie, futuromm, advieem. They are suspicious (see the full list of more or less suspicious words attached). Shouldn’t it rather be: individue, futurorum, advicem? Obviously, we have standard patterns of OCR misreadings here: du=chi, ru=m, c=e. Let’s repeat: MGH has high quality. If these words are really OCR errors, we would have a word error rate of 0,033% in the full corpus — not much, maybe one error per ~1500 characters if we take an estimate of two wrong characters per word error. Nevertheless, @MGH, you still might want to rise your standards?

We are looking forward to this!

Georg Vogeler, Tamás Kovács, Patrick Sahle


References

Chastang, Pierre, Sergio Torres Aguilar, und Xavier Tannier. „A Named Entity Recognition Model for Medieval Latin Charters“. Digital Humanities Quarterly 15, Nr. 4 (2021). http://www.digitalhumanities.org/dhq/vol/15/4/000574/000574.html.

Englmeier, Tobias, Florian Fink, Uwe Springmann, and Klaus U. Schulz. ‘Optimizing the Training of Models for Automated Post-Correction of Arbitrary OCR-Ed Historical Texts’. Journal for Language Technology and Computational Linguistics 35, no. 1 (3 December 2022): 1–27. https://doi.org/10.21248/jlcl.35.2022.232.

Lyu, Lijun, Maria Koutraki, Martin Krickl, and Besnik Fetahu. ‘Neural OCR Post-Hoc Correction of Historical Corpora’. Transactions of the Association for Computational Linguistics 9 (4 May 2021): 479–93. https://doi.org/10.1162/tacl_a_00379.

Nguyen, Thi Tuyet Hai, Adam Jatowt, Mickael Coustaty, and Antoine Doucet. ‘Survey of Post-OCR Processing Approaches’. ACM Computing Surveys 54, no. 6 (July 2021): 1–37. https://doi.org/10.1145/3453476.

Sahle, Patrick, Bernhard Assmann. ‚Digital ist besser‘. Die Monumenta Germaniae Historica mit den dMGH auf dem Weg in die Zukunft – eine Momentaufnahme. Schriften des Instituts für Dokumentologie und Editorik 1. Hg. von den Mitgliedern des Instituts für Dokumentologie und Editorik. Norderstedt 2008. urn:nbn:de:hbz:38-23179

Torres Aguilar, Sergio. „La reconnaissance des entités nommées dans les bases numériques de chartes médiévales en latin: Le cas du Corpus Burgundiae Medii Aevi (Xe–XIIIe siècle)“. Médiévales. Langues, Textes, Histoire, Nr. 73 (2017): 47–65. https://doi.org/10.4000/medievales.8182.

Torres Aguilar, Sergio. „Multilingual Named Entity Recognition for Medieval Charters Using Stacked Embeddings and BERT-Based Models“. In Proceedings of the LREC 2022 Second Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2022), 119–28. Marseille: European Language Resources Association, 2022. https://hal.archives-ouvertes.fr/hal-03703239.

Torres Aguilar, Sergio, und Dominique Stutzmann. „Named Entity Recognition for French medieval charters“. In Proceedings of the Workshop on Natural Language Processing for Digital Humanities, 37–46. Silchar: NLP Association of India (NLPAI), 2021. https://hal.archives-ouvertes.fr/hal-03503055.


The List

indivichie
inantea
fienda
vivificae
piacitandi
munnitatem
capiuntu
deeem
peniA
anneme
voeato
eandam
dienind
futuromm
advieem
invamine
nomiae
inpe
dicionebu
adiiciam
eedem
finnavit
inmitanda
atinent
rocham
datii
ettam
actinent
edifitia
cuntosque
addicata
mottam
cuntos
peblem
excadentiam
exient
eiici
medietadem
adoamenti
ammini
evedenciam
ottentu
onnem
podii
tonnine
dacitam
monakale
nznik
exequtio
ndreka
indiccien
diviina
onmi
invadiavit
attemtata
matinam
dicia
mimificentiam
temmenta
rikizam
confirmoit
iamdictae
cuntatione
attinencia
danpna
fioda
anevaen
thiende
fiodi
niene
monificenciam
atinencia
kniif
viif
dimidiaet
iandicti
actemptet
nunnccupatam
advotiam
ocia
onoch
fatiente
individe
vivifice
aiacet
ooacti
aecepit
vidam
pedatica
dommicę
wanc
cumuata
cuntasc
divorun
cuttoni
tnctam

You may also like...

1 Response

  1. Clemens Radl says:

    When I did my first presentation of the dMGH (around 2005/2006), of course, I also addressed the problem of OCR errors. And the moderator of the panel summarized my paper by saying, “Today I learnt, that the MGH have started to produce errors.” I knew this was sooner or later going to come back to me… 🙂

    Thank you for your efforts with the list of unexpected words and for sharing the methods and the results! A first glance at the list, looks promising. There are a lot of words in there which can be without further checking used in a corpus wide search and replace (“deeem” is definitely an OCR error for “decem”, as is “aecepit” for “accepit”). Other words in the list are almost certainly false positives (“vivificae”/”vivifice”). Another interesting error is “peniA”, which is most probably the end of a page (“peni-“) and the continuation in column “A” on the following page, where the column heading was interpreted as part of the text).

    A list like this from your pipeline might be a good starting place for some basic corrections. Another approach, albeit slower, is to wait for partner projects (such as the above mentioned LTA) to lemmatize the texts and thereby identify all (or most of the) errors, which could in turn be read back into the openMGH (and dMGH). This approach is still in development and so far we haven’t yet been able to put all volumes into the openMGH. So there is still a lot of work to do.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search OpenEdition Search

You will be redirected to OpenEdition Search