.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/plot_06_most_common_errors.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_plot_06_most_common_errors.py: .. _most_common_errors_example: Exploring common errors in a transcription ========================================== It is often interesting to not only look at the rate of errors but also at what types of errors a model makes. Such exploration can reveal patterns in errors which can, for example, give insights into how to improve the performance of the model. This example will show how to use Stringalign to quickly get an overview of the top 10 most common errors in a transcription. We start by loading some toy data based on an excerpt from a digitized copy of `Lærebog i de forskjellige Grene af Huusholdningen `_ by Hanna Winsnes (1846). .. GENERATED FROM PYTHON SOURCE LINES 11-27 .. code-block:: Python from pathlib import Path import stringalign from stringalign.visualize import HtmlString, create_html_image data_path = Path("synthetic_transcription_data") predictions_path = data_path / "predicted.txt" reference_path = data_path / "reference.txt" predictions = predictions_path.read_text().splitlines() references = reference_path.read_text().splitlines() image_paths = data_path.glob("line*.jpg") metadata = [{"image_path": data_path / f"line{line_num:02d}.jpg"} for line_num in range(14)] .. GENERATED FROM PYTHON SOURCE LINES 28-29 After loading the data, we can create a :class:`stringalign.evaluate.MultiAlignmentAnalyzer` which aligns all reference/prediction-pairs and makes it easy for us to explore common transcription errors. .. GENERATED FROM PYTHON SOURCE LINES 29-35 .. code-block:: Python analyzer = stringalign.evaluate.MultiAlignmentAnalyzer.from_strings( predictions=predictions, references=references, metadata=metadata, ) .. GENERATED FROM PYTHON SOURCE LINES 36-37 We start by looking at the ``raw`` edit counts, which corresponds to single-token alignment operations. .. GENERATED FROM PYTHON SOURCE LINES 37-42 .. code-block:: Python counts = analyzer.edit_counts["raw"] for operation, count in counts.most_common(10): print(f"{operation:21}: {count}") .. rst-class:: sphx-glr-script-out .. code-block:: none REPLACED 's' -> 'f' : 10 REPLACED 'k' -> 't' : 8 REPLACED 'd' -> 'b' : 6 INSERTED 'a' : 3 REPLACED 'æ' -> 'e' : 3 INSERTED 'r' : 3 REPLACED 'm' -> 'n' : 3 REPLACED 'ø' -> 'o' : 3 DELETED 'n' : 2 REPLACED 'n' -> 'm' : 2 .. GENERATED FROM PYTHON SOURCE LINES 43-46 We see that the most common transcription error is converting ``s`` to ``f``. We also see three ``æ -> a`` replacements and and insertions of ``e``. It can also be useful to consider combined edit operation counts, as that allows us to find common multi-token replacements, insertions and deletions. .. GENERATED FROM PYTHON SOURCE LINES 46-51 .. code-block:: Python counts = analyzer.edit_counts["combined"] for operation, count in counts.most_common(10): print(f"{operation:21}: {count}") .. rst-class:: sphx-glr-script-out .. code-block:: none REPLACED 's' -> 'f' : 8 REPLACED 'd' -> 'b' : 6 REPLACED 'æ' -> 'ae' : 3 REPLACED 'm' -> 'rn' : 3 REPLACED 'ø' -> 'o' : 3 REPLACED 'nn' -> 'm' : 2 REPLACED 'kk' -> 'tt': 2 REPLACED 'sk' -> 'ft': 2 REPLACED 'Q' -> 'D' : 2 REPLACED 'y' -> 'n' : 1 .. GENERATED FROM PYTHON SOURCE LINES 52-59 When we inspect the combined edit operation counts, we see that there are some common two-token replacements, such as ``æ -> ae``, ``rn -> m`` and ``m -> nn``. Visualising the alignments -------------------------- It can also be useful to visualise the data to get a better understanding of why the model makes the mistakes it does. In particular, we can use the :meth:`MultiAlignmentAnalyzer.alignment_operator_index ` method to iterate over the :class:`AlignmentAnalyzer ` instances that contain transcriptions with a specified edit. .. GENERATED FROM PYTHON SOURCE LINES 59-72 .. code-block:: Python most_common_error = counts.most_common(1)[0][0] # We create a long HTML string to display the visualisation in Sphinx. # If you're using Jupyter, then you can use ``IPython.display.display`` in each iteration instead. table_html = "" for line_analyzer in analyzer.alignment_operator_index["combined"][most_common_error]: image_html = create_html_image(line_analyzer.metadata["image_path"]) alignment_html = line_analyzer.visualize(which="combined") table_html += image_html + alignment_html HtmlString(table_html) .. raw:: html
Reference:Predicted:
at have en Potetesqv at have en Potetesqv
æ ae
rn at male dem paa, rn at male dem paa,
d b
a a
d b
et et
s f
parer parer
Reference:Predicted:
Tid; men i en mindre Huusholdning lø Tid; men i en mindre Huusholdning lø
nn m
er det er det
s f
ig i ig i
kk tt
e at e at
Reference:Predicted:
beleilig beleilig
s f
te; thi da te; thi da
k t
an man faae dem rene ved an man faae dem rene ved
Sk Ot
rubning, rubning,
Reference:Predicted:
om om
A U
aret. Man faaer mere Meel af aret. Man faaer mere Meel af
s f
am am
m rn
e e
Q D
vantum vantum
Reference:Predicted:
og og
s f
lippe at lippe at
sk ft
r r
æ ae
lle de lle de
m rn
, og om de , og om de
nn m
e Gnidning i e Gnidning i
kk tt
e er e er
s f
a a
a n
Reference:Predicted:
an an
sk ft
affe en Ting, der affe en Ting, der
s f
jelden bliver brugt mere end en jelden bliver brugt mere end en
G O
ang ang
Reference:Predicted:
man. De t man. De t
ø o
rre meelagtige Potetes give mee rre meelagtige Potetes give mee
s f
t Meel, og det t Meel, og det


.. GENERATED FROM PYTHON SOURCE LINES 73-74 We see that the OCR-model struggles particularly with the long s of the Fraktur typeface, so a natural step to improve performance could be to train a model with more Fraktur text. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 0.059 seconds) .. _sphx_glr_download_auto_examples_plot_06_most_common_errors.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_06_most_common_errors.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_06_most_common_errors.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_06_most_common_errors.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_