.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/plot_01_emoji_ocr_evaluation.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_plot_01_emoji_ocr_evaluation.py: Toy emoji OCR example ===================== This example demonstrates how Stringalign accurately computes evaluation metrics even for complex inputs like emojis, where other tools may return misleading results by default. The default behaviour of, for example, Jiwer is not to cluster based on grapheme clusters, so if we compute the CER for strings with ZWJ-emoji sequences, we can get surprising results .. GENERATED FROM PYTHON SOURCE LINES 11-25 .. code-block:: Python import io import json from pathlib import Path import jiwer import PIL.Image import stringalign jiwer_cer = jiwer.cer("🐈‍⬛", "🐦‍⬛") stringalign_cer, _analyzer = stringalign.evaluate.compute_cer("🐈‍⬛", "🐦‍⬛") print("Jiwer:", jiwer_cer) print("Stringalign:", stringalign_cer) .. rst-class:: sphx-glr-script-out .. code-block:: none Jiwer: 0.3333333333333333 Stringalign: 1.0 .. GENERATED FROM PYTHON SOURCE LINES 26-29 We see that Jiwer gets only 1/3 CER, even though 100% of the characters are wrong. This artificially low error is caused by Jiwer tokenizing (and therefore aligning) based on code points, so 🐈‍⬛ and 🐦‍⬛will be treated as 🐈‍[ZWJ]⬛ and 🐦‍[ZWJ]⬛. Stringalign on the other hand, tokenizes based on grapheme clusters so 🐈‍⬛ and 🐦‍⬛ are correctly treated as two emojis and not six code points. (See :ref:`grapheme_clusters` for an introduction to grapheme clusters). .. GENERATED FROM PYTHON SOURCE LINES 31-33 Lets see how we can use Stringalign to accurately calculate the CER for a synthetic dataset with some toy OCR transcriptions containing emojis. .. GENERATED FROM PYTHON SOURCE LINES 33-49 .. code-block:: Python import io import json from pathlib import Path import PIL.Image import stringalign def load_image(path: Path | str) -> PIL.Image.Image: path = data_path / path return PIL.Image.open(io.BytesIO(path.read_bytes())) data_path = Path("emoji_ocr_evaluation_data") dataset = json.loads((data_path / "lines.json").read_text()) .. GENERATED FROM PYTHON SOURCE LINES 50-52 Look at one sample ------------------ .. GENERATED FROM PYTHON SOURCE LINES 52-57 .. code-block:: Python print(f"Gold standard:\n{dataset['samples'][0]['gold_standard']}\n") print(f"Transcription:\n{dataset['samples'][0]['transcription']}\n") load_image(dataset["samples"][0]["image"]) .. rst-class:: sphx-glr-script-out .. code-block:: none Gold standard: 🪄🐈‍⬛🎃 🐻‍❄️🎿🥶 Transcription: 🪄🐦‍⬛🎃 🐻⛸️🥶 .. GENERATED FROM PYTHON SOURCE LINES 58-60 Evaluate transcriptions ----------------------- .. GENERATED FROM PYTHON SOURCE LINES 60-72 .. code-block:: Python references = [sample["gold_standard"] for sample in dataset["samples"]] predictions = [sample["transcription"] for sample in dataset["samples"]] tokenizer = stringalign.tokenize.GraphemeClusterTokenizer() # This is the default, but it's still nice to be explicit evaluator = stringalign.evaluate.MultiAlignmentAnalyzer.from_strings( references=references, predictions=predictions, tokenizer=tokenizer ) cer = evaluator.confusion_matrix.compute_token_error_rate() print(f"The overall CER is {cer}") .. rst-class:: sphx-glr-script-out .. code-block:: none The overall CER is 0.10619469026548672 .. GENERATED FROM PYTHON SOURCE LINES 73-75 Look at the performance for the different lines ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. GENERATED FROM PYTHON SOURCE LINES 75-84 .. code-block:: Python for alignment_error in evaluator.alignment_analyzers: sample_cer = alignment_error.confusion_matrix.compute_token_error_rate() jiwer_cer = jiwer.cer(alignment_error.reference, alignment_error.predicted) print(f"Reference:\n{alignment_error.reference}\n") print(f"Predicted:\n{alignment_error.predicted}\n") print(f"CER: {sample_cer:3.2%}, Jiwer CER: {jiwer_cer:3.2%}\n\n") .. rst-class:: sphx-glr-script-out .. code-block:: none Reference: 🪄🐈‍⬛🎃 🐻‍❄️🎿🥶 Predicted: 🪄🐦‍⬛🎃 🐻⛸️🥶 CER: 42.86%, Jiwer CER: 33.33% Reference: Message Lorem Ipsum Hope you feel better soon!❤️‍🩹 Predicted: Massage Lorem lpsum Hope you feel better soon!❤️ CER: 6.38%, Jiwer CER: 8.00% Reference: What a great idea😑 Predicted: What a grea t idea🙂 CER: 11.11%, Jiwer CER: 11.11% Reference: Happy pride month! 🏳️‍🌈🌈🎉 Predicted: Happy pride moth! 🏳️‍🌈🌈🎉 CER: 4.55%, Jiwer CER: 4.00% Reference: That was close! 😮‍💨 Predicted: That was close! CER: 11.76%, Jiwer CER: 21.05% Reference: 1🐻‍❄️ Predicted: 🐻‍❄️ CER: 50.00%, Jiwer CER: 20.00% .. GENERATED FROM PYTHON SOURCE LINES 85-86 We see that stringalign computes CER correctly, even for emojis. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 0.022 seconds) .. _sphx_glr_download_auto_examples_plot_01_emoji_ocr_evaluation.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_01_emoji_ocr_evaluation.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_01_emoji_ocr_evaluation.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_01_emoji_ocr_evaluation.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_