.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/plot_00_cer.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_plot_00_cer.py: .. _cer_example: Computing the CER and WER ========================= This example shows how to compute the character error rate (CER) and word error rate (WER) with Stringalign and how the choice of tokenizer can be important. .. GENERATED FROM PYTHON SOURCE LINES 9-12 .. code-block:: Python import stringalign .. GENERATED FROM PYTHON SOURCE LINES 13-15 One way to measure the performance of a predicted text transcription compared to a reference is by calculating the character error rate (CER) and word error rate (WER) which are special cases of the token error rate (TER) (For more details on these metrics see :ref:`token_error_rate`). To calculate the CER between two strings we first need a character-level tokenizer which we use to get an alignment that we can use to compare the strings. .. GENERATED FROM PYTHON SOURCE LINES 15-27 .. code-block:: Python from stringalign.evaluate import AlignmentAnalyzer reference = "Ηello world!" predicted = "Hello world!!" tokenizer = stringalign.tokenize.GraphemeClusterTokenizer() alignment_analyzer = AlignmentAnalyzer.from_strings(reference, predicted, tokenizer=tokenizer) cer = alignment_analyzer.compute_ter() print(f"The character error rate is {cer:.2f}") .. rst-class:: sphx-glr-script-out .. code-block:: none The character error rate is 0.17 .. GENERATED FROM PYTHON SOURCE LINES 28-30 You might notice that this number is twice what we may expect. We can visualise the alignment to understand why. .. GENERATED FROM PYTHON SOURCE LINES 30-33 .. code-block:: Python alignment_analyzer.visualize() .. raw:: html

Reference:Predicted:

Η H

e e

l l

o o

w w

o o

r r

l l

d d

! !

.. GENERATED FROM PYTHON SOURCE LINES 34-37 We see that there was a :ref:`confusable character ` used for the ``H``, and the benefit of this way of computing the CER and WER is that the tokenizer is explicit. If we want to resolve confusables, then we can switch out the tokenizer. .. GENERATED FROM PYTHON SOURCE LINES 37-52 .. code-block:: Python reference = "Ηello world!" predicted = "Hello world!!" # Resolve confusables before tokenization normalizer = stringalign.normalize.StringNormalizer(resolve_confusables="intentional") tokenizer = stringalign.tokenize.GraphemeClusterTokenizer(pre_tokenization_normalizer=normalizer) alignment_analyzer = AlignmentAnalyzer.from_strings(reference, predicted, tokenizer=tokenizer) cer = alignment_analyzer.compute_ter() print(f"The character error rate is {cer:.2f}") alignment_analyzer.visualize() .. rst-class:: sphx-glr-script-out .. code-block:: none The character error rate is 0.08 .. raw:: html

Reference:Predicted:

H H

e e

l l

o o

w w

o o

r r

l l

d d

! !

.. GENERATED FROM PYTHON SOURCE LINES 53-58 Computing the word error rate ----------------------------- Similarly, if we want to compute the word error rate, we can switch out the tokenizer, and, for example, use whitespace characters to signify word boundaries. .. GENERATED FROM PYTHON SOURCE LINES 58-73 .. code-block:: Python reference = "Ηello world!" predicted = "Hello world!!" # Resolve confusables before tokenization normalizer = stringalign.normalize.StringNormalizer(resolve_confusables="intentional") word_tokenizer = stringalign.tokenize.SplitAtWhitespaceTokenizer(pre_tokenization_normalizer=normalizer) alignment_analyzer = AlignmentAnalyzer.from_strings(reference, predicted, tokenizer=word_tokenizer) wer = alignment_analyzer.compute_ter() print(f"The word error rate is {wer:.2f}") alignment_analyzer.visualize(space_alignment_ops=True) # Space alignment ops to add whitespace around each word .. rst-class:: sphx-glr-script-out .. code-block:: none The word error rate is 0.50 .. raw:: html

Reference:Predicted:

Hello Hello

world! world!!

.. GENERATED FROM PYTHON SOURCE LINES 74-80 Using different word tokenizers ------------------------------- Sometimes, we may not be too interested in how punctuation affects model performance. In those cases, we can, for example, use the :class:`stringalign.tokenize.UnicodeWordTokenizer`, which uses the word extraction algorithm described in :cite:p:`unicode-annex-29` to tokenize the strings into words without punctuation. .. GENERATED FROM PYTHON SOURCE LINES 80-95 .. code-block:: Python reference = "Ηello world!" predicted = "Hello world!!" # Resolve confusables before tokenization normalizer = stringalign.normalize.StringNormalizer(resolve_confusables="intentional") unicode_word_tokenizer = stringalign.tokenize.UnicodeWordTokenizer(pre_tokenization_normalizer=normalizer) alignment_analyzer = AlignmentAnalyzer.from_strings(reference, predicted, tokenizer=unicode_word_tokenizer) unicode_word_wer = alignment_analyzer.compute_ter() print(f"The word error rate with a SplitAtWhitespaceTokenizer is: {wer:.2f}") print(f"The word error rate with a UnicodeWordTokenizer is: {unicode_word_wer:.2f}") alignment_analyzer.visualize(space_alignment_ops=True) # Space alignment ops to add whitespace around each word .. rst-class:: sphx-glr-script-out .. code-block:: none The word error rate with a SplitAtWhitespaceTokenizer is: 0.50 The word error rate with a UnicodeWordTokenizer is: 0.00 .. raw:: html

Reference:Predicted:

Hello Hello

world world

.. GENERATED FROM PYTHON SOURCE LINES 96-105 Convenience functions --------------------- Generally, it is good practice to explicitly define your tokenizer before calculating the CER or WER. Afterall, before looking at the CER you should have an idea of what you mean by "character" for your particular problem. However, stringalign also supports three convinence functions for quickly calculating CER, WER and TER: :func:`stringalign.evaluate.compute_cer`, :func:`stringalign.evaluate.compute_wer` and :func:`stringalign.evaluate.compute_ter`. These functions will use sensible defaults and return the :class:`stringalign.evaluate.AlignmentAnalyzer` used for the calculation. To ensure reproducibility, you can inspect the returned analyzer and note the tokenization and normalization. .. GENERATED FROM PYTHON SOURCE LINES 105-115 .. code-block:: Python cer, cer_analyzer = stringalign.evaluate.compute_cer(reference, predicted) wer, wer_analyzer = stringalign.evaluate.compute_wer(reference, predicted) print(f"The CER is {cer}") print(f"We used this analyzer to compute the CER: {cer_analyzer}") print() print(f"The WER is {wer}") print(f"We used this analyzer to compute the WER: {wer_analyzer}") .. rst-class:: sphx-glr-script-out .. code-block:: none The CER is 0.16666666666666666 We used this analyzer to compute the CER: AlignmentAnalyzer( reference='Ηello world!', predicted='Hello world!!', metadata=None, tokenizer=GraphemeClusterTokenizer( pre_tokenization_normalizer=StringNormalizer( normalization='NFC', case_insensitive=False, normalize_whitespace=False, remove_whitespace=False, remove_non_word_characters=False, resolve_confusables=None, ), post_tokenization_normalizer=StringNormalizer( normalization='NFC', case_insensitive=False, normalize_whitespace=False, remove_whitespace=False, remove_non_word_characters=False, resolve_confusables=None, ) ) ) The WER is 1.0 We used this analyzer to compute the WER: AlignmentAnalyzer( reference='Ηello world!', predicted='Hello world!!', metadata=None, tokenizer=SplitAtWhitespaceTokenizer( pre_tokenization_normalizer=StringNormalizer( normalization='NFC', case_insensitive=False, normalize_whitespace=False, remove_whitespace=False, remove_non_word_characters=False, resolve_confusables=None, ), post_tokenization_normalizer=StringNormalizer( normalization='NFC', case_insensitive=False, normalize_whitespace=False, remove_whitespace=False, remove_non_word_characters=False, resolve_confusables=None, ) ) ) .. GENERATED FROM PYTHON SOURCE LINES 116-123 Evaluating multiple strings --------------------------- We can also evaluate multiple strings at once. The most obvious way to do that might be to compute the CER for each sample and take an average. However, that approach would put artificially high weight on characters in short strings. Instead, we want to compute the total number of insertions, deletions, substitutions and reference tokens across all strings, and then compute the error rates using the equations defined in :ref:`token_error_rate`. To do that in Stringalign, we create a :class:`stringalign.evaluate.MultiAlignmentAnalyzer` .. GENERATED FROM PYTHON SOURCE LINES 123-136 .. code-block:: Python from stringalign.evaluate import MultiAlignmentAnalyzer references = ["Ηello world!", "Goodbye for now :)"] predictions = ["Hello world!!", "Godbye for now!"] # Resolve confusables before tokenization normalizer = stringalign.normalize.StringNormalizer(resolve_confusables="intentional") tokenizer = stringalign.tokenize.GraphemeClusterTokenizer(pre_tokenization_normalizer=normalizer) multi_alignment_analyzer = MultiAlignmentAnalyzer.from_strings(references, predictions, tokenizer=tokenizer) cer = multi_alignment_analyzer.compute_ter() print(f"The overall CER for this dataset is {cer}") .. rst-class:: sphx-glr-script-out .. code-block:: none The overall CER for this dataset is 0.16666666666666666 .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 0.020 seconds) .. _sphx_glr_download_auto_examples_plot_00_cer.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_00_cer.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_00_cer.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_00_cer.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_