.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/plot_05_character_specific_scores.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_plot_05_character_specific_scores.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_plot_05_character_specific_scores.py:


.. _token_specific_performance_example:

Evaluating token-specific performance
=====================================

Sometimes, you might be interested in seeing how the model performs on a specific subset of characters.
Stringalign provides a couple of useful utilities for this.

.. GENERATED FROM PYTHON SOURCE LINES 10-33

.. code-block:: Python


    import stringalign

    references = [
        "Snekkermester Thor Bjørklund fra Øvre Smestad i Fåberg patenterte Ostehøvelen i 1925.",
        "Snøen smeltet i vårsola.",
        "Ved å blande blått og gult kan du få grønt.",
        "Det var et ærlig forsøk",
    ]
    predictions = [
        "Snekkermester Thor Bjorklund fra Ovre Smestad i Faberg patenterte Ostehovelen i 1925.",
        "Snoen smeltet i varsola",
        "Ved a blande blatt og gult kan du fa grønt.",
        "Det var et aerlig forsok",
    ]

    tokenizer = stringalign.tokenize.GraphemeClusterTokenizer()
    analyzer = stringalign.evaluate.MultiAlignmentAnalyzer.from_strings(references, predictions, tokenizer)
    cm = analyzer.confusion_matrix

    cer = cm.compute_token_error_rate()
    print(f"The CER is {cer}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    The CER is 0.07428571428571429




.. GENERATED FROM PYTHON SOURCE LINES 34-35

Next, we can compute token-specific statistics to see how the model performs on specific characters:

.. GENERATED FROM PYTHON SOURCE LINES 35-47

.. code-block:: Python


    sensitivity = cm.compute_sensitivity()
    precision = cm.compute_precision()
    f1_scores = cm.compute_f1_score()

    for character in "æøå":
        print(f"Statistics for {character}:")
        print(f"Sensitivity: {sensitivity[character]}")
        print(f"Precision:   {precision[character]}")
        print(f"F1 score:    {f1_scores[character]}")
        print()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Statistics for æ:
    Sensitivity: 0.0
    Precision:   nan
    F1 score:    0.0

    Statistics for ø:
    Sensitivity: 0.2
    Precision:   1.0
    F1 score:    0.33333333333333337

    Statistics for å:
    Sensitivity: 0.0
    Precision:   nan
    F1 score:    0.0





.. GENERATED FROM PYTHON SOURCE LINES 48-54

We see that the precision can be ``nan``, this happens for all tokens *not* in the predicted string, as the precision is defined by the number of times a given token was correctly identified (``true_positives``) divided by the number of times the token was predicted (``true_positives + false_positives``).
If a token never occurs in the predicted string, then this quantity is ill-defined, and becomes ``nan``.

Aggregating the summary statistics
----------------------------------
We can also aggregate the number of true positives, false positives and false negatives for multiple tokens to get an overall measure for Norwegian special characters

.. GENERATED FROM PYTHON SOURCE LINES 54-62

.. code-block:: Python


    overall_sensitivity = cm.compute_sensitivity(aggregate_over="æøå")
    overall_precision = cm.compute_precision(aggregate_over="æøå")
    overall_f1 = cm.compute_f1_score(aggregate_over="æøå")

    print(f"The overall sensitivity for æ, ø and å is: {overall_sensitivity}")
    print(f"The overall precision for æ, ø and å is:   {overall_precision}")
    print(f"The overall F1 score for æ, ø and å is:    {overall_f1}")




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    The overall sensitivity for æ, ø and å is: 0.09090909090909091
    The overall precision for æ, ø and å is:   1.0
    The overall F1 score for æ, ø and å is:    0.16666666666666669





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.018 seconds)


.. _sphx_glr_download_auto_examples_plot_05_character_specific_scores.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_05_character_specific_scores.ipynb <plot_05_character_specific_scores.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_05_character_specific_scores.py <plot_05_character_specific_scores.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_05_character_specific_scores.zip <plot_05_character_specific_scores.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_