.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/plot_07_sampling_alignments.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_plot_07_sampling_alignments.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_plot_07_sampling_alignments.py:


.. _randomised_alignments_example:

Uncertainty metrics for token-specific statistics
=================================================

This is a simple example that demonstrates how to estimate the uncertainty for token-specific metrics (see :ref:`sequence_alignment` for more information about how these can occur).
We use the same toy dataset here as in :ref:`most_common_errors_example`.

.. GENERATED FROM PYTHON SOURCE LINES 10-25

.. code-block:: Python


    from collections import Counter
    from pathlib import Path

    import numpy as np
    import stringalign

    data_path = Path("synthetic_transcription_data")
    predictions_path = data_path / "predicted.txt"
    reference_path = data_path / "reference.txt"

    predictions = predictions_path.read_text().splitlines()
    references = reference_path.read_text().splitlines()
    image_paths = data_path.glob("line*.jpg")








.. GENERATED FROM PYTHON SOURCE LINES 26-29

Normally, when we evaluate token-specific statistics, we would create a :class:`MultiAlignmentAnalyzer <stringalign.evaluate.MultiAlignmentAnalyzer>`.
However, now, we want to get uncertainty metrics on the different edit operations.
To get those, we need many :class:`MultiAlignmentAnalyzer <stringalign.evaluate.MultiAlignmentAnalyzer>` instances that use randomised alignments with different random seeds.

.. GENERATED FROM PYTHON SOURCE LINES 29-40

.. code-block:: Python


    analyzers = [
        stringalign.evaluate.MultiAlignmentAnalyzer.from_strings(
            references=references,
            predictions=predictions,
            randomize_alignment=True,
            random_state=i,
        )
        for i in range(10)
    ]








.. GENERATED FROM PYTHON SOURCE LINES 41-42

After creating the alignment analyzers, we find the set of all edit operations that occur in at least one alignment and use that to get lists of edit operation counts.

.. GENERATED FROM PYTHON SOURCE LINES 42-48

.. code-block:: Python


    all_edit_operations = {edit_op for analyzer in analyzers for edit_op in analyzer.edit_counts["raw"]}
    edit_operation_counts = {
        edit_op: [analyzer.edit_counts["raw"][edit_op] for analyzer in analyzers] for edit_op in all_edit_operations
    }








.. GENERATED FROM PYTHON SOURCE LINES 49-50

Now that we have a dictionary that maps the edit operations to their count in each of the random alignments, we can compute their mean and standard deviation.

.. GENERATED FROM PYTHON SOURCE LINES 50-58

.. code-block:: Python


    edit_operation_averages = Counter({edit_op: np.mean(counts) for edit_op, counts in edit_operation_counts.items()})
    edit_operation_stddev = {edit_op: np.std(counts) for edit_op, counts in edit_operation_counts.items()}

    for edit_op, count in edit_operation_averages.most_common(10):
        stddev = edit_operation_stddev[edit_op]
        print(f"{edit_op:19s}: {count:5.2f} +- {stddev:.2f}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    REPLACED 's' -> 'f': 10.00 +- 0.00
    REPLACED 'k' -> 't':  8.00 +- 0.00
    REPLACED 'd' -> 'b':  6.00 +- 0.00
    REPLACED 'ø' -> 'o':  3.00 +- 0.00
    INSERTED 'n'       :  2.10 +- 0.83
    REPLACED 'm' -> 'r':  2.10 +- 0.83
    INSERTED 'e'       :  2.00 +- 0.77
    DELETED  'n'       :  2.00 +- 0.00
    REPLACED 'Q' -> 'D':  2.00 +- 0.00
    REPLACED 'n' -> 'm':  2.00 +- 0.00




.. GENERATED FROM PYTHON SOURCE LINES 59-61

We see that in this case, there was no uncertainty in the most common errors, and even then, the uncertainty was not too large.
This type of uncertainty analysis can be very useful when, e.g. comparing different transcription models on token-specific metrics.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.628 seconds)


.. _sphx_glr_download_auto_examples_plot_07_sampling_alignments.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_07_sampling_alignments.ipynb <plot_07_sampling_alignments.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_07_sampling_alignments.py <plot_07_sampling_alignments.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_07_sampling_alignments.zip <plot_07_sampling_alignments.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_