.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/plot_02_tokenizations.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_plot_02_tokenizations.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_plot_02_tokenizations.py:


.. _tokenizers_example:

Overview of tokenizations
=========================

This example shows the different tokenization methods available in Stringalign
and how you can easily add your own custom tokenization if you have any specific needs.

For these examples we will not add any string normalization (see :ref:`this page for more on that <unicode_normalization>`).
See :ref:`here <cer_example>` for an example that adds a string normalization to a tokenizer.
And the :mod:`stringalign.normalize` API documentation for an overview of the normalization options available in Stringalign.

.. GENERATED FROM PYTHON SOURCE LINES 14-19

.. code-block:: Python


    import stringalign

    example_sentence = "Hello World! This is fun (example) sentence no. 10 000.😶‍🌫️"








.. GENERATED FROM PYTHON SOURCE LINES 20-22

:class:`stringalign.tokenize.GraphemeClusterTokenizer`
------------------------------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 22-25

.. code-block:: Python

    tokenizer = stringalign.tokenize.GraphemeClusterTokenizer()
    print(tokenizer(example_sentence))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ['H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd', '!', ' ', 'T', 'h', 'i', 's', ' ', 'i', 's', ' ', 'f', 'u', 'n', ' ', '(', 'e', 'x', 'a', 'm', 'p', 'l', 'e', ')', ' ', 's', 'e', 'n', 't', 'e', 'n', 'c', 'e', ' ', 'n', 'o', '.', ' ', '1', '0', ' ', '0', '0', '0', '.', '😶\u200d🌫️']




.. GENERATED FROM PYTHON SOURCE LINES 26-28

:class:`stringalign.tokenize.SplitAtWhitespaceTokenizer`
--------------------------------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 28-31

.. code-block:: Python

    tokenizer = stringalign.tokenize.SplitAtWhitespaceTokenizer()
    print(tokenizer(example_sentence))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ['Hello', 'World!', 'This', 'is', 'fun', '(example)', 'sentence', 'no.', '10', '000.😶\u200d🌫️']




.. GENERATED FROM PYTHON SOURCE LINES 32-34

:class:`stringalign.tokenize.SplitAtWordBoundaryTokenizer`
----------------------------------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 34-37

.. code-block:: Python

    tokenizer = stringalign.tokenize.SplitAtWordBoundaryTokenizer()
    print(tokenizer(example_sentence))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ['Hello', ' ', 'World', '!', ' ', 'This', ' ', 'is', ' ', 'fun', ' ', '(', 'example', ')', ' ', 'sentence', ' ', 'no', '.', ' ', '10', ' ', '000', '.', '😶\u200d🌫️']




.. GENERATED FROM PYTHON SOURCE LINES 38-40

:class:`stringalign.tokenize.UnicodeWordTokenizer`
--------------------------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 40-43

.. code-block:: Python

    tokenizer = stringalign.tokenize.UnicodeWordTokenizer()
    print(tokenizer(example_sentence))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ['Hello', 'World', 'This', 'is', 'fun', 'example', 'sentence', 'no', '10', '000']




.. GENERATED FROM PYTHON SOURCE LINES 44-46

Custom tokenizer using `nb_tokenizer`
-------------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 46-51

.. code-block:: Python

    import nb_tokenizer

    tokenizer = stringalign.tokenize.add_join()(nb_tokenizer.tokenize)

    print(tokenizer(example_sentence))




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ['Hello', 'World', '!', 'This', 'is', 'fun', '(', 'example', ')', 'sentence', 'no.', '10 000', '.', '😶', '\u200d', '🌫', '️']





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.007 seconds)


.. _sphx_glr_download_auto_examples_plot_02_tokenizations.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_02_tokenizations.ipynb <plot_02_tokenizations.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_02_tokenizations.py <plot_02_tokenizations.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_02_tokenizations.zip <plot_02_tokenizations.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_