tests.tokenizer package

Submodules

tests.tokenizer.test_tokenizers module

tests.tokenizer.test_tokenizers.test_jumanpp_tokenizer()

tests.tokenizer.test_tokenizers.test_mecab_ipadic_tokenizer()

tests.tokenizer.test_tokenizers.test_mecab_unidic_lite_tokenizer()

tests.tokenizer.test_tokenizers.test_mecab_unidic_tokenizer()

tests.tokenizer.test_tokenizers.test_spacy_ginza_tokenizer()

tests.tokenizer.test_tokenizers.test_spm_tokenizer()

tests.tokenizer.test_tokenizers.test_sudachi_tokenizer()

Module contents