0
votes
0
comments
4,162
views

Writing a tokenizer in Python

0 votes, 0.00 avg. rating (0% score)
I want to design a custom tokenizer module in Python that lets users specify what tokenizer(s) to use for the input. For instance, consider the following input: Q: What is a good way to achieve this? A: I am not so sure. I think I will use Python. I want to be able to provide […]