How can I output tokens from MWE Tokenizer? [on hold]
$begingroup$
Anyone know how to output the tokens produced using MWE Tokenizer?
For a clearer explanation of what I am asking for those who did not understand my original brief question.
The multi-word expression tokenizer (MWETokenizer) provides a method/function (add_mwe()) that allows the user to enter multiple word expressions prior to using the tokenizer on text. Currently I have a file consisting of phrases / multi-word expression I want to use with the tokenizer. My concern is that the manner in which I am presenting the phrases to the function correctly and so not resulting in the desired set of tokens to be used in tokenizing the incoming text. So this leads me to ask if anyone knows how to output the token generated by this method/function so that I can verify that I am correctly passing the phrase to the function (add_mwe()).?
nlp nltk tokenization
asked yesterday
PaulPaul
11
New contributor
Paul is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
put on hold as unclear what you're asking by oW_, Ethan, Toros91, Mark.F, Siong Thye Goh 16 hours ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
add a comment |
$begingroup$
Anyone know how to output the tokens produced using MWE Tokenizer?
For a clearer explanation of what I am asking for those who did not understand my original brief question.
The multi-word expression tokenizer (MWETokenizer) provides a method/function (add_mwe()) that allows the user to enter multiple word expressions prior to using the tokenizer on text. Currently I have a file consisting of phrases / multi-word expression I want to use with the tokenizer. My concern is that the manner in which I am presenting the phrases to the function correctly and so not resulting in the desired set of tokens to be used in tokenizing the incoming text. So this leads me to ask if anyone knows how to output the token generated by this method/function so that I can verify that I am correctly passing the phrase to the function (add_mwe()).?
nlp nltk tokenization
asked yesterday
PaulPaul
11
New contributor
Paul is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
put on hold as unclear what you're asking by oW_, Ethan, Toros91, Mark.F, Siong Thye Goh 16 hours ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
add a comment |
$begingroup$
Anyone know how to output the tokens produced using MWE Tokenizer?
For a clearer explanation of what I am asking for those who did not understand my original brief question.
The multi-word expression tokenizer (MWETokenizer) provides a method/function (add_mwe()) that allows the user to enter multiple word expressions prior to using the tokenizer on text. Currently I have a file consisting of phrases / multi-word expression I want to use with the tokenizer. My concern is that the manner in which I am presenting the phrases to the function correctly and so not resulting in the desired set of tokens to be used in tokenizing the incoming text. So this leads me to ask if anyone knows how to output the token generated by this method/function so that I can verify that I am correctly passing the phrase to the function (add_mwe()).?
nlp nltk tokenization
asked yesterday
PaulPaul
11
New contributor
Paul is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
Anyone know how to output the tokens produced using MWE Tokenizer?
For a clearer explanation of what I am asking for those who did not understand my original brief question.
The multi-word expression tokenizer (MWETokenizer) provides a method/function (add_mwe()) that allows the user to enter multiple word expressions prior to using the tokenizer on text. Currently I have a file consisting of phrases / multi-word expression I want to use with the tokenizer. My concern is that the manner in which I am presenting the phrases to the function correctly and so not resulting in the desired set of tokens to be used in tokenizing the incoming text. So this leads me to ask if anyone knows how to output the token generated by this method/function so that I can verify that I am correctly passing the phrase to the function (add_mwe()).?
nlp nltk tokenization
nlp nltk tokenization
asked yesterday
PaulPaul
11
New contributor
Paul is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
PaulPaul
11
New contributor
Paul is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited 10 hours ago
Paul
asked yesterday
PaulPaul
11
New contributor
Paul is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
PaulPaul
11
asked yesterday
PaulPaul
11
11
New contributor
Paul is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Paul is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Paul is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
put on hold as unclear what you're asking by oW_, Ethan, Toros91, Mark.F, Siong Thye Goh 16 hours ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
put on hold as unclear what you're asking by oW_, Ethan, Toros91, Mark.F, Siong Thye Goh 16 hours ago
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
add a comment |
add a comment |
0
active
oldest
votes
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
