应用NLP标记

您可以使用pandas apply函数在列的每一行上应用标记化

import pandas as pd
df = pd.DataFrame()
df['col1'] = ['ow to apply tokenization at a ','single column once ? I have two columns I w']
df['col2'] = ['unction for every row of single',' columns, can you please suggest']

             col1                        col2

0 ow to apply tokenization at a unction for every row of single
1 single column once ? I have two columns I w columns, can you please suggest

import nltk
df['col1'].apply(nltk.tokenize.word_tokenize)
日期：

0 [ow, to, apply, tokenization, at, a]
1 [single, column, once, ?, I, have, two, column...
Name: col1, dtype: object