CountVectorizer读写词汇(CountVectorizer reading and writing vocabulary)

 我目前正致力于一个相当微不足道的情绪分类计划。 一切都在训练阶段很好。 但是，我无法使用CountVectorizer来测试包含看不见的单词的新文本字符串。 
 
 出于这个原因，我试图在测试阶段为矢量化编写查找词汇表。 但是，我不知道如何创建和检索要作为参数传递的词汇表对象。 
 
 我的两种方法目前如下所示： 
 
def trainingVectorTransformation (messages):
    #--> ReviewText to vectors    
    vect = CountVectorizer(analyzer=split_into_lemmas).fit(messages['reviewText'])

    messages_bow = vect.transform(messages['reviewText'])

    feature_list = vect.get_feature_names()
    #NOT SURE HOW TO CREATE VOCABULARY
    with open("vocab.txt", "w") as text_file:
        text_file.write(str(feature_list))   

    tfidf_transformer = TfidfTransformer().fit(messages_bow)


    messages_tfidf = tfidf_transformer.transform(messages_bow)
    return messages_tfidf
 
 和 
 
def testingVectorTransformation (messages):
    #--> ReviewText to vectors
    #NOT SURE HOW TO READ THE CREATED VOCABULARY AND USE IT APPROPRIATELY   
    txt = open("vocab.txt")
    vocabulary = txt.read()


    vect = CountVectorizer(analyzer=split_into_lemmas, vocabulary = vocabulary).fit(messages['reviewText'])

    messages_bow = vect.transform(messages['reviewText'])

    tfidf_transformer = TfidfTransformer().fit(messages_bow)

    messages_tfidf = tfidf_transformer.transform(messages_bow)
    return messages_tfidf
 
 如果有人对如何正确创建和使用词汇表有任何建议，我将非常感激。 

I am currently working on a fairly trivial sentiment classification program. Everything works well in the training phase. However, I am having trouble using CountVectorizer to test new strings of text that contain unseen words.
 
For this reason I am trying to write a lookup vocabulary for vectorization in the testing phase. However, I don't know how to create and retrieve the vocabulary object to pass as a parameter.
 
My two methods currently appear as follows:
 
def trainingVectorTransformation (messages):
    #--> ReviewText to vectors    
    vect = CountVectorizer(analyzer=split_into_lemmas).fit(messages['reviewText'])

    messages_bow = vect.transform(messages['reviewText'])

    feature_list = vect.get_feature_names()
    #NOT SURE HOW TO CREATE VOCABULARY
    with open("vocab.txt", "w") as text_file:
        text_file.write(str(feature_list))   

    tfidf_transformer = TfidfTransformer().fit(messages_bow)


    messages_tfidf = tfidf_transformer.transform(messages_bow)
    return messages_tfidf
 
and 
 
def testingVectorTransformation (messages):
    #--> ReviewText to vectors
    #NOT SURE HOW TO READ THE CREATED VOCABULARY AND USE IT APPROPRIATELY   
    txt = open("vocab.txt")
    vocabulary = txt.read()


    vect = CountVectorizer(analyzer=split_into_lemmas, vocabulary = vocabulary).fit(messages['reviewText'])

    messages_bow = vect.transform(messages['reviewText'])

    tfidf_transformer = TfidfTransformer().fit(messages_bow)

    messages_tfidf = tfidf_transformer.transform(messages_bow)
    return messages_tfidf
 
If anyone has any advice on how to properly create and use the vocabulary I would very much appreciate it.

原文：https://stackoverflow.com/questions/37484369