自然語言處理簡潔自用程式碼合集
阿新 • • 發佈：2019-01-03
記錄文書處理的各種簡介的程式碼表示
1.快速去除中文標點（read的時候要以utf8格式）
def clean_str(string):
string = re.sub("[^\u4e00-\u9fff]", " ", string)
string = re.sub(r"\s{2,}", " ", string)#合併多個空格為一個
return string.strip()2.快速分詞,預設一行為一樣本
def seperate_line(string):
return ''.join([word + ' ' for word in jieba.cut(string)])
f=open("xxx" 3.分行，使得一行為一句
for line in lines
line.replace('\n','').replace('，','\n').replace('。','\n').replace('！','\n').replace('？','\n')
重新寫入4.語料訓練集生成
def load_positive_negative_data_files(positive_data_file_path, negative_data_file_path) 5.句子填充
def padding_sentences(input_sentences, padding_token, padding_sentence_length = None):
sentences = [sentence.split(' ') for sentence in input_sentences]
if padding_sentence_length !=None:
max_sentence_length=padding_sentence_length
else:
max_sentence_length=max([len(sentence) for sentence in sentences])
for i,sentence in generate(sentences):
if len(sentence) > max_sentence_length:
sentences[i] = sentence[:max_sentence_length]
else:
sentence.extend([padding_token] * (max_sentence_length - len(sentence)))
return (sentences, max_sentence_length)6.從gensim訓練模型拿詞向量
model載入
all_vectors = []
embeddingDim = w2vModel.vector_size
embeddingUnknown = [0 for i in range(embeddingDim)]
for sentence in sentences:
this_vector = []
for word in sentence:
if word in w2vModel.wv.vocab:
this_vector.append(w2vModel[word])
else:
this_vector.append(embeddingUnknown)
all_vectors.append(this_vector)
return all_vectors7.打亂np矩陣的方法
x=[0,1,2,3,4,5,6]
x=np.array(x)
np.random.seed(10)
shuffle_indices = np.random.permutation(np.arange(len(x)))
print(shuffle_indices)
x_shuffled = x[shuffle_indices]
print(x_shuffled)
輸出
[2 6 0 3 4 5 1]
[2 6 0 3 4 5 1]8.分離部分樣本為訓練集和驗證集
1.打亂樣本順序（參考上面程式碼）
2.按比例截斷