[漢字編碼報錯] UnicodeEncodeError- 'ascii' codec can't encode characters in position 0-1- ordinal not in r
阿新 • • 發佈:2019-02-04
原始程式碼
# -*- coding:utf-8 -*-
import pandas as pd
import jieba
def cut_msg(ustr):
# ustr = ustr.encode("raw_unicode_escape").decode("raw_unicode_escape").encode("utf8")
return " ".join(jieba.lcut(str(ustr)))
fp = "gray.xlsx"
df = pd.read_excel(fp)
df["msg"] = df["msg"].map(cut_msg)
li = df["msg" ]
with file(fp.replace(".xlsx", ".txt"), "wb") as wf:
for e in li:
wf.write(e.encode("utf8")+"\n")
報錯資訊
---------------------------------------------------------------------------
UnicodeEncodeError Traceback (most recent call last)
<ipython-input-28-b71365cb72a8> in <module>()
1 fp = "gray.xlsx"
2 df = pd.read_excel(fp)
----> 3 df["msg"] = df["msg"].map(cut_msg)
4 li = df["msg"]
5 with file(fp.replace(".xlsx", ".txt"), "wb") as wf:
/Users/yipu.si/anaconda/lib/python2.7/site-packages/pandas/core/series.pyc in map(self, arg, na_action)
2156 else:
2157 # arg is a function
-> 2158 new_values = map_f(values, arg)
2159
2160 return self._constructor(new_values,
pandas/_libs/src/inference.pyx in pandas._libs.lib.map_infer (pandas/_libs/lib.c:66440)()
<ipython-input-26-f92cd72f89b6> in cut_msg(ustr)
4 import jieba
5 def cut_msg(ustr):
----> 6 return " ".join(jieba.lcut(str(ustr)))
7
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)
解決方案
加入編碼和解碼語句
ustr.encode("raw\_unicode\_escape").decode("raw\_unicode\_escape").encode("utf8") # 這裡的 utf8 根據實際資料編碼而定
如下:
# -*- coding:utf-8 -*-
import pandas as pd
import jieba
def cut_msg(ustr):
ustr = ustr.encode("raw_unicode_escape").decode("raw_unicode_escape").encode("utf8") # 這裡的 utf8 根據實際資料編碼而定
return " ".join(jieba.lcut(str(ustr)))
後記
漢字編碼問題一直困擾著處理文字資料的童鞋,我也在不斷探索中,在此拋磚引玉,往路過大神指點迷津。