Python Japanese code encoding:UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position 0: invalid start byte


Python Japanese code encoding:UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position 0: invalid start byte



I use the following code(python3.6) to insert csv data into the database：


# -*- coding: UTF-8 -*-
import pandas as pd
import pymssql

def insert_report_pn_dictionary(server, user, password, database):
dict_list =
tw_df = pd.read_csv(r'D:/dict.csv', encoding='utf-8')
word_list = list(tw_df['WORD'])
pn_list = list(tw_df['PN'])
pn_dict = dict(zip(word_list, pn_list))
for key, value in pn_dict.items():
dict_list.append((key, value))
try:
conn = pymssql.connect(server, user, password, database)
cur = conn.cursor()
sql = ' insert into report_pn_dictionary (dict_keyword,dict_pn) '
' values(%s, %s) '
cur.executemany(sql, dict_list)
conn.commit()
except pymssql.Error as ex:
raise ex
except Exception as ex:
raise ex
finally:
conn.close()

if __name__=="__main__":
server = '10.10.10.10'
user = 'test'
password = 'test'
database = 'DBTest'
insert_report_pn_dictionary(server, user, password, database)



But there is a error. The error message is：


E:Anaconda3python.exe E:/test_opencv/TRkeywordpn/set_dictionary.py
Traceback (most recent call last):
File "E:/test_opencv/TRkeywordpn/set_dictionary.py", line 33, in <module>
insert_report_pn_dictionary(server, user, password, database)
File "E:/test_opencv/TRkeywordpn/set_dictionary.py", line 7, in insert_report_pn_dictionary
tw_df = pd.read_csv(r'D:/dict.csv', encoding='utf-8')
File "E:Anaconda3libsite-packagespandasioparsers.py", line 646, in parser_f
return _read(filepath_or_buffer, kwds)
File "E:Anaconda3libsite-packagespandasioparsers.py", line 401, in _read
data = parser.read()
File "E:Anaconda3libsite-packagespandasioparsers.py", line 939, in read
ret = self._engine.read(nrows)
File "E:Anaconda3libsite-packagespandasioparsers.py", line 1508, in read
data = self._reader.read(nrows)
File "pandasparser.pyx", line 848, in pandas.parser.TextReader.read (pandasparser.c:10415)
File "pandasparser.pyx", line 870, in pandas.parser.TextReader._read_low_memory (pandasparser.c:10691)
File "pandasparser.pyx", line 947, in pandas.parser.TextReader._read_rows (pandasparser.c:11728)
File "pandasparser.pyx", line 1049, in pandas.parser.TextReader._convert_column_data (pandasparser.c:13162)
File "pandasparser.pyx", line 1108, in pandas.parser.TextReader._convert_tokens (pandasparser.c:14116)
File "pandasparser.pyx", line 1206, in pandas.parser.TextReader._convert_with_dtype (pandasparser.c:16172)
File "pandasparser.pyx", line 1222, in pandas.parser.TextReader._string_convert (pandasparser.c:16400)
File "pandasparser.pyx", line 1458, in pandas.parser._string_box_utf8 (pandasparser.c:22072)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position 0: invalid start byte



My CSV content likes this:



WORD PN



優れる 1



良い 0.999995



喜ぶ 0.999979



褒める 0.999979



めでたい 0.999645



賢い 0.999486



善い 0.999314



I have more than 50 thousand lines data.



How to modify my code?I think this is the problem of Japanese coding.





your csv is not in utf-8. try to guess which encoding it uses. SHIFT_JIS may be the one
– bobrobbob
Jun 29 at 9:03



SHIFT_JIS





@bobrobbob I modified it as you said,it worked,thanks!
– MadFrog
12 hours ago









By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Comments

Post a Comment

Popular posts from this blog

paramiko-expect timeout is happening after executing the command

paramiko-expect timeout is happening after executing the command Can someone help me why timeout is happening after executing the command .I am trying to SSH to a machine and execute a command.I want to store the result of the executed command . I have referred paramiko-timeout but i didn't get my expected result. import traceback try: import paramiko except ImportError: raise Exception("Please install paramiko , pip install paramiko") import pexpect from paramiko_expect import SSHClientInteraction def main(): HOSTNAME = "minion5.net" USERNAME = "root" PASSWORD = "help@432" PROMPT = "root@*:~$s+" try: client = paramiko.SSHClient() client.load_system_host_keys() client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) client.connect(hostname=HOSTNAME, username=USERNAME, password=PASSWORD) with SSHClientInteraction(client, timeout=20, display=True) as interact: ...
Read more

Opening a url is failing in Swift

Opening a url is failing in Swift breakpointing inside the callback and logging success shows false. I added itms-apps to my plist: LSApplicationQueriesSchemes and put this in my app delegate: func application(_ app: UIApplication, open url: URL, options: [UIApplicationOpenURLOptionsKey : Any] = [:]) -> Bool { return true } func moreTapped() { let url = NSURL(string: "itms-apps://itunes.com/developer/quantum-productions/id979315877") if #available(iOS 10.0, *) { UIApplication.shared.open(url! as URL, options: [:], completionHandler: {(success: Bool) in }) } else { UIApplication.shared.openURL(url! as URL) } } Are you testing on a real device? – rmaddy Jun 21 at 23:57 I try on a device, get "Cannot Connect to App Store". Device is online ...
Read more

Export result set on Dbeaver to CSV

Export result set on Dbeaver to CSV Normally I use Dbeaver for windows and always export my result set like this: run my query --> select the result --> export the result set --> select export to clipboard --> done This step by step puts my result set in my clipboard and I can paste it wherever I want to work with it. The problem is that now I am using dbeaver for mac and this guide is not working. I can go on until the moment that I select my result set like in the image below: exporting data set But once I go further in the process, in the last step I get: no query Note that in "source" it was suppose to show the query that originated the result set, but instead it says just "select. As a result it does't select my result or anything (besides being "successful"). Normally my query would show up there automatically and I couldn't find any option that corrects this problem in the menus. Please help me! Thanks. Aft...
Read more