How to use a string in Chinese characters as input for another command in bash?

Question

I'm currently using a command to convert some Chinese characters into pinyin, which requires the string in my bash to be Unicode safe and put the result in another variable. I can run the following command normally:

 chinese="你好" to-pinyin.py $chinese 

It will print the output as expected, however, since I want the output in a variable, I tried to do the following:

 chinese="你好" pinyin=$(to-pinyin.py $chinese) 

And python will fail with:

Traceback (most recent call last): File "/~/to-pinyin.py", line 31, in <module> print pinyin.get(hanzi, delimiter=" ").capitalize() UnicodeEncodeError: 'ascii' codec can't encode character u'\u01d0' in position 1: ordinal not in range(128) 

Same thing will happen with backticks. I think I will work around by writing the output to a file and to a conversion there, then load the strings to a variable. How else can I fix this so that I can avoid the workaround?

EDIT:

Per request here is the output of locale:

$ locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL= 

SOLUTION USED

Thanks to muru's response and some help of this other answer I added .encode('utf-8') to the end of the printed strings in my python script.

I wish I could switch to python3, but there is no defalut pinyin package there and I can't seem to install any good pinyin package that would get my job done quickly in python3. I remember trying for a while but python3 kept refusing to import the package I had installed, so I just installed one in python2 and it worked straight out of the box.

Answer 1

This is an issue with Python 2's print, and reason to favour Python 3's consistent Unicode handling.

Now why does redirecting to a file cause problems? It’s because print() in python2 is treated specially. Whereas the other file-like objects in python always convert to ASCII unless you set them up differently, using print() to output to the terminal will use the user’s locale to convert before sending the output to the terminal. When print() is not outputting to the terminal (being redirected to a file, for instance), print() decides that it doesn’t know what locale to use for that file and so it tries to convert to ASCII instead.