[UPDATE 16 Aug 2011] Armin Ronacher has written a nice module called unicode-nazi that provides the Unicode warnings I discuss at the end of this article.
Though I can't use Python 3 for any of my projects, it does have a few nice things. One particular behaviour where it improves on Python 2 is forbidding implicit conversions between byte strings and Unicode strings. For example:
Python 3.1.2 (release31-maint, Sep 17 2010, 20:34:23)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 'foo' + b'baz'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't convert 'bytes' object to str implicitly
If you do this in Python 2, it invokes the default encoding to convert between bytes and unicode, leading to manifold unhappinesses. So in Python 2, the above example looks like:
Python 2.6.6 (r266:84292, Sep 15 2010, 15:52:39)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> u'foo' + b'baz'
u'foobaz'
This looks OK, but problems lurk just below the surface. The default encoding used in nearly all circumstances is ASCII, and this conversion will blow up if non-ASCII bytes are involved.
>>> u'foo' + b'\xe2\x98\x83'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
This is only a 3, at best 3.5 on the Russell scale of API misusability. Even worse, this conversion is done for any function implemented in C that expects unicode or bytes and receives the other type. For example, unicode() converts byte strings to unicode strings, and if not given an explicit encoding argument, uses the default encoding.
>>> unicode('bob')
u'bob'
>>> unicode('\xe2\x98\x83')
Traceback (most recent call last):
File "<stdin>, line 1, in
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
The right way to do it is call .decode() on the byte string.
>>> '\xe2\x98\x83'.decode('utf8')
u'\u2603'
Just be sure you don't call it on a unicode string by mistake, or you'll get very confused:
>>> u'foo'.decode('utf8')
u'foo'
>>> u'\u2603'.decode('utf8')
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.4/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2603' in position 0: ordinal not in range(128)
Hey! That error says "encode", but we asked for it to decode!
It turns out that since the utf8 decoder expects bytes as input, but we gave it unicode... Python helpfully tries to convert the unicode characters into bytes, using the default encoding! This is why the first decode call succeeds - all the characters in it can be converted to ASCII bytes.
Is there any hope?
Since Unicode was added to Python in 2.1, there's been a way to get Python 3's behavior. The site module in the Python standard library sets Python's default encoding on startup. If edited to use "undefined" instead of "ascii", the above examples all fail instead of converting:
>>> u'foo' + baz
Traceback (most recent call last):
File "<stdin>", line 1, in
File "/usr/lib/python2.6/encodings/undefined.py", line 22, in decode
raise UnicodeError("undefined encoding")
UnicodeError: undefined encoding
So why can't we switch to this today? Well, for one thing, lots and lots of code depends on this implicit encoding/decoding. Even code in the standard library:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/re.py", line 190, in compile
return _compile(pattern, flags)
File "/usr/lib/python2.6/re.py", line 243, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/lib/python2.6/sre_compile.py", line 506, in compile
p = sre_parse.parse(p, flags)
File "/usr/lib/python2.6/sre_parse.py", line 672, in parse
source = Tokenizer(str)
File "/usr/lib/python2.6/sre_parse.py", line 187, in __init__
self.__next()
File "/usr/lib/python2.6/sre_parse.py", line 193, in __next
if char[0] == "\\":
File "/usr/lib/python2.6/encodings/undefined.py", line 22, in decode
raise UnicodeError("undefined encoding")
UnicodeError: undefined encoding
Not to mention loads of existing third-party apps and libraries. So what can we do? Fortunately, Python allows registering new codecs. I've written a slight variation on the ASCII codec, ascii_with_complaints, which preserves the default Python behaviour, but also produces warnings.
>>> import re
>>> re.compile(u'\N{SNOWMAN}+')
/usr/lib/python2.6/sre_parse.py:193: UnicodeWarning: Implicit conversion of str to unicode
if char[0] == "\\":
/usr/lib/python2.6/sre_parse.py:418: UnicodeWarning: Implicit conversion of str to unicode
if this and this[0] not in SPECIAL_CHARS:
/usr/lib/python2.6/sre_parse.py:421: UnicodeWarning: Implicit conversion of str to unicode
elif this == "[":
/usr/lib/python2.6/sre_parse.py:478: UnicodeWarning: Implicit conversion of str to unicode
elif this and this[0] in REPEAT_CHARS:
/usr/lib/python2.6/sre_parse.py:480: UnicodeWarning: Implicit conversion of str to unicode
if this == "?":
/usr/lib/python2.6/sre_parse.py:482: UnicodeWarning: Implicit conversion of str to unicode
elif this == "*":
/usr/lib/python2.6/sre_parse.py:485: UnicodeWarning: Implicit conversion of str to unicode
elif this == "+":
<_sre.SRE_Pattern object at 0xb76d6f80>
Hopefully this will be useful as a tool for ferreting out those Unicode bugs waiting to break your application, as well.
