Foundations of Python Network Programming 978-1-4302-3004-5

CHAPTER 5 ■ NETWORK DATA AND NETWORK ERRORS
But you cannot put such strings directly on a network connection without specifying which rival
system of encoding you want to use to mix your characters down to bytes. A very popular system is UTF-
8, because normal characters are represented by the same codes as in ASCII, and longer sequences of
bytes are necessary only for international characters:
>>> elvish.encode('utf-8')
'Nam\xc3\xa1ri\xc3\xab!'
You can see, for example, that UTF-8 represented the letter ë by a pair of bytes with hex values C3
and AB.
Be very sure, by the way, that you understand what it means when Python prints out a normal string
like the one just given. The letters strung between quotation characters with no leading u do not
inherently represent letters; they do not inherently represent anything until your program decides to do
something with them. They are just bytes, and Python is willing to store them for you without having the
foggiest idea what they mean.
Other encodings are available in Python—the Standard Library documentation for the codecs
package lists them all. They each represent a full system for reducing symbols to bytes. Here are a few
examples of the byte strings produced when you try encoding the same word in different ways; because
each successive example has less in common with ASCII, you will see that Python's choice to use ASCII
to represent the bytes in strings makes less and less sense:
>>> elvish.encode('utf-16')
'\xff\xfeN\x00a\x00m\x00\xe1\x00r\x00i\x00\xeb\x00!\x00'
>>> elvish.encode('cp1252')
'Nam\xe1ri\xeb!'
>>> elvish.encode('idna')
'xn--namri!-rta6f'
>>> elvish.encode('cp500')
'\xd5\x81\x94E\x99\x89SO'
You might be surprised that my first example was the encoding UTF-16, since at first glance it seems
to have created a far greater mess than the encodings that follow. But if you look closely, you will see that
it is simply using two bytes—sixteen bits—for each character, so that most of the characters are simply a
null character \x00 followed by the plain ASCII character that belongs in the string. (Note that the string
also begins with a special sequence \xff\xfe that designates the byte order in use; see the next section
for more about this concept.)
On the receiving end of such a string, simply take the byte string and call its decode() method with
the name of the codec that was used to encode it:
>>> print '\xd5\x81\x94E\x99\x89SO'.decode('cp500')
Namárië!
These two steps—encoding to a byte string, and then decoding again on the receiving end—are
essential if you are sending real text across the network and want it to arrive intact. Some of the
protocols that we will learn about later in this book handle encodings for you (see, for example, the
description of HTTP in Chapter 9), but if you are going to write byte strings to raw sockets, then you will
not be able to avoid tackling the issue yourself.
Of course, many encodings do not support enough characters to encode all of the symbols in certain
pieces of text. The old-fashioned 7-bit ASCII encoding, for example, simply cannot represent the string
we have been working with:
>>> elvish.encode('ascii')
Traceback (most recent call last):
...
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 3: ordinal
not in range(128)
72