You are converting strings back and forth between C and Python, but the C encoding

is of a dubious or unknown nature. For example, perhaps the C data is supposed to be

UTF-8, but it’s not being strictly enforced. You would like to write code that can handle

malformed data in a graceful way that doesn’t crash Python or destroy the string data

in the process.

Here is some C data and a function that illustrates the nature of this problem:

/* Some dubious string data (malformed UTF-8) */

/* Output character data */

In this code, the string sdata contains a mix of UTF-8 and malformed data. Neverthe‐

less, if a user calls print_chars(sdata, slen) in C, it works fine.

Now suppose you want to convert the contents of sdata into a Python string. Further

suppose you want to later pass that string to the print_chars() function through an

extension. Here’s how to do it in a way that exactly preserves the original data even

though there are encoding problems:

/* Return the C string back to Python */

/* Wrapper for the print_chars() function */

if (!PyArg_ParseTuple(args, "U", &obj)) {

return NULL;

}

if ((bytes = PyUnicode_AsEncodedString(obj,"utf-8","surrogateescape"))

== NULL) {

return NULL;

}

PyBytes_AsStringAndSize(bytes, &s, &len);

print_chars(s, len);

Py_DECREF(bytes);

Py_RETURN_NONE;

}

If you try these functions from Python, here’s what happens:

>>> s = retstr()

>>> s

>>> print_chars(s)

>>>

Careful observation will reveal that the malformed string got encoded into a Python

string without errors, and that when passed back into C, it turned back into a byte string

that exactly encoded the same bytes as the original C string.

This recipe addresses a subtle, but potentially annoying problem with string handling

in extension modules. Namely, the fact that C strings in extensions might not follow the

strict Unicode encoding/decoding rules that Python normally expects. Thus, it’s possible

that some malformed C data would pass to Python. A good example might be C strings

associated with low-level system calls such as filenames. For instance, what happens if

a system call returns a broken string back to the interpreter that can’t be properly

decoded.

Normally, Unicode errors are often handled by specifying some sort of error policy, such

as strict, ignore, replace, or something similar. However, a downside of these policies

is that they irreparably destroy the original string content. For example, if the malformed

data in the example was decoded using one of these polices, you would get results such

as this:

>>> raw = b'Spicy Jalape\xc3\xb1o\xae'

>>> raw.decode('utf-8','ignore')

>>> raw.decode('utf-8','replace')

>>>

The surrogateescape error handling policies takes all nondecodable bytes and turns

them into the low-half of a surrogate pair (\udcXX where XX is the raw byte value). For

example:

>>> raw.decode('utf-8','surrogateescape')

>>>

Isolated low surrogate characters such as \udcae never appear in valid Unicode. Thus,

this string is technically an illegal representation. In fact, if you ever try to pass it to

functions that perform output, you’ll get encoding errors:

>>> s = raw.decode('utf-8', 'surrogateescape')

>>> print(s)

>>>

However, the main point of allowing the surrogate escapes is to allow malformed strings

to pass from C to Python and back into C without any data loss. When the string is

encoded using surrogateescape again, the surrogate characters are turned back into

their original bytes. For example:

>>> s

>>> s.encode('utf-8','surrogateescape')

>>>

As a general rule, it’s probably best to avoid surrogate encoding whenever possible—

your code will be much more reliable if it uses proper encodings. However, sometimes

there are situations where you simply don’t have control over the data encoding and

you aren’t free to ignore or replace the bad data because other functions may need to

use it. This recipe shows how to do it.

As a final note, many of Python’s system-oriented functions, especially those related to

filenames, environment variables, and command-line options, use surrogate encoding.

For example, if you use a function such as os.listdir() on a directory containing a

undecodable filename, it will be returned as a string with surrogate escapes. See

Recipe 5.15 for a related recipe.

PEP 383 has more information about the problem addressed by this recipe and surro

gateescape error handling.