I ran a script this afternoon that died mys­te­ri­ous­ly without any output. It was using SQLAlchemy to query all the rows from a large table so that they could be trans­formed into JSON Lines to be loaded into Elas­tic­search. When I reran my script, I noticed this time that something had printed Killed at the very end.

A little research convinced me that the OOM Killer was the likely assassin. I looked in /var/log/kern.log and I found that my process had used up almost all of the 8GB on this system before being killed.

The query had to be the problem. A little more research led me to augment my …continue.

On Mac/Linux, os.path.join is an alias for posixpath.join, which always joins path segments with /. On Windows, os.path.join is an alias for ntpath.join, which always uses \. When dealing with URLs, we always want forward slashes, regardless of platform, so posixpath.join should be used to build URL paths.

Running:

from __future__ import print_function

from six.moves.urllib_parse import urljoin as abs_urljoin
from posixpath import join as path_urljoin

def urljoin(site, path):
    return abs_urljoin(site, path)

def test_join(site, path):
    result = urljoin(site, path)
    print("'{0}' + '{1}'\n\t-> '{2}'".format(site, path, result))
    return result

local_path = path_urljoin('2016', '07', '12', 'release', 'index.html')

test_join('https://www.example.com', 'foo/bar/quux.js')
test_join('https://www.example.com', local_path)
test_join('https://www.example.com/', local_path)
test_join('https://www.example.com/prefix', local_path)

Yields:

'https://www.example.com' + 'foo/bar/quux.js'
  

Python 2.6 introduced the format method to strings. In general, format is now the preferred way to build strings instead of the old % formatting operator.

One exception is with the logging module, where the best practice is to use %s and %d. Why? First, %s is the idiomatic way to use logging, which was built years before format was introduced. Second, if there's a literal % in the in­ter­po­lat­ed values, logging will be unhappy, since there won't be cor­re­spond­ing arguments in the call. It won't fall over, since “The logging package is designed to swallow exceptions which occur while logging in production. This is so that errors which occur while handling logging …continue.

Yesterday, while at PyCon, I whipped up a quick, brute-force answer to the House­Ca­nary PyCon2016 Progamming Challenge in a few minutes. That was sufficient to pass the first two test cases and win me a very pretty House­Ca­nary t-shirt.

The answer ran in O(n⁴) time, so it failed miserably on the larger problem sets in the third and fourth cases. I mulled it over and came up with an O(n²) solution that runs in reasonable time on the larger problem sets. On the second test case, input1.txt, runtime drops from 5.2s to 0.2s.

I submitted my new answer. I'll learn on Monday if I won the speed challenge.

I made a number of updates to the Fly­ing­Cloud Doc­u­men­ta­tion tonight. I hope to give a lightning talk about Fly­ing­Cloud at PyCon on Monday evening or Tuesday morning, and I put together some slides for that too.

I like to use io.StringIO rather than the older cStringIO.StringIO, as it's Python 3–ready io.StringIO is also a context manager: if you use it in a with statement, the string buffer is au­to­mat­i­cal­ly closed as you go out of scope.

I tried using io.StringIO with unicodecsv, as I wanted to capture the CSV output into a string buffer for use with unit tests. unicodecsv is a drop-in re­place­ment for Python's built-in csv module, which supports Unicode strings.

with io.StringIO() as csv_file:
    write_csv_rows(csv_file)
    lines = csv_file.getvalue().split('\r\n')
    return lines[:-1]  # drop empty line after trailing \r\n

It failed horribly with TypeError: unicode argument expected, …continue.

I rarely use doctests, but I do have some code that uses them.

Although I still mostly write Python 2, I usually import several features of Python 3:

from __future__ import unicode_literals, print_function, absolute_import

Un­for­tu­nate­ly uni­code_lit­er­als doesn't play well with doctests.

The following code will pass with python2 -m doctest demo.py, but not with python3:

from __future__ import unicode_literals, print_function, absolute_import

def upper(s):
    """
    Convert `s` to upper case.

    >>> upper('Hello!')
    u'HELLO!'
    """
    return s.upper()

Python 3 complains:

Failed example:
    upper('Hello!')
Expected:
    u'HELLO!'
Got:
    'HELLO!'

The …continue.

Despite being a bona fide per­for­mance expert—I spent a couple of years as the Per­for­mance Lead for Mi­crosoft­'s IIS web server product about 15 years ago—I still forget to measure rather than assume.

I wrote some code today that imported nearly 300,000 nodes into a graph from a 500MB XML file. The code was not par­tic­u­lar­ly fast and I assumed that it was the XML parser. I had been using the built-in streaming parser, cEle­ment­Tree iterparse. I assumed that using the lmxl iterparse would make the code faster. It didn't.

Then I had the bright idea of tem­porar­i­ly disabling the per-node processing, which left only the XML parsing. Instead of …continue.

I wanted to raise Python's IOError for a file-not-found condition, but it wasn't obvious what the parameters to the exception should be.

from errno import ENOENT

if not os.path.isfile(source_file):
    raise IOError(ENOENT, 'Not a file', source_file)
with open(source_file) as fp:
    return fp.read()

IOError can be in­stan­ti­at­ed with 1, 2, or 3 arguments:

IOError(errno, strerror, filename)

These arguments are available on the errno, strerror, and filename attributes of the exception object, re­spec­tive­ly, in both Python 2 and 3. The args attribute contains the verbatim con­struc­tor arguments as a tuple.

IOError(errno, strerror)

These are available on the errno and strerror attributes of the exception, re­spec­tive­ly, in both Python 2 and …continue.

Last night, Adam Porad and I were one of five teams pitching our startups at the PuPPy-organized PyCon Startup Row Pitch Night:

Techstars Seattle and PuPPy [Puget Sound Pro­gram­ming Python] presents PyCon Startup Row Pitch Night. The time has come again for you, the members of PuPPy, to select Seattle’s startup rep­re­sen­ta­tive to travel to PyCon in Portland to represent our Python community and startup scene at the annual conference produced by the Python Software Foundation.

We were pitching MetaBrite and our technology that captures receipts, yielding receipt in­for­ma­tion to users and onsumer insights. We use Python ex­ten­sive­ly—we've written 120,000 lines of Python code for web services, web apps, machine learning, image processing, and …continue.