[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

RFC 1738 allows passwords in URLs, in the form <scheme>://<username>:<password>@<host>:<port>/<url-path>. Although passwords are deprecated by RFC 3986 and other newer RFCs, it's oc­ca­sion­al­ly useful. Several important packages in the Python world allow such URLs, including SQLAlchemy ('post­gresql://scott:tiger@localhost:5432/my­data­base') and Celery ('amqp://guest:guest@localhost:5672//'). It's also useful to be able to log such URLs without exposing the password.

Python 2 has urlparse.urlparse (known as urllib.parse.urlparse in Python 3 and six.moves.url­lib_­parse.urlparse in the Six com­pat­i­bil­i­ty library) to split a URL into six components, scheme, netloc, path, parameters, query, and fragment. The netloc cor­re­sponds to <user>:<password>@<host>:<port>.

Un­for­tu­nate­ly, neither Python 2 nor 3's urlparse properly handle the userinfo (username + optional password in the netloc), as …continue.

New MetaBrite Dev Blog post, Ob­fus­cat­ing Passwords in URLs in Python.

I wanted to turn a list like ['*.zip', '*.pyc', '*.log'] into ['--exclude', '*.zip', '--exclude', '*.pyc', '--exclude', '*.log'].

A simple list com­pre­hen­sion doesn't work as desired:

In [1]: excludes = ['*.zip', '*.pyc', '*.log']

In [2]: [('--exclude', e) for e in excludes]
Out[2]: [('--exclude', '*.zip'), ('--exclude', '*.pyc'), ('--exclude', '*.log')]

The trick is to use a nested com­pre­hen­sion:

In [5]: [arg for pattern in excludes
             for arg in ['--exclude', pattern]]
Out[5]: ['--exclude', '*.zip', '--exclude', '*.pyc', '--exclude', '*.log']

I worked on a Bash script today that sets up various pre­req­ui­sites for our build. We need a recent version of Docker but our Bamboo build agents are running on Ubuntu 14.04, which has a very old version of Docker. The script upgrades Docker when it's first run. The script may be run more than once during the lifetime of the agent, so the second and subsequent calls should not upgrade Docker.

Basically, I wanted

if $DOCKER_VERSION < 1.9; then upgrade_docker; fi

Un­for­tu­nate­ly, it's not that easy in Bash. Here's what I came up with.

install_latest_docker() {
    if docker --version | python -c "min=[1, 9]; import sys; ↩
v=[int(x) 

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

I spent some time today struggling with setuptools, trying to make a Python source package not only include a data file, but also install that file.

├── MANIFEST.in
├── README.md
├── my_stuff/
│   ├── bar.py
│   ├── foo.py
│   ├── __init__.py
│   └── quux.py
├── models/
│   └── long_ugly_name_20151221.json
└── setup.py*

include models/*.json

from setuptools 

At this month's PuPPy (Puget Sound Pro­gram­ming Python) Meetup, I heard a brief mention of Python f-strings as a new feature coming in Python 3.6.

In essence, they offer a simpler, more versatile method of string formatting and in­ter­po­la­tion over existing methods. F-strings can include not only symbol names but Python ex­pres­sions within strings. With str.format, you can write 'Hello, {name}'.format(name=some_name). You can control various aspects of how name is formatted, such as being centered within a field—see PyFormat and Python String Format Cookbook for ex­am­ples—but no more complex expression is allowed between the braces.

Herewith some examples of f-string ex­pres­sions drawn from PEP 0498:

>>> date = datetime.date(1991, 10, 12)
>>> f'{date} was on a 

On Stack­Over­flow, someone wanted to print triangles in Python in an M-shape. Various clumsy solutions were offered.

Here's mine which uses the left- and right-jus­ti­fi­ca­tion features of str.format.

• "{0}+++{0}".format('Hello!') produces two copies of the zeroth argument to format (here 'Hello!'), separated by three plusses: Hello!+++Hello!.
• "{:<4}".format('x') left-justifies 'x' in a 4-character field; i.e., 'x   '.
• "{:>4}".format('x') right-justifies 'x' in a 4-character field; i.e., '   x'.
• "{:>{}}".format('x', width) right-justifies 'x' in a width-character field.
• 'ab' * 4 yields 4 copies of 'ab'; i.e., 'abababab'.

Putting them together:

.. code:: pycon

>>> WIDTH = 4
>>>
>>> for a in range(1, WIDTH+1):
...     print("{0:<{1}}{0:>{1}}".format('*' * a, WIDTH))
...
*      *
**   

…continue.

I was debugging a filtering directory walker (on which, more to follow) and I was trying to figure out the mysterious suffix that fnmatch.translate appends to its result, \Z(?ms).

fnmatch.translate takes a Unix-style glob, like *.py or test_*.py[cod], and translates it character-by-character into a regular expression. It then appends \Z(?ms). Hence the latter glob becomes r'test\_.*\.py[cod]\Z(?ms)', using Python's raw string notation to avoid the backslash plague. Also, the ? wildcard character becomes the . regex special character, while the * wildcard becomes the .* greedy regex.

A Stack­Over­flow answer partially explains, which set me on the right track. (?ms) is equivalent to compiling the regex with re.MULTILINE | re.DOTALL. The re.DOTALL modifier makes the . special character match any character, including …continue.

I'm doing some Python profiling and I wanted to use the Run­SnakeRun utility to view the profile data. Un­for­tu­nate­ly, that's not straight­for­ward on Mac OS X if you use a virtualenv, and it's even less easy if you're using the Python installed by the Homebrew (brew) package manager.

There are several problems:

• Installing wxPython on OS X 10.10. This is the cross-platform GUI API toolkit used by Run­SnakeRun.
• Getting wxPython installed in a virtualenv
• Running wxPython apps in a virtualenv on the Mac.

We use Nose to run unit tests. I noticed that we had some tests that weren't being run, and it took me some time to work out why. Eventually, I found this checklist, which told me to "make sure the files in your tests directory are not executable".

A quick chmod -x *.py later and nosetests project/tests suddenly found all the tests.

Now that I know what to look for, I found this in the Nose docs:

It is important to note that the default behavior of nose is to not include tests from files which are executable.

--exe: Look for tests in python modules that are executable. Normal behavior …continue.