A stylised depiction of Rob

Primary Unit

Quit to Linux

Usually I dislike commenting on stuff in the news, instead keeping to topics I have at least some direct experience of — even if I’m frequently documenting my own lack of experience.

But I believe that a lot of people are misinterpreting Gabe Newell’s keynote speech at LinuxCon, in which he said that Linux is the “future of gaming”. In just over a day that Ars article has drawn over 400 comments but only a handful have come close to what I believe is the real issue.

(If you’ve only read summaries, I urge you to watch Gabe’s talk. It’s only about 20 minutes long.)

Most of the reaction concerns Windows 8, its software store and tightly-controlled computing environments in general. This misses the point.

Here’s my take: Valve are preparing for the next decade of PC gaming.

But first let’s do a bit of groundwork that will address the common arguments against Gabe’s talk (or what those people imagine he said) and put us in position to view the wider landscape.

Valve began considering the direction of the PC industry “several years ago”. When Gabe talks about “some bad thinking” it’s clear that he means iOS and the tightly-controlled App Store. It should be obvious that Steam would not have been able to exist in that environment.

Steam started on Windows. So where’s Windows now? Doing OK. Windows 8 is a tad confused but doing fine. The Windows Store that plenty of commenters think that Valve fears will eat its lunch? A bit like the Mac App Store. Nothing to worry about.

But where is Windows going? To answer that you must consider where Microsoft is going.

No-one knows where Microsoft is going. I don’t think that Microsoft even knows where Microsoft is going.

Its 13-year CEO has just been fired and it desperately needs to unhook itself from the twin Windows and Office morphine drips. Selling Windows licences to PC-makers hits a snag when PC unit sales fall off a cliff and it’s difficult to sell Office to people on platforms where they can’t buy it even if they wanted to.

If Microsoft is going to survive it needs to get out of that business, though it may be a case of having to destroy the village in order to save it. (Hat-tip.)

So where does that leave Windows? It’s not clear but the mishmash of tablet and desktop OS that is Windows 8, coupled with the Windows Store, suggests Microsoft wants to head in Apple’s direction. It is this which I think Gabe was referring to when he branded Windows 8 a “catastrophe”.

The end of Windows as we know it? Impossible! Well maybe not. Microsoft’s in a bind and who knows what it is willing to throw into the engine in order to keep the wheels turning?

Valve makes a lot of money from Steam but it’s dependent on having a stable environment where it can more or less do what it likes, ship whatever client software it wants with a store run in a way that suits the business and its customers. Not the rules of a third party.

Linux provides that platform. It is stable in the sense that it has no master which can suddenly make a self-interested change that makes life difficult for Valve.

This is not about the quality of Windows 8, how much business the Windows Store is going to take from Steam or an objection to giving a controlling party some say or a monetary cut.

This is about moving to a platform in which Steam can exist without a guillotine blade hanging over its head.

However, Valve can’t just snap its fingers and have everyone switch to Linux (as many, many commenters noted). But let me tell you a story.

My Steam account is 10 years old this month. I signed up in September 2003. At the time Steam was a joke. There was a horrendous GIF doing the rounds that unfortunately summed up the feelings of a lot of users. Steam was rubbish. About the only thing you could do was play Counter-Strike 1.6.

When Valve announced that you would be able to download Half-Life 2 instead of buying a physical copy I wondered why anyone would want to spend several days downloading a game when you could just nip to the shops.

A decade later I have about 200 games registered on my Steam account and I haven’t bought a disc copy of a game in two years (and that was an exceptional case).

The Linux user experience can often suck. Few ordinary gamers use it and few developers release games for it. But these are solvable problems.

Valve’s plan for Linux is not for now, it’s for the future. But it starts now.

Get your US-ASCII out of my face

TL;DR If you’re having problems with Ascii encoding errors in OS X scripts when everything should be UTF-8 make sure you’ve explicitly set $LANG in your ~/.bash_profile or ~/.bashrc

When I was writing the script in yesterday’s post I came across this bizarre text encoding problem:

UnicodeDecodeError: 'ascii' codec can't decode byte
0xe2 in position 5: ordinal not in range(128)

Ascii what? We’re all Unicode now, don’t you know? If I’m taking in UTF-8 text, keeping it in a Unicode string, and then writing it out to UTF-8 encoded text files where is goddamn Ascii creeping in?

This first occurred at work, while I was trying to work, so my horrendous hack solution was to switch the temporary files into binary mode and encode the two strings into UTF-8 before writing them out. (This was when the script was still a text filter, so the problem was only on the back end.)

That’s a horrible hack. Bytes! Yuck! So in my next attempt I explicitly opened the files with a UTF-8 encoding. Which was still gross. And totally unnecessary. This is Python 3, everything’s Unicode, all the defaults are UTF-8, why am I having to specify the encoding at every point?

This was the point at which I found a relevant Stack Overflow question. Unfortunately the answer there was to tinker with Python’s start-up files in a way that was always discouraged and had since been removed from Python 3.

Hm. But what’s this about sys.setdefaultencoding? So I started poking around with this terrible little script:

python:
#!/usr/bin/env python3
import os
import sys
import locale
entire_file = open(os.environ['BB_DOC_PATH'], 'r')
print(os.environ)
print(entire_file.encoding)
print(sys.getdefaultencoding())
print(locale.getpreferredencoding())

For the last three lines, which print the encodings used, I got:

US-ASCII    # File encoding
utf-8       # sys.getdefaultencoding
US-ASCII    # locale.getpreferredencoding

So the system setting wasn’t the culprit. But why is the system encoding different to the locale, which seems to determine the default encoding with which files are opened?

Locale refers to the environment variables in my Terminal, but when I try echo $LANG it’s set to en_GB.UTF-8 — surely that’s not the problem?

WRONG.

As this thread eventually made me realise, I didn’t actually have $LANG set to anything — it was all smoke and mirrors that cleared away when I ran the script through BBEdit. (I also had a similar problem with Applescript’s do shell script and it looks like Sublime Text users have also run into it.) Checking the os.environ output from the script above confirmed it: no $LANG.

The root of the problem is that if a locale isn’t set it defaults to the C locale — meaning US-ASCII in OS X. The solution is to explicitly set $LANG in your .bash_profile:

bash:
export LANG="en_GB.UTF-8"

I’ve also got that line in ~/.bashrc for safety, but do bear in mind that some shells will ignore both ~/.bash_profile and ~/.bashrc.

You could set the other LC_* keywords as well, if you like, but $LANG should be enough according to the man page for locale:

LANG         Used as a substitute for any unset LC_* variable.  If LANG
             is unset, it will act as if set to "C".  If any of LANG or
             LC_* are set to invalid values, locale acts as if they are
             all unset.

Update 2013-09-21

I’ve just re-watched Ned Batchelder’s great Pragmatic Unicode video and while setting the $LANG environment variable is a handy solution on your own machines, it’s not a good fit when you’re writing a program that will run on systems where you can’t control the default encoding.

In such cases it makes sense to adopt his “Unicode sandwich” approach of explicitly decoding I/O and then working on the resulting Unicode.

Solo diff

At work we edit the reporters’ copy in plain text files using TextWrangler, duplicating their original at the bottom of the file before we start subbing. It’s a nice, quick way of having something to refer back to without the hassle of using (and training people to use) a real VCS.

It began as a loose convention, with different people having their own way of marking off the original copy. To make it completely painless, and to bring in a standard method, I wrote this short Python script a couple of months ago:

python:
#!/usr/bin/env python3
from sys import stdin
orig = stdin.read()
dupe = "{0}\n\n\n\n#### Original Copy ####\n\n\n\n{0}".format(orig)
print(dupe)

I installed it to everyone’s text filters folder — TextWrangler provides the document’s text (or the selection) to text filters via stdin — and people tend to use it.

We hired two new subs over the summer and I got into the habit of using the “Compare Two Front Windows” command to check their subbed copy against the original, so I could offer them specific advice.

But manually creating two new windows and pasting in the text can be a pain — you end up juggling lots of windows and it’s easy to mix up the copy so that the “older” pane is showing the subbed text.

Thankfully both of the new people use the duplication script consistently, so it’s easy to use a regular expression to split the file into original and subbed versions. Armed with this knowledge, I wrote a script to automate the comparison, using TextWrangler’s twdiff command-line tool.

The code below is written for BBEdit, which I use at home. Except for BB_DOC_PATH, I swap out bb for tw in the script used at work.

python:
 1:  #!/usr/bin/env python3
 2:  
 3:  import os
 4:  import re
 5:  import subprocess
 6:  
 7:  os.chdir('/tmp')
 8:  with open(os.environ['BB_DOC_PATH']) as full_file:
 9:    sub_copy, orig_copy = re.split(r'#{2,} original copy #{2,}',
10:                                   full_file.read(), flags=re.I)
11:  
12:  with open('bb_orig_copy', 'w') as orig_file:
13:    orig_copy = orig_copy.strip() + '\n'
14:    orig_file.write(orig_copy)
15:  
16:  with open('bb_subbed_copy', 'w') as sub_file:
17:    sub_copy = sub_copy.strip() + '\n'
18:    sub_file.write(sub_copy)
19:  
20:  subprocess.call(['bbdiff', 'bb_orig_copy', 'bb_subbed_copy'])

The entire file is split into subbed and original sections in lines 9 & 10, and on lines 14 & 18 each of those is written to a file (in /tmp — line 7). The leading and trailing whitespace is stripped from each and a newline added in order to clean up the comparison.

The regex on lines 9 & 10 uses a variable number of hash marks and the case-insensitivity flag because on occasion I forget to duplicate the copy first, edit the story, and then go back and fill in the separator by hand.

All of the file interaction is handled in with blocks (context managers), which automatically close the files once the block’s statements are executed. Not a big issue in a script this size, but good practice.

Finally we call the diff utility with the names of the two temporary files.

I’ve got this saved in TextWrangler’s scripts folder. Initially I wrote it as a text filter, splitting the contents provided on stdin. But since filters replace the document (or selection) the text had to be written to stdout at the end of the script so that TextWrangler didn’t delete everything.

And it’s also not a filter, so I was pleased to find out that several environment variables are set when a script is invoked, one of which is the path to the file that the script was called on.

I’ve tested the script with Python 2.7.5 and it seems to work fine, but with the differences in its Unicode handling I can’t say for certain that some nasty encoding gremlin won’t raise its head.

Promptless Mercurial

My time-wasting with Git and Mercurial continues apace — although I think I’m largely done learning how to use each — and I’m at the point where I start fussing over trivial details.

This time it’s prompts. I’ve had Git’s status in my prompt for quite a while, since it comes bundled, but as I’ve been spending so much time mucking about with Mercurial I decided to give Steve Losh’s hg-prompt a try.

It’s nice and you use it in a similar way to Mercurial’s own templates, which are very powerful. But it does introduce a slight delay at the command line, which I haven’t noticed with the Git prompt. Maybe I’ll decide this isn’t a big deal and change my mind in the future, but at the moment it was just enough to put me off.

I also like to keep my prompt nice and short. Right now it’s just the basename of the working directory and the dollar sign (or hash if root). The prompt in my home directory is just:

~ $

So I’ve decided to go promptless and ditch the additions for both Git and Mercurial. This should help get me in the habit of checking each kind of repository in a similar way, instead of glancing at the prompt for Git and typing commands for Mercurial.

If you use the -sb options with Git’s status command it gives you a short status with the current branch on top, which is adequate for my needs.

By default there isn’t a Mercurial command to do this, but it’s trivial to create one using an alias — and one that’s superior to git status -sb. Here’s what I’ve got in my .hgrc file (line breaks added for clarity):

now = !$HG log -r . --template
      "{label('log.changeset', rev)}
      {label('branches.active', '    {branch}')}
      {label('bookmarks.current', '    {bookmarks}')}\n" ; $HG status

The exclamation mark at the start tells Mercurial to interpret it as a shell alias — this lets us call hg status at the end. The alias calls hg log for the parent of the working directory (the commit you currently have checked out).

Using the template system we ask for that changeset’s revision number, branch and any bookmarks associated with it. The label stuff surrounding rev, branch and bookmarks colours the output — you can otherwise ignore it.

Revision numbers are a convenient but strictly local way of referring to changesets. I include the branch name and bookmarks because you can use either (or both) to branch your work; there’s no point just printing the named branch if you’re using bookmarks, and vice versa.

When called inside a repository, hg now prints something like this:

hgrepo $ hg now
20    experimental    super-duper

If there are any changes in the working directory that’s followed by the usual output of hg status, like so:

Image of the output of hg now when there are changes in the working directory.

Terminal countdown

I’ve got some annual leave booked for the very end of September, which is pretty far away still — but I wanted to know exactly how many days I have to go before my break.

Usually I would just query Wolfram Alpha but, since I’ve been spending a fair amount of time at the command line, I wondered if date in Unix could give me the answer.

It’s easy to ask date to add or subtract an amount of time:

bash:
$ date -v +2w   # Now plus two weeks
Sat 31 Aug 2013 13:14:20 BST

However there’s no built-in way to find the difference between two dates. So I wrote one. Now originally I was going to walk through that shell script but there were enough problems that I rewrote it in Python this morning.

But before I get to the Python code I want to discuss the shell script.

It took a date in ISO format (because) and ran it through date to get back a Unix timestamp. This was then subtracted from the timestamp for the current date and divided by the number of seconds in a day to give the number of days difference.

Unfortunately writing the script gave me a few headaches:

  • Unix timestamps can break. Passing a date before 1902 caused date to choke on my machine, and this varies from system to system.
  • Commands you expect to be reliable can differ wildly between systems. I used egrep to check the format of the provided date and used \d as the digit wildcard. But on my work machine, which runs Mac OS 10.6, you can only use \d with grep -P. That breaks grep on 10.8 at home. The nasty solution was to use an explicit range: [0-9]. Gross.
  • The bash scripting syntax is insane. Want to check if stdin is connected to an interactive terminal, so you know you won’t be getting data on stdin? Use -t 0. Meanwhile in Python: sys.stdin.isatty().

The timestamp problem is an edge case as days aren’t a useful measure of time past a certain, short distance, so it was mainly the reliability and syntax complaints that pushed me to rewrite the script. I don’t consider \d to be exotic and with the removal of the -P option in newer versions of grep how can you expect to use it in portable code?

As for the syntax, while I was pleased I’d managed to lump together enough parts to make a working program, it’s not clear by any means — and this is just a 45-line wrapper around date! String comparisons in bash get to use != and == but numbers are stuck with -ne and -eq? Oh god it’s horrible.

Anyway, let’s move on to the Python script:

python:
 1:  #!/usr/bin/env python3
 2:  
 3:  from __future__ import print_function
 4:  import sys
 5:  import os
 6:  import re
 7:  from datetime import datetime
 8:  
 9:  if len(sys.argv) == 2:
10:    date = sys.argv[1]
11:  elif not sys.stdin.isatty():
12:    # Called in a pipeline
13:    date = sys.stdin.read().rstrip('\n')
14:  else:
15:    # Print usage message if no date is supplied
16:    script = os.path.basename(sys.argv[0])
17:    print('Usage:  {0} YYYY-MM-DD\n'
18:          '        prints number of days until or since the given date'
19:          .format(script))
20:    sys.exit(64)
21:  
22:  if not re.match(r'\d{4}-\d{2}-\d{2}', date):
23:    print('Abort:  your date is not in YYYY-MM-DD format')
24:    sys.exit(65)
25:  else:
26:    then = datetime.strptime(date, '%Y-%m-%d').date()
27:  now = datetime.now().date()
28:  days = (then - now).days
29:  
30:  if days == 0:
31:    print("That's today!")
32:    sys.exit(0)
33:  
34:  msg = str(days) + ' day'
35:  if days not in [1, -1]:
36:    msg += 's'
37:  
38:  if days < 0:
39:    msg = msg.lstrip('-') + ' ago'
40:  else:
41:    msg += ' ahead'
42:  print(msg)

It works in much the same way as the shell script. (In case anyone’s wondering, the print_function import is for Python 2 compatibility — just change the shebang line to plain python.)

Lines 9–13 handle input from an argument or stdin, and a usage message is printed (lines 14–20) if nothing is passed.

Next is the regular expression to check the date’s format (using \d! Yes!). A warning is printed and the script exits if it doesn’t match, otherwise a datetime object is created using a strptime format string in the same way you would with date at the terminal. date() is immediately called on the object (line 26) because we want to work with full days only.

One nice thing about Python’s datetime module is timedelta, which lets you subtract one date from another and returns the difference — we do that in line 28 and extract the days attribute.

From here we’re just constructing the message printed to the user, with a check for the current date and then whether “days” should be plural. If the date is in the past we trim the minus sign in line 39, then stick on an appropriate adverb and print.

All told, we end up with this:

$ days 2013-12-25
130 days ahead
$ days 2013-01-01
228 days ago

I’ve posted both the Python and shell versions as a gist if you’d like to compare the two.

Commit summary length hooks

Despite planning to take a break from doing computer work in my spare time, I’ve haven’t stopped playing with Mercurial and Git.

Right now I’m learning towards Mercurial (and am using it to track these blog posts) but I will still be using Git, not least because of GitHub.

So when I wrote my first commit hook I did it in both flavours. It’s a simple Python script that rejects a commit if the first-line summary is too long (over 50 characters) or too short (10 characters). The idea is to make sure hg log and git log (--short|--oneline) are informative but brief.

Mercurial

python:
 1:  #!/usr/bin/env python3
 2:  
 3:  import os
 4:  import sys
 5:  from termtools import colour
 6:  from subprocess import check_output
 7:  
 8:  node = os.environ["HG_NODE"]
 9:  message = check_output(["hg", "log", "-r", node,
10:                          "--template", "{desc}"]).decode()
11:  summary_length = len(message.splitlines()[0])
12:  
13:  if summary_length > 50:
14:    print(colour("! Changeset summary is too long (> 50c)", "red"))
15:    sys.exit(1)
16:  elif summary_length < 10:
17:    print(colour("! Changeset summary is too short (< 10c)", "red"))
18:    sys.exit(1)

Many of Mercurial’s hook types provide useful data as shell variables. In line 8 we get the node (hash) of the changeset about to be committed and use that on line 9 & 10 to fetch the commit message.

The call to check_output on those lines is a little confusing, but it’s essentially the equivalent of this terminal command:

hg log -r $HG_NODE --template {desc}

Line 11 finds the length of the first line of the commit message, and lines 13–18 checks its length. If the summary is too long or short a warning is printed and the script exits with a non-zero status to tell Mercurial to reject the commit.

The colour function called on lines 14 and 17 is used to turn the warning text red. I’ll explain it below.

Git

python:
 1:  #!/usr/bin/env python3
 2:  
 3:  import sys
 4:  from termtools import colour
 5:  
 6:  message = open(sys.argv[1])
 7:  summary_length = len(message.readline().splitlines()[0])
 8:  
 9:  if summary_length > 50:
10:    print(colour("! Commit summary is too long (> 50c)", "red"))
11:    sys.exit(1)
12:  elif summary_length < 10:
13:    print(colour("! Commit summary is too short (< 10c)", "red"))
14:    sys.exit(1)

Git’s hooks include one specifically for the commit message, and when it is called Git feeds it a file containing the message — that’s what we open on line 6.

We read a single line from the file on line 7, but still use the splitlines method as in the Mercurial version in order to gracefully handle newlines at the end — I don’t want to unconditionally chop the last character in case it’s actually part of the summary.

The if block at the end is almost identical to the Mercurial version, save for the “commit” replacing “changeset”. This is just me trying to avoid blindly using Git terminology when discussing Mercurial.

Usage

I include the hook in my main Mercurial config file (~/.hgrc), which means it applies to all repositories:

[hooks]
pretxncommit.summary_length = path/to/hook/file.py

The first part of the second line is the hook type, which sets when the script should be called, and the bit after the dot is a custom name — this lets you have multiple scripts attached to a certain hook type.

With Git things are a little more tricky, as the script (or a link to it) must appear in each repository’s .git/hooks/ directory. As far as I’m aware there isn’t a built-in way of setting global hooks. I’ve toyed with the idea of writing my own little program to help set up hooks but I’m not sure if I can be bothered. I guess it will come down to how often I use hooks (and Git itself).

(Benjamin Meyer’s git-hooks tool is designed to help with this, but it’s written as a shell script and I can’t quite get my head round it.)

Terminal colouring

In both scripts the printed warnings are run through a colour function, which is something I whipped up with exactly this situation in mind.

python:
 1:  def colour(str, col, background=False):
 2:    COLOURS = {'black': "30",
 3:               'red': "31",
 4:               'green': "32",
 5:               'yellow': "33",
 6:               'blue': "34",
 7:               'magenta': "35",
 8:               'cyan': "36",
 9:               'white': "37"}
10:    escape = "\x1b"
11:    reset = escape + "[0m"
12:    reverse = ";7"
13:  
14:    colour_code = COLOURS[col.lower()]
15:    start = escape + "[" + colour_code
16:    if background:
17:      start += reverse
18:    start += "m"
19:  
20:    return (start + str + reset)

It wraps ANSI colour codes around a string and returns it. The action occurs in lines 14-18, where control codes are concatenated with the colour number fetched from the dictionary on lines 2-9. There’s an option to colour the background instead of the text, using the reverse code. The colour codes get stuck on the front of the string and a reset code is put on the end.

You could keep this function in the hook script itself, but I keep it inside a module in Python’s search path (hence the import statements in the hooks).

If you want more features the termcolor module on PyPI allows for setting different foreground and background colours as well as bold text, underlines, etc.

I decided against using termcolor as I wanted to write my own as a test and because most of those options end up looking very ugly or are unnecessary. Coloured and reversed text is enough for me.

Hazel gating with Mercurial

Since I’ve been playing around with Mercurial I thought it would only be fair to write another version of my Hazel gating snippet.

The basic mechanism is the same (parsing the status output) and the code is barely changed:

bash:
 1:  #!/bin/bash
 2:  
 3:  cd $(dirname $1)
 4:  FILENAME=$(basename $1)
 5:  HGSTATUS=$(hg status -c "$FILENAME")
 6:  if [ "$HGSTATUS" == "" ]; then
 7:    exit 1 # Dirty
 8:  else
 9:    exit 0 # Clean
10:  fi

An important difference to note is that this script checks that the file is clean (through the -c flag on line 5), not that the file is not dirty as the Git version does. This requires the conditional to be reversed as an empty string means that the file has been modified in some way.

As I mentioned previously in my post about branch comparison, I’ve spent some time recently to learn more about Git and Mercurial. This was both to improve my pretty basic Git skills and see why Daniel Jalkut keeps banging on about Mercurial (go to the 28-minute mark).

When I learnt about hooks (Mercurial, Git) I was a little worried they would invalidate the time I’d spent writing the status gating snippet. But while hooks are very handy and powerful, there’s plenty off actions that Hazel is more suited to or more convenient for — this just acts as a nice safety check so it doesn’t step on your toes while you’re working.

Easy branch comparison with Mercurial

I’ve spent a lot of time recently learning Git and Mercurial, and trying to decide which one I prefer and want to keep using. Both are nice, but one thing I like about Git is how easy it is to quickly see which commits are on one branch but not another.

The commands below show commits on the feature branch that aren’t present on master, and are all equivalent:

git log feature ^master
git log ^master feature
git log master..feature

Unfortunately doing the same with Mercurial requires a lot more typing:

hg log -r "ancestors('feature') and not ancestors('master')"
hg log -r "::'feature' and not ::'master'"

Which is a shame, and a bit odd since Mercurial has the incoming and outgoing commands to see which commits are coming in from (or going out to) a remote repository. I like to think of the comparison above as a local equivalent, to see which commits are going to be pulled across branches in a merge.

Thankfully this can be put right with a quick alias in your .hgrc file. This is what I’ve got in mine:

[alias]
compare = log -r "ancestors('$1') and not ancestors('$2')"

It’s general enough to allow comparisons between any two commits. Named branches and bookmarks both work fine, as do revision numbers and commit hashes.

What’s nice about this alias is that any hg log options you’d like to use can just be stuck on the end as usual, since it’s just a shortcut to that command. For example these work as you’d expect:

hg compare feature master
hg compare feature master --graph
hg compare feature master -l 20 -p

Sunny with a chance of Python

Learning Python is the next goal that I’ve set myself, though I have put it off for a bit while I improve my maths skills.

But I do have one little Python script that I’ve been playing with for a while, which I first wrote to dip my toe in and see if I liked the language. (I do, very much so).

It’s incredibly basic but complements my automatic weather script, which saves a lot of time and tedium at work, by fetching a text summary of the weather for the entire UK.

python:
 1:  #!/usr/bin/env python3
 2:  """Get a 3 to 5 day outlook for the UK from the Met Office."""
 3:  
 4:  from urllib.request import urlopen
 5:  from bs4 import BeautifulSoup as bs
 6:  from subprocess import Popen, PIPE
 7:  
 8:  api_url = ("http://datapoint.metoffice.gov.uk/public/" +
 9:             "data/txt/wxfcs/regionalforecast/xml/515")
10:  api_key = "Your Datapoint API key here"
11:  
13:  def get_weather():
14:    """Retrieve and parse the text summary forecast"""
15:    raw_xml = urlopen(api_url + "?key=" + api_key)
16:    outlook = bs(raw_xml).find(id="day3to5").get_text()
17:    return outlook
18:  
19:  if __name__ == "__main__":
20:    # Encode and pipe outlook to pasteboard
21:    outlook_bytes = get_weather().encode()
22:    Popen("pbcopy", stdin=PIPE).communicate(outlook_bytes)

As shown at the top, this is Python 3 code. I have a version written in Python 2 which is very similar. It uses the built-in ElementTree XML parser instead of the nicer Beautiful Soup used in this script as I both wanted to get my head around the interface and for reasons to do with the Python version at work.

Lines 8–10 set up variables needed to access the Met Office API, which is done in the get_weather function below.

Beautiful Soup is used to parse the API response in line 16. I love its dead simple method for finding elements, especially compared to ElementTree’s insistence that you provide the XML namespace:

python:
pattern = (".//{www.metoffice.gov.uk/xml/metoRegionalFcst}" +
           "Period[@id='day3to5']")

My favourite part of the script is also the newest. If it is run directly from the command line (line 18) it pipes the forecast to the clipboard using pbcopy. I knew Python could start external processes but I was blown away by how easy this was. Previously the script printed the forecast and I used a shell alias to send it to the pasteboard. Yuck. This is much better.

The way Python 3 handles strings requires an encoding step in line 21, whereas Python 2 can just send a string through communicate — as it’s already bytes. This caught me out at first but otherwise it was incredibly simple to implement.

So, like I said, a trivial program but one that has made me eager to learn more Python.

Five different kinds of grey

My post from a couple of weeks ago about the changes to this site briefly mentioned that despite having this domain for about five years I’d never been satisfied with its design.

I’ve actually lost count of the number of times I’ve switched themes but, spread across Blogger, WordPress and Tumblr, it must be at least 10. None of them stuck.

Part of that is likely that I never stuck at it, never put in the effort to make the posts worthwhile. But maybe one of the reasons why is because I knew I wasn’t going to be happy with how the results looked.

But I am happy with the design now. Having put in the time to learn HTML and CSS (with some help), it wasn’t too hard to make something that I’m at least a little proud of. And now I’d like to share some of the thinking behind it.

It’s pretty simple — largely because I can handle this kind of simplicity well with my limited skills. But it’s just a personal blog and I don’t want to overcomplicate matters.

I took a lot of cues from Dr Drang, Marco Arment and John Gruber, all of whom have great sites with top work framed with a straightforward design.

I’d love to claim some kind of inspiration but, when it came down to it, I made a fairly arbitrary decision to stick the meta stuff on the left, the content in a big long column on the right and then call it a day. On smaller screens the meta column sits on top of the content and the social links disappear entirely on particularly thin screens (phones) as they’re not essential. My earlier sketches had elements boxed or ruled off, but a few technical difficulties pulling this off convinced me to mostly just use whitespace.

All text is set in Rooney. A while ago I had got obsessed with Tisa and had initially planned to use that. But I eventually had trouble reading long Instapaper articles in it — the vertical strokes tended to run together when I was tired. After searching the Typekit library for a while I settled on Rooney, which is similar in many ways to Tisa — but with enough subtle differences and little quirks to avoid the problem I had with Tisa. Also as lovely as Tisa is it can be a bit cold and clinical, while I find that Rooney feels more warm and welcoming.

Paragraphs are limited to roughly 65 characters (about 12 words) a line, though I’m still tweaking the exact length. I had initially had an idea to accommodate margin notes in the extra space but technical hurdles, a lack of anything to put there and fear of looking incredibly pompous made me decide against it. A remnant of this can be seen in the blockquotes, code blocks and coloured boxes I use, which all stretch the full width of the column. Code blocks are the only element that make use the extra space, but the others get it because I like how their background colours can help break up long columns of text.

The coloured boxes I mentioned above I call “flags” and came about when I decided I wanted a way to flag updates to posts. (Again, an influence of Dr Drang: see the updates in this post.) I then generalised the style to be applicable to a range of things, such as flagging download links and warnings.

The flags use icons from Symbolset’s Standard font, which I use in a few places on the site — probably not enough to justify the weight of an extra web font but I like the font a lot and had been itching to use it for months. Similarly, the social icons in the sidebar use Symbolset’s Social font. I use the circular version to give them an even appearance.

It took me a long time to pick out the dark blue used as the left-hand border of the flag above, which is also the primary link colour. Generally, I’m not great at picking colours but Spectrum helps enormously. Before using Spectrum I’d end up with a murky mess, but now I’m actually very pleased with the palette I have created. It’s mostly blue and grey, with splashes of green and red where appropriate.

The image below shows all the colours used on the site, apart from those used for code. (The two oranges are only used for the RSS link.)

The 14 colours used on robjwells.com

The syntax highlighting for code is another thing I lifted from Dr Drang, which is to say it is highlight.js combined with the Doc’s script to conditionally invoke the highlight functions and add line numbers.

I’ve uploaded my slightly rewritten code as a GitHub gist. The two main changes are the introduction of the module pattern (using the RJW global variable as a container) and removing the dependency on JQuery. The syntax colour scheme is based on my custom one for BBEdit.

Speaking of JQuery, nothing in my design requires it. But it’s one of those things Tumblr injects into your blog anyway. This is a bit of a shame, as I’d tried to keep page weight down (five web font files notwithstanding). The three largest downloads on this page are all things injected by Tumblr — the fourth is my 135KB Typekit CSS file.

While this doesn’t bother me a huge amount, it does add another reason for me to move away from Tumblr — almost certainly to a static system of some kind. (I have a fantasy of learning enough Python to write my own.) While I imagine that’s quite a while off, it should be a fairly easy transition — there are no dynamic elements to move over and the “Tumblrised” HTML file I have would be simple to switch to another template system.

I haven’t spent much time on Tumblr-specific features, either. The only post types I accommodate are text, links and photos. I did the work for the latter two largely because I already have posts of that type that I wish to keep, not because I plan to make more of them in the future. Again, making it easier to switch away at some point in the future.

It’s not that I dislike Tumblr, just that I’m not a great fit for Tumblr and Tumblr’s not a great fit for me and the kind of site that I want to run.

Right, we’re over 1,000 words now so it’s time to wrap it up. If you’ve made it this far, thanks! If you have any comments or questions be sure to send me a message on Twitter or App.net.