Category: Programming

Complex (2)

I have generally found NoSQL to be a disaster. Like agile processes, it allows you to dispense with certain disciplines, but for use at scale and over time it requires you to engage in substitute disciplines. Too often these are not practiced. From a recent work chat with minor adaptation:

Data hygiene is crucial. I wouldn’t be opposed to broader NoSQL/JSON use if we used JSON schemas wherever appropriate, but at that point it is probably simpler to flatten the data into tables.

A good schema is a species of defensive code; e.g., you can have higher confidence that the value you are reaching for is actually there no matter how old the document.

See also: Complex

Travis and Pylint

For awhile my team has had Travis setup to run Pylint (as well as several other lints) against our code base. However, because we didn’t start this practice from the beginning, the number of warnings was a bit daunting. We told ourselves that we would fix this over time, and set our script to always return 0 so that Travis would be happy.

Then I read: Why Pylint is both useful and unusable, and how you can actually use it. I was inspired by this to try my hand at reducing Pylint’s scope. However, I took a different approach. Instead of disabling all checks and enabling them incrementally, I adjusted our script to check only for fatal and error findings in Pylint. Pylint encodes in its exit status what levels of messages were issued.

Here is my approach:

# Fail Travis build if Pylint returns fatal (1) | error (2)
if [ $(($rc & 3)) -ne 0 ]; then
    echo "Pylint failed"
    exit 1
else
    echo "Pylint passed"
    exit 0
fi

The number of errors found by Pylint was much more manageable than the full set of messages it produced. We were able to correct these problems easily, and move to addressing warnings and other messages incrementally over time.

SmugMug uploader

[SmugMug]I’ve written a small Python script to upload pictures to a SmugMug gallery. I love SmugMug and use it extensively for family photos. I’m using this script for my personal use because it’s much simpler and much less of a resource hog than a browser-based uploader, and also because it was a fun exercise to try out the SmugMug API. You can run this script as follows to upload one or more files:

python upload.py gallery-name picture-file-name . . .

On Windows I’ve set up a desktop shortcut pointing to the script, and I can drag and drop a pile of picture files onto the icon and it will upload away. I’ve tested it using both Python 2.5 using simplejson, and also using Python 2.6 which has simplejson built in. Earlier versions of Python may require you to change the import of hashlib to md5, and change the hashlib.md5() invocation to a md5.new() invocation. You’ll also need to modify the script to contain your email address and SmugMug password, and obtain a SmugMug API key for your own development use, but this is a very painless process. Here is the script:

#!/usr/bin/python

##########
# Requirements: Python 2.6 or
#               simplejson from http://pypi.python.org/pypi/simplejson
##########

EMAIL='...'
PASSWORD='...'

##########
APIKEY='...'
API_VERSION='1.2.2'
API_URL='https://api.smugmug.com/services/api/json/1.2.2/'
UPLOAD_URL='http://upload.smugmug.com/photos/xmlrawadd.mg'

import sys, re, urllib, urllib2, urlparse, hashlib, traceback, os.path
try    : import json
except : import simplejson as json

if len(sys.argv) < 3 :
  print 'Usage:'
  print '  upload.py  album  picture1  [picture2  [...]]'
  print
  sys.exit(0)

album_name = sys.argv[1]
su_cookie  = None

def safe_geturl(request) :
  global su_cookie

  # Try up to three times
  for x in range(5) :
    try :
      response_obj = urllib2.urlopen(request)
      response = response_obj.read()
      result = json.loads(response)

      # Test for presence of _su cookie and consume it
      meta_info = response_obj.info()
      if meta_info.has_key('set-cookie') :
        match = re.search('(_su=S+);', meta_info['set-cookie'])
        if match and match.group(1) != "_su=deleted" :
          su_cookie = match.group(1)
      if result['stat'] != 'ok' : raise Exception('Bad result code')
      return result
    except :
      if x < 4 :
        print "  ... failed, retrying"
      else :
        print "  ... failed, giving up"
        print "  Request was:"
        print "  " + request.get_full_url()
        try :
          print "  Response was:"
          print response
        except :
          pass
        traceback.print_exc()
        #sys.stdin.readline()
        #sys.exit(1)
        return result

def smugmug_request(method, params) :
  global su_cookie

  paramstrings = [urllib.quote(key)+'='+urllib.quote(params[key]) for key in params]
  paramstrings += ['method=' + method]
  url = urlparse.urljoin(API_URL, '?' + '&'.join(paramstrings))
  request = urllib2.Request(url)
  if su_cookie :
    request.add_header('Cookie', su_cookie)
  return safe_geturl(request)

result = smugmug_request('smugmug.login.withPassword',
                         {'APIKey'       : APIKEY,
                          'EmailAddress' : EMAIL,
                          'Password'     : PASSWORD})
session = result['Login']['Session']['id']

result = smugmug_request('smugmug.albums.get', {'SessionID' : session})
album_id = None
for album in result['Albums'] :
  if album['Title'] == album_name :
    album_id = album['id']
    break
if album_id is None :
  print 'That album does not exist'
  sys.exit(1)

for filename in sys.argv[2:] :
  data = open(filename, 'rb').read()
  print 'Uploading ' + filename
  upload_request = urllib2.Request(UPLOAD_URL,
                                   data,
                                   {'Content-Length'  : len(data),
                                    'Content-MD5'     : hashlib.md5(data).hexdigest(),
                                    'Content-Type'    : 'none',
                                    'X-Smug-SessionID': session,
                                    'X-Smug-Version'  : API_VERSION,
                                    'X-Smug-ResponseType' : 'JSON',
                                    'X-Smug-AlbumID'  : album_id,
                                    'X-Smug-FileName' : os.path.basename(filename) })
  result = safe_geturl(upload_request)
  if result['stat'] == 'ok' :
    print "  ... successful"

print 'Done'
# sys.stdin.readline()

I am donating this script to the public domain. You are welcome to use and modify it as you please without conditions. I’d appreciate hearing about your experience with this script or any changes and improvements you’ve made; please leave a comment. Thanks!

Update 2010-07-20

Since I first posted this, I’ve updated it as follows:

  1. Add a Content-Type header of ‘none’. This is to workaround a bug in the SmugMug API.
  2. Use basename() to send only the file’s basename for X-Smug-FileName.
  3. Rewrite safe_geturl() to loop up to five times if the upload attempt fails. I’ve found that uploading is surprisingly unreliable, and re-attempting the upload generally works fine.
  4. Add a commented call to readline() at the end of the script. In my case, I run my script by dragging files onto an icon on my Windows desktop, which causes it to run in a DOS window and vanish when done. If you uncomment this line, it will wait for you to press Enter when it is done uploading. You’ll be able to see any files that weren’t uploaded successfully.

Update 2010-11-28

SmugMug made a recent change to their API’s login behavior which broke this script. While the new login behavior is not documented in the API docs, the fix is apparently to use a session cookie along with the session ID. While it’s a bit of a kludge, I’ve updated the script above to save this cookie in a global variable and submit it on subsequent requests.

Update 2011-06-24

I’ve fixed a bug in the script causing it to wrongly report a failure for certain requests that don’t send back the session cookie. The fix involves testing whether a set-cookie header was returned before accessing the header.

Update 2013-10-01

Version 1.2.0 of the SmugMug API has stopped working, so I have updated the script to use version 1.2.2 of the API.

Editor Color Scheme

Recently Slashdot featured a discussion of the best color scheme for programming. From that discussion I’ve discovered the zenburn color scheme, and have switched to it. I like the fact that the background is not stark black; the reduced contrast feels easier on my eyes.

Here are some resources I found from that discussion for color schemes:

What color scheme do you prefer for programming? There are several others at the vim color scheme test, above, that I’m also interested in trying out.

See also: Programming Fonts

GCC binary conditional

I recently ran into a nifty GCC extension to the C/C++ language, the binary conditional:

z = x ?: y;

At first glance this looks like the C++ ternary operator:

z = x ? y : w;

But notice that this new operator above is binary — it has only two parameters.  Unlike the dash and greater-than symbols in the C arrow operator (pointer -> member), GCC does not require that the binary conditional’s question mark and colon be adjacent, but you should probably write them adjacently to better distinguish them from the ternary operator.

But what does the binary conditional do?  In short, it is analagous to the Perl || operator, the Python or operator, and the Ruby || operator.  It evaluates to the value on the left unless the value on the left evaluates to false, in which case it evaluates to the value on the right.

x y x ?: y
0 0 0
0 80 80
NULL 0x16E212B4 0x16E212B4
15 0 15
15 20 15
0x16E212BC 0x16E212E8 0x16E212BC

You may wonder why the C || operator can’t be used for this same purpose. The reason for this is that C’s || operator performs a pure logical or operation: it always collapses the result value to 0 or 1. For example, the expression 80 || 0 evaluates to 1, not to 80. However, the expression 80 ?: 0 evaluates to 80.

That’s pretty nifty, although ?: is certainly a bit unfortunate; it’s not obvious from looking at the operator what it should do. Worse, it appears that the binary conditional is unique to GCC. I’ve tried this with several other C/C++ compilers without success.

There is, however, a more portable way to accomplish the same thing. Instead of writing x ?: y, you can write the equivalent x ? x : y. This is a little less concise, but it has the advantage that any skilled C programmer can immediately understand what it does. And it is more portable.

Richard Scarry and hexadecimal

When you read a hexadecimal number out loud, how do you pronounce the letters?

At my workplace, I’ve grown used to our custom of pronouncing the letters using the Joint Army/Navy Phonetic Alphabet standardized in 1941. The letter digits are pronounced Able, Baker, Charlie, Dog, Easy and Fox. Under this scheme, the hexadecimal number 0x7F8D3BC0 would be pronounced “Seven Fox Eight Dog Three Baker Charlie Zero.” This was disorienting to me at first, but after eight years this is now so natural that this is how I pronounce the digits in my mind even if I’m not speaking them.

We’ve started collecting Richard Scarry’s children’s books. Richard Scarry writes with a degree of detail and whimsy that holds an adult’s interest — much like old-school Sesame Street. (How far it has fallen — modern-day Sesame Street is much too postmodern, pluralistic, saccharine and juvenile for my taste. I console myself by searching for old Sesame Street clips on Youtube.) Recently I was amused and pleased to discover that one of Richard Scarry’s characters is named Able Baker Charlie! What a strange juxtaposition of worlds for me — programming and children’s books.

Able Baker Charlie is a mouse. He is a baker, and assists Baker Humperdink, a pig. Despite his small size, Able Baker Charlie is capable assisting with any step of the baking process, from stoking the oven, to mixing the dough, to putting loaves in the oven, and even delivering bread around Busytown. Below you may see a picture of Able Baker Charlie ably distributing French baguettes to Louie’s Restaurant.

Richard Scarry served in the U. S. Army during World War II. No doubt this is the source of the Able Baker Charlie aptonym. It still gives me a chuckle every time we read it.