Category: Agile

Complex (2)

I have generally found NoSQL to be a disaster. Like agile processes, it allows you to dispense with certain disciplines, but for use at scale and over time it requires you to engage in substitute disciplines. Too often these are not practiced. From a recent work chat with minor adaptation:

Data hygiene is crucial. I wouldn’t be opposed to broader NoSQL/JSON use if we used JSON schemas wherever appropriate, but at that point it is probably simpler to flatten the data into tables.

A good schema is a species of defensive code; e.g., you can have higher confidence that the value you are reaching for is actually there no matter how old the document.

See also: Complex

Complex

I’ve joked for awhile that EDR and similar systems like CrowdStrike or Carbon Black would become Skynet. Or, more likely, a tool of international espionage and cyber–warfare. It doesn’t feel good to be vindicated. (Now imagine someone accomplishing this with a major browser or password manager application.) Complex and highly interconnected systems are difficult to make stable, resilient, or secure; and cannot possibly be made anti-fragile. (This is a lesser reason why I miss my old pickup truck.) I’m not very excited about Kubernetes for the same reason. It’s also partly why I’m not very excited about artificial intelligence; but additionally because analysis of data, whether by machine or by human, does not automatically involve either wisdom or decisiveness (see also Edwin Friedman and Nassim Taleb).

Crossposted on I gotta have my orange juice.

Occam’s razor

On my nontechnical blog I reflect on simplicity:

I’ve been appealing to Occam’s razor lately as a rule for evaluating architectural decisions and their tradeoffs. In particular, architectural decisions must take into account not only ideal considerations, but also a team’s capacity to develop, maintain, and support these decisions. Simplicity has its own rewards regardless of the size of your team, but the smaller the team, the more aggressively you must press for that simplicity. Don’t multiply entities unnecessarily!

Rest

Rest

We considered how stress and self-discipline result in growth and strength, whether that is physical, mental, emotional, or spiritual. However, an important corollary of this is that intervals of rest are needed so that we are able to recover stronger instead of ending up progressively worn down.

From nature and our own experience we can see that this rest needs to happen on several cycles. There is a daily rest (1/3 of our time is spent sleeping), a wise principle of weekly rest (one day out of seven), and a yearly rest (winter, vacations). We could even consider the wisdom of longer cycles of rest (e.g., taking sabbatical every 7 to 11 years as many universities practice for their faculty, and as Intel has done).

These principles apply not only to organic life but also to organizations. While agile principles and techniques do increase team efficiency and productivity, it is a mistake to think that agile’s goal is continuous apparent productivity. There are a number of shatterings of continuous apparent productivity that are necessary to healthy agile product development. It is important to brainstorm, learn, conduct retrospectives, take time to refactor, experiment and evaluate alternatives . . . and also to rest. Paradoxically, all of these ways of taking time to slow down often help to improve your team’s long-term productivity.

Obviously our individual daily, weekly, and annual cycles of rest help with the health of our agile team. But the team itself should also be engaging in rest. There are many possibilities here, including team outings and shared meals, team training, and planning for gap sprints or gap weeks to focus on lighthearted or experimental work (what if I rewrote this in Clojure, Haskell, or Racket). In keeping with the spirit of agile, the team should evaluate its own need for rest and plan appropriate kinds of rest.

Crossposted to I gotta have my orange juice.

Travis and Pylint

For awhile my team has had Travis setup to run Pylint (as well as several other lints) against our code base. However, because we didn’t start this practice from the beginning, the number of warnings was a bit daunting. We told ourselves that we would fix this over time, and set our script to always return 0 so that Travis would be happy.

Then I read: Why Pylint is both useful and unusable, and how you can actually use it. I was inspired by this to try my hand at reducing Pylint’s scope. However, I took a different approach. Instead of disabling all checks and enabling them incrementally, I adjusted our script to check only for fatal and error findings in Pylint. Pylint encodes in its exit status what levels of messages were issued.

Here is my approach:

# Fail Travis build if Pylint returns fatal (1) | error (2)
if [ $(($rc & 3)) -ne 0 ]; then
    echo "Pylint failed"
    exit 1
else
    echo "Pylint passed"
    exit 0
fi

The number of errors found by Pylint was much more manageable than the full set of messages it produced. We were able to correct these problems easily, and move to addressing warnings and other messages incrementally over time.