Programmers: how to make the systems guy love you

Edit: There’s now a counterpart to this entitled “Sysadmins: how to make the programmers love you“.

As a coder working for an organisation you are focused on building something to meet a defined spec.  This spec will say things like “background should be blue”, “tabulated interface for user management” and “should not crash”.  It may come with a set of images stitched together by a UI designer of how the site/app/thing is expected to look – either crayoned on the back of a fag packet or more likely mocked up in Photoshop so the UI programmers can pull it apart again for the logos and buttons.  In a really good spec you’ll find a storyboard defining how it should behave, ready-made content and all the copy up front.  We wish.

Things are not always like this.  Sometimes in projects where the end result is not entirely defined (“doomed ones”) the spec may be little more than a set of wireframes and a brief hand-wavy description about what it is to do.  Hmm.

But so far so good.  It is positive that a spec exists at all and that we have some consensus on what we are making.  From an engineer’s perspective the better defined and more rigid a spec is the better – the more agreement we can have about the location of a target, the higher our chance of hitting it.

The most common problem with specs is when they’re written by someone without an idea of the things programmers really need to know.  They know what they’d like the product to do but overlook “make it robust” or “make it easy to setup on our CentOS 5.5 servers”.  To non-engineers these objectives are obscure and here a problem arises – when they go unmentioned it’s easy to fulfill the specification while producing an article that’s a total pain in the butt.  A project may be declared a success but still it can be excruciating for people to maintain, compile, release, troubleshoot or in any other way interact with.

Here’s how to avoid it…

1) Communication

Almost all instances of the systems guy biting someone could be avoided by one simple thing.  Communication.  Early in your project try taking him out for a coffee and saying something like this:

“Hey Alex, our team is working on this awesome new site for our wise, beneficent client called flying-cars-for-pets.com.  It’s done in PHP5 on the Cake framework.  The spec says it’s gonna launch quietly but we overheard the client saying they plan to email 15 million people on day 1 to drive traffic to the site.”

You just saved the guy who runs your servers a whole world of pain.  Now he has a rough idea of the technologies you’re using and how much traffic the site will see.  Not quite as good as real performance & bandwidth figures but hey, at least there’s a fighting chance of getting it hosted right.  You even told him the hostname.

If the systems person (this time a girl – I know some awesome lady systems folk) hasn’t taken a holy vow of silence she should respond:

“Thanks man.  I’ve hosted CakePHP sites before and they run just fine on our platform.  I know it works well upto version 1.2.8.  If you’re expecting a lot of traffic the code needs to be efficient – have a link to this cool page I found on optimizing Cake sites.  Let me know when you have a pre-release ready and I’ll load test it to flush out any performance problems that’ll hurt us on launch day.”

Start a dialogue early to avoid hostage negotiations later.  If somebody’s paying the systems person especially to run your hosting environment they probably know a thing or two about getting apps live.  Tap that knowledge early.

Here’s what not to do.

“Oh hai!  U can haz code for new site.  We been riting it 6 weeks but not tell u.  It need <interminable_list_of_libraries> and dedicated support services on other hosts and these custom PHP modules compiled up from (buggy) source and it needs root on the DB server cos it was okay on our dev vm’s yeah?  And the code is shit-slow but that can be ur problem.  And we did tell client it go live tmrow for million of users.  We go to pub nau.  Have fun setting it up!”

This has happened way too many times.  The developers hand you a hairball.  They know it’s a mess, they know it’s going to blow up but never mind because once release 1.0 is tagged in Subversion it ain’t their problem anymore.  Classy, folks.

2) Documentation

Yup.  You can save a huge, huuuuuge amount of heartache simply by documenting the thing that you’ve built.

Really this is a part of communication.  But it’s such a broad subject it deserves a section of its own.

Most IT folk would rather be stung to death by killer bees than document the stuff they’ve made.  Once I was in that camp too – I saw requests for documentation as an affront, a barely-concealed hint at my mortality, a suggestion that the things I made might be less than 110% perfect.  [Matt, if you read this I'm sorry about the documentation wars we fought five years ago.  I was being a total ass.]

That was before my career turned toward working with larger teams to host their stuff and spending less time writing code.  Running the infrastructure seemed like more fun.  You get to play with screwdrivers and build great big powerful machines.  And on these awesome machines runs… other people’s code.  Suddenly seeing the other side of the coin convinced me how vital it is that all but the most trivial of technical objects (even a Linux host, a backup system, even your goddamn network) should come with a few instructions for those who aspire to operate them reliably.

There are a bunch of common excuses for not documenting an artifact:

• “Documentation wasn’t mentioned in the spec” – Neither were ‘should not grow wings and fly away to Jupiter with the CEO’s daughter’ and ‘must not be constructed out of cow dung’.  Yet somehow you managed to meet these unwritten requirements.  The spec’s author overlooked them because they considered it common sense, not because they dream of  a product constructed from the effluvia of ruminant beasts.  Likewise documentation.  Here’s another sensible exercise – next time you see a spec without docs as a deliverable ask the spec’s owner “do you want no documentation?”.  Unless they are a masochist or never expect to see the object again after it’s built they will say “whoops”.  [Note to project managers: add "documentation" to your spec, assign time to it and be careful to explain what you mean]
• “Nobody else will need to maintain it, I’ve got everything I need in my head.” – So you plan never to have a hangover, die, leave the company, get promoted or forget any piece of knowledge ever?  Are you the amazing immortal omniscient programmer?
• “It’s not my job” – so whose job is it?  The magical documentation pixie in the corner?  Unless you’re fortunate enough to have an official documentation-writer on the team and you’ve given them all the information they need it is your job.
• “It’s so simple it doesn’t need documenting” – you think this because you understand the thing you have made.  To you it is simple; to others it may not be.  As a simple test find a knowledgeable stranger (i.e. someone who can code but hasn’t seen your project) and give them a copy of your app.  Without any pointers see how long it takes them to get it working from scratch.
• “My code is self-documenting” – yeah, and monkeys might fly out of my butt!  Tell that to the poor sap who has to reconfigure your app for the UAT server 12 months after you left the company, or at five in the morning fix the never-ending DB query you missed.  See how much use your self-documenting code is to someone who hasn’t days to spend reading it all.
• “We’ll get round to it later” – you won’t.  This is a form of technical debt; you genuinely believe you’ll get round to it later but non-urgent tasks like writing the instructions will always get preempted by other work.  If your project is a success you’ll face an immediate demand to write more code; if it’s a failure the thing will (or at least should) be canned outright.  Documentation doesn’t get written unless you do it alongside the work. [Agile folks: this means docs should happen before the end of the sprint.  It certainly won't happen afterward.]

…and now a few of the real reasons:

• “It’s boring.  I’d rather be writing code” – yes.  Even in the most fulfilling job there will probably be less interesting tasks.  Sorry about that.  Take pride in your work and do them anyway – you want to make the finished article as good as you can, don’t you?
• “We can get away without doing it” – if you don’t plan to be with the organization for long and nobody is there to force you to do a good job and you lack pride in your work maybe you can.  But six months down the road when the object you made blows up for entirely unforeseen reasons those left behind to pick up the pieces will find out where you live and burn your house down.
• “Job Security” – I have a hard time believing anyone is such an asshole they actually behave like this.  What a negative way to keep one’s job.  But apparently it happens; if you’re the one guy sick enough to think this way your world must be an awful place.  I pity you.
• “Management won’t assign time to it” – okay, so this one isn’t your fault but still you must push to fix it.  Likely your manager is under pressure to produce results from someone who wouldn’t know a maintainable product if it bit them on the ass.  As a technical person it’s still your responsibility to educate.  Ask pointed questions like “won’t it be a problem if this crashes and no-one knows what to do?”.  If a complex technical object is intended to be maintained for any length of time it must come with some instructions.  Without them you’re being asked to do half a job.  Maybe there are political reasons for this – e.g. the management will not be the one tearing our his hair looking for the slow query at 5am or expects to be promoted before it becomes a problem.

You can always find reasons not to do something you know you oughta.  A corporate bureaucracy will supply you with dozens.  Fight them and help each other build solid, reliable applications – not ticking timebombs.

So what should I be documenting?

After all that you’ll be thinking “Shit, he wants me to rewrite War and Peace”

Nah.  Less is more.  Overly verbose documentation is unwieldy: change one line of code and you need to rewrite fifteen pages of instructions nobody cared about anyhow.  The objective is to make the organisation more productive and less accident-prone – it’s not like you’re getting paid by the column inch.

Key to this is imagining yourself in the shoes of whoever will run and maintain your product.  If it’s a website think of future coders and your operations folks.  If it’s an iPhone app ditto coders.  And if you’re in charge of a startup dream ahead to getting bought – it sure won’t hurt when you say to Google’s acquisition team “and here are the instructions you’ll need to integrate our product”.

Here are some of the usual suspects on planet web-app:

• Overview. What’s it for?  You might expect everyone to know but trust me, as a guy who’s inherited ten-year-old webservers teeming with hundreds of random sites the guy who becomes responsible for your product may not.  And if he doesn’t know he’ll probably wind up trampling on it with his size-ten sysadmin boots while trying to patch the machine up to date and lock out the Romanian spammers who’ve been using it to blast out spam for V1@GrA for months using the undocumented mail.pl CGI script you left in place.  Not that this has happened to me before, oh no.
• The architecture from a 10,000ft perspective.  What goes where?  How do the components speak to each other?  Where is data stored?  Which component goes wrong the most?  What external resources does it use?  All this is obvious to an established developer on a project but to someone joining a project the knowledge is worth its weight in gold.  A diagram here is worth a thousand words.
• Dependencies. What does it need to work?  Give just enough information for a competent person to recreate the environment.  If you’re feeling really charitable suggest some software versions.  Perhaps a simple list – “Apache 2.2, mod_rewrite, MySQL >= 5, PHP >= 5, PHP GD library, PEAR geolocation > 1.1″.  Add any non-standard instructions for their setup.
• Configuration. What resources (e.g. database) does it use and in what config files are these defined?  If they are hardcoded I will personally come around your house and nail a dead animal to the door.  Where can we find a copy of the initial schema to setup a new dev instance?  If your application is served from an application server (e.g. Apache, JBoss) what specialized setup is required?
• Common problems. What are the five things most likely to break it?  If somebody is trying to fix a problem and you left them a solution on page 5 of the manual they’re going to be really, really grateful.  Once it’s working again they’ll buy you a beer to say thanks for all the trouble you saved them.  Seriously, they will.

Don’t go overboard.  Give just enough information for a knowledgeable outsider to get your product going and to troubleshoot it when something has gone wrong.  Even if you do plan to support this thing forever think of all the time you’ll save by giving colleagues a place to check before they bug you.

Documentation Done Right on the Daily WTF says it better than I can.

How do I document?

It’s a matter of convention in your organisation.  If no convention exists, start one.  So long as it’s sensible your manager will likely thank you for saving them the bother.  If one does just go with the flow.  And because it’s a convention stick to it.

Avoid a proliferation of document stores.  If there’s one single place you can search for a bit of knowledge you urgently need (“NetApp full”, “<myapp> mysql configuration” etc.) you’re onto a winner.  You will be able to work more efficiently and in a crisis nobody will be left scratching their head muttering “ug, mungo not know how thing work”.

Here are some sensible places you might put documentation:

• A wiki. If the organisation is small likely they will have only one (or zero – ask for one?).  Wikis are great; they are searchable and unlikely to be mislaid.  You can link to other pages.  They have a revision history.  You can tell them to email you when a page is altered.  Nevertheless some managers dislike them because they can become messy and their contents outdated.  But isn’t this true of all the options?
• Confluence. Basically a wiki with bells on; popular in corporations.
• Sharepoint. Popular in Microsoft shops.  I haven’t used it but others tell me it’s tolerable.
• Google docs. Reasonable choice; a single online repository where access can be granted to your colleagues.  Kinda like a wiki only not as good.
• A big fat word document. Ew.  Don’t do this if you can help it.  It’s unlikely to be searchable, many different versions will exist to confuse the unwary and in a sudden crisis nobody will know where to look for it.
• With your code. In text files within a /doc directory in your source release?  More practical for compiled applications than websites.

As far as is practicable within the organisation (okay, some of this needs to stay private) share your documentation.  It’s useless if nobody can find it.  A centralised repository of wit and knowledge is better than an archipelago of lost, unmaintained islands that eventually sink unnoticed beneath the waves.

The more visible your documentation is, the less likely it is to be lost.  Projects come and go but nobody’s going to trash all of docs.mycompanyname.com.

In case the last 2,000 words didn’t say it enough times – if you want to make a reliable, maintainable technical object it must come with meaningful instructions.

3) Orderliness & Predictability

Remember what I said before about conventions?  They’re lovely things.  With a little knowledge of the local customs the world becomes a predictable place and we can figure almost anything out.  This ordered world will be free of nasty surprises.

First I’ll start with a simple example of disorder.

Recently I was asked to migrate a website from its creaking, out-of-date, unpatched server onto a shiny new Ubuntu 10.04 LTS box.  To protect the guilty party let’s say this site was called marshmallow.com.  [Sorry to the real marshmallow.com - I'm sure your site is awesome.  Marshmallow is just the first thing that came into my head.]

First of all I login to the old host (administered ad-hoc by a band of merry developers) to get a copy of its code and data…

# apachectl -S 2>&1 | grep marshmallow
port 80 namevhost marshmallow.com (/var/www/vhosts/marshmallow.com/conf/httpd.include:70)

# grep -i DocumentRoot /var/www/vhosts/marshmallow.com/conf/httpd.include
       DocumentRoot /var/www/vhosts/marshmallow.com/httpdocs

Great, so the code’s in ‘/var/www/vhosts/marshmallow.com/httpdocs’.  I rsync a copy over to the new server.

I know it’s a PHP-MySQL driven site.  So how about that database…

Sadly there’s zero documentation.  I email the hapless offshore developer to ask how to set it up and the response is… confusing.  I set off on a hunt for the site’s databases myself.

# mysql -e "SHOW DATABASES" | grep -i marshmallow
+-------------------------+
| Database                |
+-------------------------+
| marshmallow             |
| marshmallow_marshmallow |
| marshmallow_test        |
| marshmallow_test2       |
+-------------------------+

Looks like they were using the production server to test on as well.  Oh well.  I dump out a copy of ‘marshmallow’, rsync it across to the new host, load it into MySQL there and create some access credentials.  Then I make a wild guess that ‘configuration.php’ in the root of the site is a good place to put them.

In they go and… BOOM, nothing works.  500 errors everywhere.  I figure out some random Apache configuration this site needs and try again.

It works.  Sorta.  Crucial bits of the website are broken – most of it in fact.  I check and recheck my config, check the DB loaded okay, check for charset issues, check everything I can think of.  All the angles are covered.  I email the developer again.  His response?

Are u running old version of the db?  We have use marshmallow_test2 and it work fine!

Why.

The fuck.

Did you put the website into production.

Using the test database?

Why?  Why in the name of all things holy did you do that?  Clearly your English is good enough to know what the word ‘test’ means.  You’ve used it twice.  You even created a database whose name matches perfectly the name of the site and the for some braindead reason 18 months ago you went and pointed the production website at the test DB anyway.  Do you think I’m telepathic?

Over a year ago some developer modified the database.  Maybe he was fixing a bug, maybe he was adding a feature.  There aren’t records so we don’t know.  He did his thing, verified it worked and figured that since the test DB was a recent copy of production he could just, y’know, point the live website at it and go home five minutes earlier.

This is a fine example of disorder.  To save a moment’s work in propagating a change to the site’s production database the developer changed the site from a relatively orderly, predictable setup to one where we can no longer intuitively know how it works.

FAIL.

Disorder comes from sloppiness.  It comes from making odd tweaks to a setup without thinking about the long-term implications.  It comes from not thinking about how a stranger will react to your system when they meet it.  It comes from not thinking ahead – “what if a stranger needs to work with this” or “what if someone sees ‘test’ and accidentally drops the wrong database”?

While the product still works it’s become an accident waiting to happen.  And your systems guy will absolutely hate you for that.  Especially when a disk fails and it turns out that the “production” DB he’s been diligently backing up for the last year wasn’t production at all and all the site’s data has been lost.

Now a counterexample.  What’s order?

I’ve worked with lots of great developers but for this example I’m going to use my good friend Ian Tempest.  Because this is the positive example I’m going to use his real name.  Ian is an absolute joy to work with because the things he makes are documented, logical and reliable.  They almost never go wrong, and even if they do there’ll be plenty of information on how to fix them.

[Quick plug: if you need LAMP development work doing to a high standard, talk to this chap]

The last site Ian handed me for hosting came with:

1. A wiki page documenting its setup, server requirements and directory structure.  Even examples of Apache config the site likes to use.
2. Configuration in well-defined locations – in ‘application/config’.  Who’da thunk it.  Even better it came with alternative config files for different installations for easily symlinking the correct one into place.
3. A directory structure – not everything splurged in the web root where it’s viewable to anyone with a web browser and the ability to guess the filename.
4. Administration scripts – to dump the site’s database for backups and flush data back to test instances.
5. Sample crontab entries – so housekeeping jobs get run at the right time.

And woah, look at that config.  Any bit of behaviour or environment I might ever need to change, all one sanity-inducing place.

Check the project’s directory structure:

ahewson@company-web-01:/var/www/sitename$ ls -l code
total 36
drwxr-xr-x  4 siteuser www-data 4096 2010-10-04 16:03 admin
drwxr-xr-x  3 siteuser www-data 4096 2010-10-04 16:03 apache-conf
drwxr-xr-x 12 siteuser www-data 4096 2010-10-04 16:03 application
drwxr-xr-x  3 siteuser www-data 4096 2010-10-04 16:03 crontab
drwxr-xr-x  4 siteuser www-data 4096 2010-10-04 16:03 dataimports
-rwxr-xr-x  1 siteuser www-data 2348 2010-10-04 16:03 nightly_xtra.sh
drwxr-xr-x  3 siteuser www-data 4096 2010-10-04 16:03 sql
drwxr-xr-x 11 siteuser www-data 4096 2010-10-04 16:03 system
drwxr-xr-x  7 siteuser www-data 4096 2010-10-04 16:03 www

A sane, logical person made this.  A sane, logical person who knows not everything belongs in the DocumentRoot (non-Apache folk: DocumentRoot corresponds with the visible root of your website online) and that it’d sure be good to include some config samples and admin scripts.

The first time I came to set this site up it worked like a charm.  Instead of the hours-long hell of “you hardcoded what?” and “whaddya mean Perl 4!?” the systems person often endures it took five minutes.  Literally five minutes.  We went to the pub and celebrated with many, many beers.

4) Configurability

You’re building a truck, not a house.

During its lifetime an object you make is likely to be replicated, moved or otherwise tinkered with at least once.  It should be easy to uproot and reconfigure your app to use new resources.

How do do config right

Here’s an example from the good Mr Tempest’s application/config/application.cfg:

$config = array_merge($config, array(

 'pageTitle' => 'This Website Name',
 'memCachePrefix' => 'www_',
 'memCacheHost' => '127.0.0.1',
 'memCachePort' => '10011',
 'extraStyle' => 'html {background: url(/img/bodyback2.gif) repeat-x;} ',
 'reportFilesBaseDir' => '/var/www/this_site/assets/Reports',
 'xmlUploadDir'       => '/var/www/this_site/assets/XML_Uploads',
 'importedXmlDir'     => '/var/www/this_site/assets/imported_XML',

 'databases' => array(

   'default' => array(
     'user'     => 'mysite_db_user',
     'pass'     => 'a_password',
     'host'     => 'db-master',
     'database' => 'mysite_data'
   ),

   'users' => array(
     'user'     => 'mysite_db_user',
     'pass'     => 'another_password',
     'host'     => 'db-master',
     'database' => 'mysite_users'
   )

 )
));

Mmmm, baby.  Ain’t that some funky config.

Even someone who’s never written a line of PHP can figure out what this means. If the server broke in the middle of the night and they need to uproot that website they can do it in minutes.  Compare it to the wild database chase in the last section.  This is so, so much better.

How to do config even better…

But we could go further.  In some past projects we’ve ended up doing the following…

# cat /etc/apache2/sites-available/mysite
...
     SetEnv  CONFIG_FILE  "/var/www/mysite/configfile.ini"
...

This is setting an environment variable, “CONFIG_FILE”.  Non-web apps have been finding their config this way for decades.

What’s great about that environment variable (in Apache-land, YMMV elsewhere) is it’s set only for that application.  You could have a different SetEnv for each site you build to point that site at its own .ini.  Whatever language you wrote your site in – PHP, Perl, Ruby, whatever – it’s blindingly obvious where the configuration is.

Better still you could do this:

# cat /etc/apache2/sites-available/mysite
...
     SetEnv  CONFIG_FILE  "/var/www/global_web_config.ini:/var/www/mysite/configfile.ini"
...

See what’s going on?  Resources shared by all sites can be defined once in a file read by all sites.  ‘global_web_config.ini’ could define the names of your database servers, the location of your temp directory and the port memcached listens on – anything which is true for all applications on your platform.

But I digress.  Even single, per-site .ini files are awesome.

So what’s so great about the humble .ini file?  Nothing.  And everything.  Simplicity is the key and its syntax is so very forgiving.  .ini files date back to the dawn of time (well, at least Windows 3.1) and absolutely everything can read them.  PHP, Perl, Python, Ruby, Java, everything has libraries to read a .ini file.

# cat /var/www/mysite/configfile.ini

; Config file for my site; updated 2010-10-14
;
; Changelog:
;    2010-10-13: changed something else
;    2010-10-11: changed something

[database]
   db_host=localhost
   db_name=myapp_db
   db_user=myapp_user
   db_pass=dfgXnfj

[filesystem]
   temp_dir=/var/www/mysite/tmp
   upload_dir=/var/www/mysite/upload_tmp

Try to use the same key names across applications – then anyone who’s setup a website for your organisation will have a pretty good idea how the others work.

It doesn’t have to be Windows .ini format.  It could be JSON or XML or ancient Mayan knotted-rope language for all I care.  Just pick a convention in your organisation and stick with it.  Document it.  Print the docs out and sellotape them to the glasses of every new developer the day they start.

By now you’re thinking “who the hell is this control-freak?”.  Go on, admit it.  If we’ve worked together probably you thought that a lot.  But trust me, if you’ve spent the last ten years putting larger and larger sites onto web platforms you start to care about standardizing the way they get setup.

5) Separation of Code, Data and Config

Follow the configuration rabbit hole a bit further.  I’ve been banging on about this for years and it’s led me to the concept of an application-instance – a distinct personality resulting from the confluence of three things:

• Code – I also mean other blobs of stuff that are unchanged in all versions of the site – for example static images and CSS.
• Data – For example your production database (huge, unwieldy, full of unimaginably private, valuable data) vs. a small test version of the DB containing only a handful of entries for your developers to play with.
• Configuration – Depending on where it’s running your code will need different resources.  Some of this configuration will be sensitive – for example your DB password or your AWS key.  You do not want these in your source repository and you certainly don’t want them on every single developer’s machine.

Only by combining all three do we get a web application.  Take the production site and swap in UAT’s configuration, suddenly it’s UAT.  Need to update that to trunk without changing any of your DB settings?  Fine, configuration is outside the code tree so go wild.  Change the configuration and the data and your schizophrenic website just became a developer’s test instance, now without a copy of the live database and your AWS keys riding the subway home every night on a developer’s laptop.

See where this is heading?  We have an abstraction which holds true for almost all web applications.  Keep these three magical objects separate it’s a breeze to instantiate your site anywhere you wish, with any personality you wish.

6) Bringing it All Together

We can summarize all this with a single sentence:

Be mindful of others who need to work with your product.

It’s hardly ITIL.  It won’t lead to IT totalitarianism or drown your team in red tape.  But it will bring a sense of order to your project – a comforting predictability which lowers the learning curve for new developers and greatly reduces the friction when it comes into contact with others.  Think of it as a bare minimum for teams in a hurry; the absolute least you can do to avoid making a confusing mess.

Build applications this way and the systems guy will love you.