I get the train into Charing Cross from Brockley every day.  I tend to use an Oyster Card because, well, it’s convenient.  Except when it goes wrong and you have to deal with a bunch of people in India to try and get a refund.

This morning the ticket barriers at Brockley were broken.  No way to swipe in with my oyster card.  In fact the barriers were in pieces and there was no way to use any kind of ticket!  I checked with the staff (evidently a lot of people were confused) and was told to go through anyway and that I’d be charged at Charing Cross.

So with some deep misgivings off I went.  09:17 Southern train to London Bridge then the SouthEastern service to Charing Cross.

When I arrived at Charing Cross the gate staff said they’d seen a few people from Brockley with this problem and – clearly disinterested – told me I’d have to pay the full fare for a single ticket.  That’s 40p more than the fare I’d have paid with Oyster.  I questioned this, pointing out that I’d tried to pay the correct Oyster fare and that no additional cost had been mentioned they refused to negotiate and repeatedly told me it was “just the regulations”, “I’d have to complain to Tfl for a refund in writing” and there was nothing they could do.  When I asked in return for the staff to give me a written explanation of why they were imposing the excess charge, they refused.

They further explained that the inflated fare was covered by the Oyster terms and conditions – but didn’t offer a copy or say whether those terms applied, given I had not been informed of the additional fare or been given the opportunity to decline it.  Can you be bound by a contract when no money has changed hands and nobody’s told you the price isn’t as originally advertised?

I suggested I use my Oyster card to pay the correct fare at that end of the journey but was told “the system isn’t setup for it”.  Then I suggested I simply pay the correct Oyster fare in cash and was told that – despite having remained polite and calm throughout – that I would be arrested if I tried to leave the station without paying for a full single ticket.

W. T. F.

Eventually I and another passenger with the same problem called the police over and they apologetically explained they really would have to arrest us if we wouldn’t pay the additional fare.

And that was that.  Basically imprisoned at Charing Cross or forced to pay a higher, unannounced fare.  I’d played nice, remained polite throughout and tried to follow the rules.  So much for that.

Whoever employs the staff at Charing Cross – let’s talk about this.  Overcharging passengers at the far end of the journey when it’s your own equipment at fault and threatening them with arrest if they don’t submit to the mafia-style shakedown is truly beyond the pale.  Furthermore, telling passengers at Brockley they could pay at the other end but neglecting to mention the price would be higher – isn’t that simply fraud?

I know it’s only 40p but seriously, Tfl, SouthEastern, what the HELL are you playing at?  What kind of a service threatens its customers with a criminal record when all they try to do is pay the advertised price for something?

NB: SouthEastern/Tfl/whoever – so as to avoid the popular “privacy forbids us from discussing individual cases” excuse – you have my full permission to publicly discuss what happened this morning.  Please, I really want to hear your version.

But the rest is utter dross.

“Your CV came up in our search” they bleat, “we have an excellent opportunity for you to earn £15k+bens per year as a Windows support monkey in the bowels of some godforsaken corporate in Wolverhampton”.  “See what a great favour we’re doing you”.

It isn’t a favour.   They aren’t your mum and they don’t have your best interests at heart.  Some can do a passable impression of competent, empathic human beings (from experience I’d say about 10%) but most are illiterate swaggering liars who don’t have a clue what all those acronyms on your CV mean.  They want your commission and they’ll say anything to get the meat (that’s you) into the grinder.  They look and quack like estate agents but cost ten times the price and have even less regulation.

So this torrent of adverts for jobs I won’t like.  Most are from lazy agents who’ve done a simple keyword search (I suspect using copies of my CV they’ve illegally scraped from job sites and dumped into their own database) and mass-mailed a tragically effusive “this job is perfect for you!!!!” to any poor soul who came up.  Most turn up just after 6pm because agents believe you just love to get job adverts after a hard day’s work.  And god, their stunted attempts to make the jobs sound cool… there shoud be a special hell for anyone who actually refers to IT experts as ninjas and gurus or tries to link the job to some irrelevant current affair.

Any agents reading this: don’t try to make your emails fun, it instantly marks you out as desperate.  And a tosser.

For a while (for sport?) I’d occasionally reply to the least appropriate.  “Yes, I’d love to be a COBOL developer for a major retail bank…” to see how far along in the process I’d get.  But in the end the stream of invitations to apply for shitty jobs got me down.  Asking them to stop met with variable results; some people just can’t believe you don’t want to hear about their ‘opportunities’.

“Why not use my nerd-fu powers to filter them” I thought.  Screw it, how many could there be?

So the List…

84 so far and that’s just London.  They must breed like rabbits.

This is a non-exhaustive list of the domains used by IT agents from my email in the last couple of years.  It’s UK-centric but I’ll add suggestions from other countries.  Global namespaces FTW.

Download: Domains recruiters mail you from (updated 2011-10-31)

How to Use it?

Turn the list into a set of filtering rules in your favourite mail client.  For me it’s a joy incorporated into a sieve ruleset on my server.  As for the all-important gmail – I can’t find any way to import a bunch of pre-made rules.  Squeak if you have an idea.

Unless you plan to be in your current job forever I wouldn’t suggest routing them all to /dev/null.  Tag them and file into a “jobs” folder to plough through whenever you’re on a long journey.

Here’s a trivial python script to compile that list into a sieve rule:

#!/usr/bin/env python
f = open('recruiter_domains.txt')
doms = filter( lambda d: len(d)>2, f.readlines() )
doms = map( lambda d: '"*@' + d.strip() + '"', doms)
print "# " + '-'*70 + "\n#   Filter messages from recruitment consultants"
print "#   needs require \"fileinto\"; and require \"regex\"; at top of script\n# " + '-'*70 + "\n"
print "if address :matches \"from\" [\n  " + ', '.join(doms) + "\n  ] {\n    fileinto \"recruiters\";\n}\n\n"

See that recruiters?  YES I CAN DO “SCRIPTING”.  Do you know what a scripting are?  That’s one right there ^^

Disclaimer: I’m not responsible for, uh, anything.  Ever.  If this list of domains burns your house down it ain’t my fault.

What if you really do want to block them from your own domain?  If you run Postfix try this:

sed 's/^\(.*\)/\1     REJECT  Agencies not welcome at this domain/' recruiter_domains.txt > /etc/postfix/recruiters
echo "smtpd_sender_restrictions = hash:/etc/postfix/recruiters" >> /etc/postfix/main.cf
postmap hash:/etc/postfix/recruiters
/etc/init.d/postfix reload

Sorted; any email from a domain in that list will bounce.  But think about this first – you might be missing out on some great “opportunities”.

Finally…

I don’t hate all recruiters – just the lazy, incompetent, dishonest ones.  If you’re one of the few who’ve made a career out of matching up talented engineers with jobs they’ll love, go in peace.

The rest of you… it’s better left unsaid.

Yesterday afternoon it went off.  “Oh no!” I thought, “My babies are sick!”

But no…

Hi from Orange. You’ve been chosen to try our free new messaging service Bright Stuff! That’s news, competitions and exclusive offers from brands we think you’ll love, sent direct to your phone. For more info go to brightstuff.orange.co.uk (free to access in the UK only) or to opt out send STOP to 200000

This service sounds awesome.  You say you’ve chosen me to receive news, competitions and exclusive offers from brands you think I’ll love?  I feel honoured.  How was I chosen, a magical Willy Wonka ticket?  Have you been carefully watching my texts (occasional messages like “client-web-01 is unavailable” and “disk full, clean out yo’ shi”) and deduced that I’d just LOVE a stream of news, competitions and exclusive offers from brands you think I’ll love?

Doubt it.

How Does This Happen?

What is Orange Bright Stuff?  A quick Google reveals this…

“At the moment we are inviting users to be part Bright stuff. For more info on the service visit http://brightstuff.orange.co.uk.”

No Orange, you aren’t inviting them.  Inviting means asking somebody if they’d like to take part.  You forcibly signed me up to it.

Who on earth thought this was a good idea?  I imagine the meeting went something like this…

“Hey, Tarquil, we need to find a better way of monetizing those cheap-ass PAYG customers. How can we screw more shareholder value out of the poorest section of our subscriber base?”

“I know Barry, let’s use them as a captive audience for advertising from other companies.  We’ll make a fortune.  But nobody really wants ads so how can we dress up a shit sandwich full of them to make it taste better?”

“Got it!  Let’s just send them to millions of subscribers who haven’t asked.  The ones who pay the least so it doesn’t matter if a few leave.  And we’ll throw in the odd bit of garnish so we can pretend we’re offering them a service.  We’ll send them news and competitions and adverts, all for free.  And if we send it to millions of subscribers by default you just know most will be too dumb or lazy to opt out!”

“Tarquil, that’s brilliant.  We can get paid to send ads to people and it makes the Orange News team look like they do something useful.  We’ve been trying to justify that news portal for years.  But if we can only compress celebrity gossip into 160 character texts and pump them down the lines mixed in with a torrent of shit we’ve got it made.”

“Barry, that’s a brilliant value proposition.  We’re both great. Marketing is so great.  Let’s all go and buy Porsches!”

Tossers.  They’re figments of my imagination and already I’m seething.

So You Want to Opt Out?

There’s more to the story.  I try to unsubscribe.  I reply to the message saying “STOP” (and have to change the number since Orange used a different sender number so if you reply without changing it your “STOP” doesn’t count. And back comes another message saying, roughly, “you’ve asked to opt out of Orange Bright Stuff. are you really sure? If you really don’t want to receive our torrent of crap text STOP to 200000 again.”

So I text “STOP” to 200000 once more (is this getting repetitive?). The phone bleeps again.  Hurrah, are we done?  “You’ve now opted out of Bright Stuff and the service will stop in 24 hours time [why? does it take 24 hours to update the "don't send pointless crap" field in your database?] To opt out of other marketing messages from Orange call 450″

Note the “other marketing messages”.  They haven’t used the ‘M’ word before, they know it turns us plebs off.

Wow.  Orange, you’re really determined to send me marketing.  Isn’t two STOP’s enough to convince you I don’t want it?  Or is this just a ploy to make opting out reeeeally hard so customers will stop trying and you can boast to advertisers about your massive reach? And, er, what’s to stop you changing the name of the service and signing me up again?

I call 450, hit 2 and then 4 (hope this saves a few seconds of someone’s life) and am told I’ll have to pay 25p to be customer services representative.  Orange, WTF? I have to pay 25p for service?  And finally I get to explain it all to a nice Indian-sounding lady who checks the “do not send this guy rubbish” box on Orange’s intranet. She promised to refund the 25p but only after I’d asked.

Corporate marketers – here’s a quick, simple rule. Your customers will NEVER thank you for signing them up to marketing crap.  Don’t act like some tinpot spammer, unless you have something good to tell your customers leave them the hell alone. One-off mailshots about things they really care about (special offer, everything free today) might be okay but signing them up for a shitstream you’re probably making money out of?

Nice one Orange.  You’re sending a very clear message that says “we don’t give a damn about service, you don’t give us enough money so we’re going to make more out of you”.

A month ago I had a parcel shipped from the US.  A week after the parcel arrived came a letter from FedEx saying customs charges needed to be paid.  “Fair cop” I thought as I scanned the invoice.  But wait, what’s on it?

• £10.84 for the customs charge
• £10 “administration fee” from FedEx

Errr, say what now?  Where did I ever agree to pay some unspecified “administration” charge?  And what work have FedEx done to justify £10?  Greedy f*ckers.

The FedEx Website

The invoice has a URL on it – which I assume is for online payment.  I go to the FedEx website and try to create an account.

Fail!

“Oh well” I figure, “their website must be really busy on a Sunday night”.  I email FedEx about it and resolve to try again on Monday.

Fail again!

Nice one FedEx.  Greedy, incompetent f*ckers.

The Post

By the close of play Monday nobody’s replied to my email so I write them a letter.  Something along the lines of “what’s all this £10 administration charge about?”.  Since I don’t dispute the customs charge (well, except morally – why should one have to pay to import equipment that can’t possibly be made in one’s own country?) I include a cheque for it.

A few days later the cheque gets cashed.  Since the letter was wrapped around the cheque it’s certain that someone at FedEx has seen it.  From this I infer FedEx accept £10.84 is all they deserve.

But wait, what’s this?

On June 25th I get home to find an “overdue invoice” letter.  FedEx seem to have noted my payment of the customs charges (they’ve knocked £10.84 off the invoice amount) but assert the rest is still due.  No attempt to respond to my dispute, just a big scary “OVERDUE INVOICE”.

Nice way to look after people, FedEx.  This goes out tomorrow; I wonder if you’ll have the manners to respond?

Monday 27th June

Dear Sir/Madam,

Earlier this month I received from FedEx an invoice (no. 5-431-32874) for customs charges applied to a parcel shipped from the US along with a “clearance administration charge” of £10.00.

Since the service I’ve received in this instance was  poor (broken website, no response to emailed communications) and the invoice contained no information on what “administration” FedEx have performed, I queried it.  On June 7th I sent a letter with a cheque for the £10.84 customs charge (which I do not dispute) together with a question about why you feel entitled to levy a £10 fee on top of the sum already paid to you to transport the parcel.

It is certain that FedEx received this letter since you cashed the cheque enclosed with it.

You have not deigned to reply to my original letter querying the £10 charge you’re trying to impose.  Instead you sent an overdue notice, acknowledging my payment of the customs charge but claiming £10 is still due.

So I’ll ask you again –

1. WHAT services have FedEx performed to justify a £10 “administration” charge?
2. WHAT agreement have I made with FedEx to pay any charges you deem fit?
3. WHY when I queried the original invoice have you twice ignored my communications, despite cashing the cheque included in the most recent?

As I said in my letter the level of customer service you’ve displayed in this instance is abysmal and it’s hardly reasonable to charge people for “administration” when you won’t even answer an email.  That you feel justified in sending me overdue notices while ignoring my query about dubious charges rather underlines the fact.  Is this the level of service all customers should expect from FedEx?

If FedEx want my custom in future I suggest you resolve the situation by abandoning your “administration” fee together with explaining why you’ve given such appalling service and some compensation for the growing amount of time I’ve had to waste chasing you over it.

Yours Faithfully,
Mr A. Hewson

Judging by this (unexplained “administration” charge, broken website, refusal to communicate about a queried invoice) I’d suggest anyone thinking of having stuff shipped to the UK with FedEx think again.

I’ve had an email saying my bill is awaiting payment and suggesting I view it online.  This seems plausible; as a citizen of the developed world I use gas and electricity and I am indeed your customer. It ended with the boilerplate text:

“the internet is inherently insecure and EDF Energy plc cannot accept any liability for the integrity of this message or its attachments. No employee or agent of EDF Energy plc or any related company is authorised to conclude any binding agreement on behalf of EDF Energy plc or any related company by e-mail.”

It’s a fascinating statement.  An unqualified disclaimer would seem to render the entire email paradoxical.  I pondered this for a while.  Are you authorized to conclude a binding agreement with me that you don’t accept liability for the email?  Do I actually have a gas bill or is this some elaborate Kafkaesque logic puzzle?  And if you are authorized to make a legally binding commitment to the bill existing, am I authorized to believe it?

Down this route lay madness.  I opted to accept the missive’s veracity and should a bill actually exist – god forbid – perhaps even pay it.

Clicking the link I was presented with the following, brilliant message:

“If you are new to MyAccount, please click the link below and follow the simple instructions to sign up for MyAccount. If you were registered for MyAccount previously, you will simply need to register again so you can start using our new and improved online service. “

I don’t know if I had a MyAccount before but since you’ve dropped everyone’s details and want us all to register again the point is moot.  I spent the next few minutes of my life filling in the form.  Even the field asking where I was born, though I am genuinely baffled about what my childhood provenance (Scunthorpe if it matters) has to do with the utility bill for a small flat in London.

Alas!  Your website does not like me.  It responds:

“We are currently unable to process your request due to technical reasons. Please try again or you can contact us. Our residential customers can call us on 0800 096 9000 and business customers can call us on 0800 096 2255″.

…so like a good customer I called 0800 096 9000.  Only to hear a message saying you’re closed.

Since your website doesn’t work and you accept no responsibility for things you may or may not have emailed me can we please go back to paper-based billing?  The old system where you concocted a wildly inaccurate estimate, printed it on paper in a legally binding fashion and posted it to my house so I’d ring you up with a less absurd reading and pay worked much better.

Kind Regards,
Alex Hewson.

PS. Probably you won’t get to read any of this after ‘should a’ in the fourth paragraph since the textbox on your “Contact us – Email” page (which isn’t actually an email, it’s a web form) truncates comments at the thousandth character.

PPS.  Since the Internet is an inherently insecure medium I accept no legal responsibility for this communication.

PPPS.  Also I may or may not disclaim responsibility for the previous disclaimer.

—

Addendum: The first postscript was wrong.  When I submitted it to the web form your site responded with this astounding error:

…so unless I print out my message and mail it – rather like one of those old-fashioned letters you’re trying to phase out – you’re unlikely to ever read any of this.

Yes EDF.  Your website is a truly epic failure.

This was written to expand on an idea I touched on in the last piece about webservers – the idea that when a browser on a modern PC takes 250 milliseconds to fetch a web page, hanging around for a further thirty seconds just in case another request gets made is just plain selfish.

What’s All This KeepAlive Nonsense?

Remember when Internet Cafes weren't full of tramps?

We’ll start with a history lesson.  Think back to 1996 – a time when dinosaurs still ruled the earth, Oasis were in the charts and we printed out our web pages at the local cybercafe to read at home later.  The language spoken by web browsers and servers – the HTTP Protocol – was simplistic.  HTTP/1.0 really was basic: when your client (read: Netscape Navigator) wished to retrieve a page it would connect to a server, request the object, snarf it and then go away.

After finding your server the conversation went something like this:

*START*
Client connects to port 80 on server.
Client says "hey, get me /~mocko/mylifestory.html"
Server responds with the file
Server closes the connection
*FINISH*

With such a powerful and simple concept it’s easy to see how the web caught on.  Over the slow connections of the day (low bandwidth, high latency) all that work throwing up and tearing down the connection could impose an overhead of seconds.

As the web exploded it became evident that HTTP/1.0 had a few limitations.  Clever people banged their heads together for a while and in 1997 they came up with an extended version, HTTP/1.1.  It added awesome new features like:

• Virtual Hosting – so more than one domain could be served from a single server.
• Persistent Connections – aka. KeepAlive – so clients could hang around for a while and fetch many objects in the same request.
• Pipelining – a client firing off many requests before the server has responded to the first one so the server could send a whole string of files without pausing for instructions once each completed.

HTTP/1.1 made a big difference.  Suddenly you didn’t need to buy your own server if you wanted to host a little website – you could easily rent space for a few dollars on a server owned by someone else.  The number of domains exploded: almost a million new domains were registered in the last six months of 1997 alone.

And these persistent connections.  What were they for?

The earliest web pages were ridiculously simple.  A single file of HTML (“Here is my home page on the interwebnets”).  Your browser connected, snagged the file and rendered the content into a readable page.

But as connection speeds grew (old enough to remember your first 14.4k modem?) people started to include other content in their pages.  Garish “under construction” images, midi files, all sorts of crap.  Like all new technologies taste came late to the party.  And all of a sudden loading a page didn’t mean just connecting and grabbing some HTML – browsers had to:

1. Connect & get page
2. Parse page for references to objects included
3. Connect again to retrieve every single object

With the overhead of starting a fresh TCP connection every time this was really inefficient.  For a complex page (say one with ten images and some godawful background music) you’d have to go through that connect-request-disconnect dance twelve times.  Madness.  Browsers would do their best to start displaying the page before all the objects had loaded (remember when the images used to arrive five minutes after the text?) but user experience was always, to put it politely, crap.

I won't punish you with the MIDI file.

Enter persistent connections.  With the outbreak of HTTP/1.1 a client would connect, load the HTML for the web page, remain connected while determining what other objects to load and then request them too.  All within the lifetime of a single connection.  For a simple page containing a photo of Gillian Anderson[1] and a tinny MIDI rendering of the X-Files theme tune your Netscape[TM] browser would:

*START*
Connect to server on port 80.  Announce we're talking HTTP/1.1
Request "/~mocko/gillian-anderson-is-a-fox.html"
Receive data; analyse it to determine included content
Request "/~mocko/foxy_gillian_anderson.jpg" and "/~mocko/spooky_music.mid"
Receive both objects in order without delay for further requests
Client has nothing more to request so closes connection
*FINISH*

HTTP/1.1 did away with the terrible overhead of negotiating a TCP connection (remember, this could take seconds) for every single object.  One connection, one disconnection.  It made a big difference.  Pipelining helped even more – on a nascent Internet where your request for the next file took seconds to reach the server giving it a shopping list of items to deliver really cut down on the waiting time.

So KeepAlive exploded into the world like Season 2 of The X-Files.  As the complexity of web pages increased it greatly reduced the time needed to deliver them.  Webmasters (remember them?) loved it and so did the latte-sipping, apple-using cognoscenti in their local cybercafe.

What’s Up With KeepAlive?

In the last 13 years the web has changed.  Everybody in the Western world has a high-bandwidth, low-latency connection to it: even my phone can handle a Skype video chat with flawless quality and no dropouts.  Whilst still a good thing pipelining and keepalive don’t make the difference they used to.  Pages load in milliseconds now, not minutes.

To give users the fastest experience possible browsers have gotten smarter.  Rather than opening a single connection to the server they open a few in parallel.  The copy of Firefox I’m writing in has a “network.http.max-connections-per-server” setting of fifteen.  And… duh-duh-duuuuh… once it’s fetched a page it holds some open just in case the user (or more likely some Javascript in the page) requests more content from the server.  My Firefox has a”network.http.keep-alive.timeout” setting of 115 seconds.

Let’s see a simple example…

*START*
Connect to server on port 80.  Announce we're talking HTTP/1.1
Request "/funky/web/app"
Receive data; analyse it to determine included content
Request "/js/jquery.js" and "/images/header.png" and a zillion other things
Receive all objects in order without delay for further requests
Client has nothing more to request BUT HANGS AROUND JUST IN CASE
...long wait
*FINISH*

What’s wrong with this behaviour?  Nothing from the browser’s point of view.  It helps get pages in front of the user’s eyeballs a little faster.  Nobody likes waiting for their web pages anymore.

But what about the server?

Here’s the rub.  Webservers (well, most small-scale ones) support a very low number of concurrent connections.  In a world where clients connected, requested stuff then went away that was fine.  You’ll only had a few on the line at once – web users spend most of their time reading shit, not pressing refresh or clicking links as fast as they can.

Modern web browsers are selfish.  Let’s see how long my copy of Firefox[2] hangs around after loading a simple web page from www.w3.org.  I’ll use a simple script to time how long Firefox remains connected to W3′s poor server…

mock-air:~ mock$ cat TestKeepAlive.sh
#!/bin/bash

while /usr/bin/true
do
 now=`date "+%H:%M:%S"`
 lingering=`netstat -W | grep -c 'hans-moleman.w3.org'`
 printf "%-12s %1d\n" $now $lingering
 sleep 1s
done

And the result:

mock-air:~ mock$ ./TestKeepAlive.sh
17:41:06     0
17:41:07     1    # Request made & small page delivered
17:41:09     1    # almost instantaneously
17:41:10     1
17:41:11     1    # Browser still connected
17:41:12     1
...
17:41:34     1    # STILL hanging around
17:41:36     1
17:41:37     1
17:41:38     0    # Gone.  Total waiting time 30s seconds.
^C

So what just happened?  Firefox connected, fetched a copy of the first web page ever (all 1011 bytes of it) then remained connceted to the remote server for a further 30 seconds.  That’s 30 seconds longer than a browser would have done in 1993.

So What’s Your Beef?

The problem is this.  Servers can only support a finite number of concurrent connections.  The number depends more on the server’s software architecture than its hardware.  For Apache – still by far the most commonly used server on the web – the default is 150 although for a modern installation linked against an interpreter for dynamic web pages 150 connections would leave machines with less than 4GB of RAM grinding to a halt.  And the more users visiting your website the worse it’ll get.

It’s the webserver equivalent of getting stuck behind a 90 year old in the queue at your local supermarket.  Buying the shopping should take, what, 30 seconds?  But instead they’ll follow it up with a five-hour-long conversation with the cashier about the weather, the government and the football while you quietly seethe and visualise throwing them into a pit of ravenous weasels.  That’s what your browser is doing right now to every webserver it talks with.

Handling lots of concurrent connections has been a long-running issue in computer science.  For a in-depth treatment see “The C10K Problem“.

Webservers with more modern threading models cope better with this than Apache.  FastCGI can help by making webserver processes much more lightweight and load balancers can mitigate connection starvation by terminating thousands of client connections themselves and forwarding requests to the webserver through a small pool of long-lived KeepAlive connections.

KeepAlive isn’t evil per se and it was introduced for a good reason.  But it’s abused terribly by modern web browsers and the result is our poor webservers have thousands of clients tying up processes when only a handful are actively transferring any data.

The problem is especially pronounced in situations of high traffic.  In the last few years I’ve had to run systems which deal with thousands of connections per second from thousands of separate clients, and when you’re in that business KeepAlive is public enemy #1.  I’ve solved more people’s load issues than I can count by adding the simple line “KeepAlive Off” to apache2.conf.

How Do We Fix It?

There are two simple answers and a difficult one.  Betcha weren’t expecting that.

1. For any type of webserver – unless you expect clients to poll your server every couple of seconds (chat app?) reduce the length of time a client is allowed to hang around after making her request.  In Apache it’s called, predictably, “KeepAliveTimeout” and the default is a baffling 15 seconds.  That’s 15 seconds hanging around wasting resources – an aeon in computer terms.  In a modern world where our clients all run lightning-fast computers and have DSL internet connections three seconds ought to be enough.
2. Also for any webserver – if for each hit you only expect to serve a single object KeepAlive is worthless to you.  This was true of the LJToys bugserver – a dedicated webserver I wrote to accept and record isolated “pings” of data from many thousands of web clients.  In this case you can stop browsers hanging around by turning it off completely.  “KeepAlive Off” in Apache’s parlance.
3. Now the hard one: let the KeepAlives stay but use a different sort of webserver.  The killer for Apache and others of its generation is that every connection will use up a process – and that eats up memory on your server way too fast.  But there are alternatives which can efficiently multiplex thousands of connections while using up only a handful of resources.  Nginx is the most popular example.  But remember, they’re relative newcomers and nowhere near as well supported as venerable old Apache.

It’d also help a bunch if we could get over the selfish browser problem.  Using a single connection to transfer the hundred-or-so objects in the average web page is great; hanging around afterward for 30 seconds for shits ‘n giggles is not.  If the browser makers could drop that value right down (say 1 second?) or better yet automatically derive a value for each user’s connection life at the far end of the wire would be a damn sight simpler.

So far Apache et al. have stood the test of time – probably because existing applications and skills in the market are centred around the LAMP stack.  NetCraft’s latest stats sure make interesting reading and I wonder if alternatives like Nginx with much fresher designs will eat the incumbents lunches in the next few years.

Footnotes

[1] It is a common misconception that the Internet was developed as a resilient means of distributing images of kittens and naked ladies during the impending nuclear holocaust.  The Internet was actually created to publish humorous sites about teletubbies and images of Gillian Anderson.

[2] Examples use Firefox because that’s what was closest to hand.  Most modern browsers suck in the same way.

Radio 4 Coverage

On Thursday I went on Radio 4′s “You and Yours” to bring more attention to the issue.  Right now the programme is still on iPlayer – listen to it here.

A dinosaur

TfL (who are a shining example of the right way to do open data) produced an excellent illustration of the benefits to the public.  Later the editor of Wired magazine drew a parallel between National Rail Enquiries and the dinosaurs.

NRE’s spokesman Edward Welsh reiterated their line that it was unintentional that they made the data available for unlicensed use prior to October 2010, and that they won’t negotiate with me because I said nasty things about them “when opening up discussions”.  That isn’t entirely true: discussions had already taken place and I didn’t go public until told by NRE that a free license was unlikely.  You can see that because, uhhh, it’s what the original blog post was all about!

License Appeal

A month after my appeal arrived at National Rail Enquiries and four hours after Radio 4 broadcast their programme, NRE responded.  While the Code of Practice says appeals should be addressed in writing to NRE it appears one must embarrass them on national radio to elicit a response.

I was emailed the letter from Mr Scoggins in a pdf.  In it he says:

• My application for a license is still rejected.
• Even if I had a license the critical remarks I have made about ATOC/NRE would place me in breach of it.  [So the NRE license says you may not criticise NRE?]
• Former use of NRE’s Live Departure Boards feed [even though that feed was freely accessible and documented on the Internet by NRE themselves] was illegal.
• I had not engaged with NRE constructively to gain a license.
• My blog posts are intended to give NRE a poor reputation.
• I am therefore not a reputable or reliable partner for them share the data with.

These are National Rail Enquiries’ official reasons why I cannot use the feed of train times data.  Let’s take them apart one at a time.

“Illegal” usage of the data

While my web app was completely free, contained no adverts, existed solely as a public service to commuters and was written carefully to follow NRE’s instructions on querying their service, National Rail Enquiries assert I was using the data illegally.

This is despite the fact that NRE were providing the data feed and documentation  for using it freely on their website – and a licensing requirement was not prominently displayed.  In fact until October 2010 it was even listed as an open data feed on the London Datastore website.  Programmers had been using the feed for a variety of public-spirited endeavours for a long time before and I find it hard to believe NRE were unaware of this.

Presumably in NRE’s eyes the army of other developers who had come up with novel uses for the information were also breaking the law.

Here we see NRE’s real attitude to the live departure information.  By presenting it to the public without their approval one is breaking the law.  One could hardly describe this attitude as “open”.

No constructive engagement / not a reputable partner

If we look back to the dialogue documented in my original post from October it’s obvious that I made several attempts to engage with National Rail Enquiries before publicly criticising them.

• In late 2009 I wrote asking for a commercial license and had no response at all.
• I tried again a month later, this time by registered post, and was told that absolutely no commercial usage of the data was permitted.  NRE now say this was in error.
• When NRE announced in October 2010 that license keys would be required I emailed Derek Parlour straight away to ask for one.  Judge for yourself whether that email was constructive; I think it was as sweet as pie.
• When ordered to shutdown my web application I continued to attempt a constructive engagement with NRE asking whether the app could be kept going while we negotiated.

Unconstructive?  Hardly.  I did my utmost to engage in a positive and reasonable manner to gain a license to use the data.

Only when NRE ordered me to take down my little application and made it clear that they were intent on charging a fee did I take the issue out into the open.

Related to this: Chris says that even if I had a license the things I’ve said about National Rail Enquiries would be a breach of its terms.  Let’s examine that again – to get a license to use the rail data from NRE you must agree not to say anything critical of them.

Sweet Jesus, what is a contract term like that doing in a fair and civilised society?  Is it even valid under English law?

Blog posts intended to damage NRE’s reputation

It is unfortunate if the Chief Executive of National Rail Enquiries feels my blog posts have damaged his organisation’s reputation.  However all I’m doing is reporting their actions and statements – if NRE’s reputation is suffering it can only be as a result of their own deeds.

If National Rail Enquiries want a better reputation their Chief Executive should resign and make way for someone who isn’t trying to exploit the traveling public for financial gain.

So What are the Licensing Rules?

NRE have not told us what the rules are for granting licenses.  However I’ve tried to catalogue NRE’s responses to license applications (get in touch if you want your attempt listed) and it looks like their decision is split along a simple line.

• If you’re applying to use the data for a nonprofit website a license is offered on fairly reasonable terms.
• However if you intend to make a mobile phone application – even if you’re not making a penny – you’ll be met with licensing terms so draconian you can’t possibly go ahead.  You can’t afford to pay a four-figure license sum if you’re a small-time hacker building a free iPhone app.
• Of course if you’ve criticised them in the media you can get lost.

National Rail Enquiries told one applicant that as a private company they are “not in a position” to supply the data for free.  For two reasons this is an outrageous statement:

1. It flies in the face of the current government’s open data policies.  I can only hope that when future franchises are awarded live train data must be published on a fair and open basis and not withheld from developers as a punitive measure.
2. Until late November National Rail Enquiries were supplying the data for free so they’re clearly capable of doing so.  On the Radio 4 programme NRE’s spokesman went on record as saying that they are supplying the data for free to at least one developer – so telling other developers they are not in a position to supply data for free is an outright lie.

National Rail Enquiries real intention, I believe, is to profit from passengers increasing usage of live train data on mobile devices.  Somebody high up in NRE has decided to ignore the public and state pressure for open data and instead try to “maximise shareholder value” by exploiting the monopoly they have on Live Departure Boards data to make money from anyone who wants to access it from an iPhone or Android app.

Still not convinced?  Go to www.nationalrail.co.uk and search for a train journey.  At the time of writing the results page bombards you with adverts.  I counted four, three of them for NRE’s own 0871-rate “TrainTracker” phone service.  Doesn’t it look like they’re doing their damnedest to claw money out of you?

Need more convincing?  See this Guardian article from June 2009 when NRE were investigated for threatening mobile app developers.

What Next?

Are National Rail Enquiries going to pursue legal action against me for “illegal” usage of the data prior to October 2010?  I hope not.  Gee, all I was trying to do was help commuters get to work quicker.

In any case it’s clear by now I’m not going to gain any traction with the staff of NRE/ATOC.  I’m not aware of a single free Android or iPhone application they’ve given a workable license for (please correct me if I’m wrong) and since it would reduce their revenue from paid applications I reckon they’ll make just about any excuse to avoid granting one.

In the coming days I’ll make a complaint about NRE’s abuse of its monopoly on live departure information directly to the Office of Rail Regulation.   I’ll also try to open a dialogue with the Secretary of State for Transport; while he has no direct control of National Rail Enquiries (it is indeed a private organisation) it would be very much in the public interest if all future franchises included a requirement for open data in their contracts.

This isn’t going to blow over anytime soon.  The attitude to data within the Association of Train Operating Companies is the antithesis of openness and it’ll take nothing less than direct government intervention or pressure slowly applied as new franchises are awarded to change this dinosaur mentality.  For now I fear we’re stuck with them and anybody who wants an app for live train times on their phone will end up paying the Scoggins Tax.

— snip —

Here’s another IT recruitment gem.  I was going to print the lot out and mail it to TechID‘s directors with “are your staff supposed to be this rude?” in 48-point Arial Bold at the top but a quick look on Companies House WebCheck reveals… guess what, the director is one “Nick Blake”.

It starts simply enough…

Subject: Senior Systems Administration – Opportunity
From: “Nick Blake – Identity Solutions UK” <nick.blake@techid.co.uk>
Date: Thu, 22 Jul 2010 15:17:00 +0100
To: <nick.blake@techid.co.uk>

I have spoken to you previously however with the downturn in last 18 months the market has been quiet!
Things are picking up and I have an opportunity that may interest you or perhaps you know someone?
I have your CV although it is out of date so if you could send an updated version for our records it would be great.

[vaguely appropriate but permanent job description follows]

“The future of technical professional recruitment”

www.nuera-rec.co.uk search specialists join Identity Solutions UK

Identity Solutions UK Ltd
145-157 St. Johns St,
London,
EC1V 4PY

I haven’t any record of speaking to a Nick Blake before, or anyone at a company called “Identity Solutions”.  And the “To: <nick.blake@techid.co.uk>” is very suspicious.  Normally emails are “To: <you>”.  Has he just BCC’ed this to a bunch of people?  Like a trusting fool I give him the benefit of the doubt and send a recent CV.

Subject: Re: Senior Systems Administration – Opportunity
From: Alex Hewson <mock@mocko.org.uk>
Date: Thu, 22 Jul 2010 16:14:06 +0100
To: nick.blake@techid.co.uk

Hi Nick,

That is quite appropriate for me but I’m only interested in contract roles. Here’s the latest CV for your records but do check with me before sending it to anyone. With regard to rates I’m looking at £XXpd+

All the best, Alex.

[The X's are mine.  Let's pretend they're kisses.]

He responds with a curt “Ok thanks”.

On Saturday (!) Nick emails me another “opportunity”.  Actually I’m really curious about that word – who is it an opportunity for?  I guess “opportunity” is just one of those great sounding but meaning-neutral words salespeople love.

Subject: opportunity
From: “Nick Blake – Identity Solutions UK” <nick.blake@techid.co.uk>
Date: Sat, 24 Jul 2010 15:46:38 +0100
To: <jobs@mocko.org.uk>

Hi Alex,

Would this position interest you ?

Cross skilled Experienced Systems Administrator – UK-SouthEast-50K- ***RIGHT TO WORK IN THE UK**** more details

http://www.nuera-rec.co.uk/jobDetails.php?vacid=32

Regards

Nick Blake
Managing Consultant

Well done Nick.  Remember when I told you “only interested in contract roles” in the last email?  No, you didn’t, probably because you shoved my CV in your database without bothering to read the email then went to the pub.

BTW - if it’s still up check the vacancy.  At the time of writing it’s for an “Experience Systems Administrator”.

Subject: Re: opportunity
From: Alex Hewson <mock@mocko.org.uk>
Date: Sat, 24 Jul 2010 23:41:56 +0100
To: nick.blake@techid.co.uk

Hi Nick,

As I told you on Thursday I’m only interested in contracts. Can you refrain from mailing me anything else?

Regards, Alex.

I expect it to end here.  Another agent gets the huff when you call them out for ignoring your requirements.  But no… a response, at 14:08 on a Sunday.   Weeeeeird.  Who’s doing recruiting at 14:08 on a Sunday?  Some kind of IT recruiter voodoo where they think candidates are likely to read your mail if it wastes their spare time?  Personally I was watching jet fighters do a whole load of awesome at the Farnborough Airshow.  Did you know an F16 can fly in corkscrews upside-down?  Seriously, that’s some awesome shit. That’s how I spend my Sundays, not emailing people jobs they don’t want.

Subject: RE: opportunity
From: “Nick Blake – Identity Solutions UK” <nick.blake@techid.co.uk>
Date: Sun, 25 Jul 2010 14:08:10 +0100
To: “‘Alex Hewson’” <mock@mocko.org.uk>

Ok,

I’ll get admin to remove you from our main frame databases of both companies.

Regards

Nick Blake
Managing Consultant

Uhh…. what now “?  It’s “mainframe” - even my gran knows that.  If you’re a headhunter in the IT industry you sure as hell should.  And if you’re a small IT recruitment consultant I sincerely doubt you’ve got one.  By now I’m thinking “who is this clown?”

From:  “mock@mocko.org.uk” <mock@mocko.org.uk>
Date: Sun, 25 Jul 2010 15:11:07 +0100 (BST)
To: Nick Blake – Identity Solutions UK<nick.blake@techid.co.uk>
Subject: Re: opportunity

Hi Nick,

Mainframe is all one word.   Hope that helps with the IT.

Best,
Alex.

A little later:

Subject: Re: opportunity
From: nick.blake@techid.co.uk
Date: Sun, 25 Jul 2010 15:38:57 +0000
To: “mock@mocko.org.uk” <mock@mocko.org.uk>

Sarcastic prat!

Regards

Nick Blake
Managing Consultant

Nice one Nick.  Service with a smile.

But you’ve got a problem.  Your server’s hosed and none of those potential readers can see it.  Worse, your story will only be top of the Slashdot/Reddit/whatever for a few hours and this means you’ve got to fix the problem right now before the world loses interest.  You haven’t time to buy new hardware, setup a complex load balancing solution or rewrite half your app’s codebase.  What are you going to do?

Follow uncle Alex’s checklist and we’ll soon have you back on the air.

This is written on the assumption that you’re running a standard, out-of-the-box setup on RedHat, Ubuntu, CentOS or some other common Linux distro with SYSV init scripts.  It also presumes you have Prefork MPM (the default), MySQL as your database and no other major apps running on there.  If you’re already running some funky config then this tutorial isn’t for you.

NB: Uncle Alex makes no warranties about his checklist and accepts no liability if destroys your server, fails to solve the problem, runs over your wife or sleeps with your dog.  Follow these instructions at your own risk.  But you’re desperate, right?

1) Gain Access

If your server is really hosed you will find it hard to SSH in.  If you have console access (e.g. on a virtual hosting package, HP’s excellent iLO or you’re stood in front of the machine) login through that – you’ll save a minute waiting to negotiate a SSH session into your overloaded machine.

Try to run ssh inside a terminal with a scrollback buffer (putty, mac console, gterminal, whatever) because your server may be so laggy it’s quicker to hunt for the output of previous commands than running them again.  If you’re running SSH from the command line use the ‘-v’ flag so you can see how the connection progresses.

It might take a while…

mock@hostname:~$ ssh -v mock@my.slashdotted.server
OpenSSH_5.2p1, OpenSSL 0.9.8l 5 Nov 2009
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to my.slashdotted.server [x.x.x.x] port 22.
debug1: Connection established.
....
....
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
....
....
You have mail.
Last login: Sat Nov  6 02:13:31 2010 from your.desktop.computer

mock@my.slashdotted.server:~$

Be patient.  Resist the temptation to open multiple sessions when the first doesn’t connect immediately: you’ll only load the machine even more.  For each connection the server needs to negotiate a secure pipeline, do authentication for your user and start a process for your shell.  On a healthy Linux box this is usually instantaneous – but right now yours ain’t healthy.

The waiting is the hardest part.

…………………………………………………… connected?

Good.

Best practice be damned – become root straight away.  Yeeee-haw cowboy.

mock@my.slashdotted.server:~$ sudo -s    # or 'su -' if you're in 1990

2) Now to Work.  Diagnosis…

You’re on a heavily loaded machine.  Commands will take forever to complete so you’ve got to make each one count.

First we want to diagnose the problem.  Since interaction with a hosed server is painfully slow we need to do this in as few steps as possible.  But we don’t want to kill Apache and free up resources before we’ve examined the system’s state to figure out what’s wrong…

Bandwidth or server load?

A slashdotting can kill your server in one of two ways: saturating its internet connection with the data you’re serving (rare these days, ISP’s have a lot of bandwidth) or inundating your host with many more connections than it can handle.  We want to figure out which as quickly as possible.

root@my.slashdotted.server:~$ time cat /proc/loadavg
45.25 31.26 22.73 1/135 27844

real    0m3.002s
user    0m2.002s
sys     0m1.000s

Count in your head how long this really takes to return output.  One elephant, two elephant, three elephant…

The ‘cat /proc/loadavg` part is obvious, you want an idea of the system’s load.  But why `time`?  Because it tells us how long it took your host to load the `cat` program into memory, run it and print out the contents of /proc/loadavg.  Perhaps it took a long time because the system’s loaded or perhaps it ran quickly and the result took forever to reach you through the congested network.  And the best part of it is if your shell’s bash `time` is a builtin – so we can know it imposed almost no overhead.

So does the machine’s timing for ‘real’ match what you counted in your head?

1. If `cat /proc/loadavg` ran quickly but the load is high your network connection is likely okay.  But the load is huge – what do you do?  Read on…
2. If the load is low (say <10) but it took forever for that output to reach you probably your server’s network link is saturated.  Your server is healthy, it just has a bandwidth issue.  Skip the next section.

3) Getting the Load Down (or “It’s almost always the RAM”)

Look at the first three figures in /proc/loadavg.  On a healthy Unix server the load averages ought to be no higher than your number of CPU’s.  A little higher (maybe 2x) is bearable but really high numbers mean you’ve got a system load problem.   The extreme figures in the example – “85.25 73.26 41.73″ in our example would triangulate your position as “up shit creek”.

In case you you were thinking “hey, how many CPU’s have I got?” here’s a simple way to find out:

root@my.slashdotted.server:~$ grep ^processor /proc/cpuinfo
processor       : 0
processor       : 1

You’ll see a ‘processor:’ line for each CPU.  The server in this example has two cores.

For a better understanding of load averages read the excellent page at LinuxInsight.  In short – if the first number is higher than the last your problem is getting worse.

How many apache processes?

One dead giveaway for system load issues is the number of Apache processes.  That’s not to say Apache itself is the problem – but if you have hundreds of them backing up on a small server it means something’s wrong…

# on debian/ubuntu
root@my.slashdotted.server:~$ ps ax | grep -c /usr/sbin/apache2
151

# on redhat/centos/suse etc.
root@my.slashdotted.server:~$ ps ax | grep -c /usr/sbin/httpd
151

151?  That’s a lot of processes.  Why, if you’d maxed out Apache’s default ‘MaxClients’ setting then added one more for the `grep` it’d be 151.  I wonder if that’s a coincidence…

If you’re running a modern webserver – one where Apache has modules enabled for PHP, Perl, Python or some other language interpreter – each of those processes is going to be big.  So big that together they eat way more RAM than your computer has.   For example on my little 32-bit Ubuntu 10.04 VM each Apache process eats 27Mb of physical RAM.  And that’s with mod_php alone; add Python or Perl as well and it gets bigger.  If you live in 64-bit land even more so.

On a machine with limited memory a 32-bit OS is almost always better.

Do you have enough RAM to run 150 processes who each need 27Mb of RAM?  Apache will eat 4Gb before you even begin to think about OS services or MySQL.  Unless you have tons of RAM you can’t afford that many.

[No, I don't know why the distros all ship with a default so high.  Naughty Apache, why can't you autodetect it and set a sensible value at start time?]

So let’s figure out how much memory each of your Apache processes is using:

root@my.slashdotted.machine:~$ top -u www-data -b -n 1
top - 21:21:32 up 3 days, 26 min,  2 users,  load average: 66.6 66.6 66.6
Tasks: 301 total,   1 running, 300 sleeping,   0 stopped,   0 zombie
Cpu(s):  ...
Mem:    408944k total,   310848k used,    98096k free,    12172k buffers
Swap:  1048568k total,   574372k used,   474196k free,    93884k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16228 www-data  20   0 64072  26m 4996 S  0.0  6.7   0:02.49 apache2
16275 www-data  20   0 62564  25m 3672 S  0.0  6.4   0:02.66 apache2
...
...

On a RedHat-derived machine you’ll want to use “-u apache” – they do things differently over there.

Right now half your RAM is on the disk and only this wee critter can get to it. He ain't in a hurry...

Woah buddy, look at that “Swap:  1048568k total,   574372k used” line.  That means half your memory has been swapped out to disk.  That ain’t good; your computer’s been reduced to the speed of that little mechanical head skittering around inside your hard drive.  The Linux kernel does its best to swap out less-used pages of RAM first but with the kind of overload you’re seeing half your code is executing off of here.  Mr Babbage is on the phone and he wants his mechanical computer back…

Now you know enough about the problem to stop Apache.  Kill it dead with:

# in RedHat/Suse/Centos:
root@my.slashdotted.machine:~$ /etc/init.d/httpd stop

# in Debian/Ubuntu
root@my.slashdotted.machine:~$ /etc/init.d/apache2 stop

It might take a while for all those processes to terminate.  Apache will try to finish any requests it was servicing but if you have 150+ processes all fighting for memory and CPU they’ll take a minute or two to come back from swap and terminate.

All stopped? Good. Already the load is starting to fall and your machine is much more responsive to terminal commands.  You’re off the air so let’s keep it short and sweet….

Open up Apache’s main config file in your favourite editor. In RedHat land it’s /etc/httpd/httpd.conf; in Debian/Ubuntu you’re looking for /etc/apache2/apache2.conf.

If you’re running with the standard config that came with your machine Apache is likely using the Prefork MPM. This means it runs a bunch of processes waiting for incoming connections and will always try to keep a couple of spare for the next client. If you have enough memory that’s great; otherwise it results in what’s called “swapping yourself to death”.

Look for the MPM settings. It’ll be a section of config something like:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers       10
    MaxClients            150
    MaxRequestsPerChild   0
</IfModule>

Most of these settings are sane.  But look at MaxClients, it’s the upper limit on the number of processes Apache will run when the whole world is trying to connect.  Reduce it to a level commensurate with your machine’s amount of RAM.

As a rule of thumb:

• For 512Mb of RAM MaxClients should be no higher than 20
• For 1Gb, 20
• For 4Gb, 75

That default of 150 is way too generous for a small server.  It comes from a more innocent time before everyone had mod_php (or python or perl or whatever) loaded into Apache and processes were a lot smaller.

KeepAlives considered harmful

So now you have Apache configured to run a number of processes roughly in line with what your hardware supports.  While you’re tinkering with it there are a couple more useful changes you can make:

1. Search for the “Timeout” setting.  Usually it’s way too high – something like 300 seconds.  This means that if a client browser’s connection to you hangs (like if their DSL cut out or their PC crashed) the Apache process you had serving them will hang around for 300 seconds in case they come back again.  Screw ‘em, drop that value right down.  15 seconds is enough.
2. Search for the “KeepAliveTimeout” setting.  I could write a whole essay on the way modern browsers are greedy with HTTP KeepAlives.  If you can only afford a handful of Apache processes do you really want each tied up by an idle client waiting for their user to click the next link?  Until your load spike subsides set it to something like how long it takes to deliver your pages to the average user.  3 seconds, not 30.  HTTP KeepAlives deserve an essay all of their own – maybe one day I’ll sit down and write it.  [I did]
3. Depending on the kind of content you’re serving you might even switch KeepAlive to “Off”.  If you’re only serving a one or two objects in the average request (i.e. the page getting hammered does not contain much CSS or images) KeepAlive costs you more under high load.  Think of this again if you shift content off to a CDN.

For the time being resist the temptation to mess with your php.ini or any other interpreter-level settings.  They are complex and quick to anger.  You want the low-hanging fruit.

By now you should have a sane Apache config, one at least vaguely suited to your hardware.  Start the service up again…

# Ubuntu/Debian
root@my.slashdotted.machine:~$ /etc/init.d/apache2 start

# RedHat/CentOS/Suse
root@my.slashdotted.machine:~$ /etc/init.d/httpd start

If you made a mistake in the config Apache should tell you here. But probably you’re fine and Apache will go on its merry way.

So now you have a webserver where:

• Fewer clients can be connected at once, which means fewer of those heavyweight Apache processes will be competing for memory at any one time.  Those processes which do exist now run a lot faster.
• Clients are not permitted to stay on the line for ages after you’ve delivered their web page.  If KeepAlive is still on they get cut off after KeepAliveTimeout seconds; if you disabled it entirely they get one object per connection and the process is immediately free to serve another user.

Run `top` and keep an eye on those load figures.  Are they going up or down?  Is your website responsive again?

What about the database?

Of course, there might be some underlying reason why your Apache processes were backing up like the holding pattern above Heathrow.   Maybe it was browsers holding open persistent connections forever (which you just stopped) or maybe it’s something else.  Maybe your code running inside all those Apache processes is waiting to use some shared resource on the system, something that’s dog-slow and causes each process to pause until it’s available.

People have described optimising MySQL far better than I can in this article but here’s a few tips:

• Allocate an amount of RAM (mainly key_buffer_size and innodb_buffer_pool_size in my.cnf) appropriate for your server.  The defaults are measly; allowing MySQL to hold more data in memory can be a big win.
• Workloads vary so there are no hard-and-fast rules
• Use `show processlist` to look for running queries
• The query cache rarely helps – writes to a table invalidate it
• Use the slow query log to identify queries taking a long time to run
• Slow queries can often be sped up by adding an index to the appropriate columns
• Use the smallest data type available for each column.  Unsigned integers where appropriate.  The smaller you make your table the more of it will fit in RAM.

I could go on.

The single biggest thing you can do to solve database issues is hit the database less.  See section 5 for suggestions on caching with common web applications.

4) Reducing your Bandwidth

So it turns our your server is connected to the Internet via a piece of wet string and you’re trying to push way more data than it can handle.  Did your ISP tell you it was on an “unlimited” 100Mbit connection?  Did they tell you how many users that 100Mbit is shared between?  Oops.

Or maybe you’re just terrified of the bandwidth bill.  Either way, while enduring a traffic spike it’s a good idea to reduce your network traffic as far as possible.  The benefits are not solely financial:

• Delivering less data to your client means they need be connected to your host for a shorter time.  Think how long it takes to deliver a 300k jpeg to the average DSL user – that’s going to take at least a couple of seconds, time your valuable Apache process could be using to run the code which generates your dynamic content – the only bit that really changes from user to user.
• You want to deliver as few objects as possible per page impression.  The more you can shift to someone else’s server (or even eliminate entirely!) the better.  Get it down to the magic number of 1 and you can turn off KeepAlives and free up a lot of Apache processes.

How to do this?  Simple answer: move static assets out to a CDN.

“But that’ll take ages!” you wail.  ”I haven’t time to do a deal with Akamai or open an Amazon CloudFront account!”

Don’t worry, there are some very simple CDN’s out there.  Coral Cache is here to save your day.  For free these guys will cache static objects from your server and relay them to your visitors.  At times of low load the Coral Cache slows down your site (objects fall out of the cache and Coral must re-retrieve them for every hit) but when you’re sustaining a dozen hits per second that isn’t a problem.  Just remember to switch if off (or get an account with a proper CDN) once the panic is over.

Big stuff first

You don’t have hours to spend out figuring out what to move.  Small objects (little user icons, spacer gifs, menu options) aren’t your problem – look at the pages seeing the highest traffic and figure out what the largest static objects are.  Probably they will be article images (e.g. that 150k jpeg of your mom) and maybe, if there’s a lot of it, your CSS and Javascript includes.  In almost all cases the body text of the page is the least of your worries.

Firefox will help  you figure out what the largest objects within a page are…

1. Navigate to your page
2. Right-click for the context menu
3. Click “view page info”
4. Move to the “Media” tab.  You’ll see a table of objects loaded as parts of the page.
5. At the top-right hand corner you’ll see a button for selecting what columns the table should show.  Ensure “size” is checked.
6. Look at the sizes of objects included in your page.  Starting with the largest items ask yourself “does this really need to be generated by my server for each hit?”

Of course, if your server’s really hosed loading the page will be hard.  You’ll just have to guess what the largest assets were.  Unless you’ve some big embedded media (videos, audio) it’s almost certainly the images.

Hope the nerds don’t mind me using their page about steam engine design as an example.  Maybe steam engine design will come back into the vogue and they’ll get slashdotted.

Immediately we can see the best candidates for moving off the server: those jpeg images add 131k per page view. If you have fifty hits a second that’s going to chew 6.4 megabytes of bandwidth every second.  Yowch.  I don’t wanna be the guy who pays your hosting bills!

Related: this is why big sites (eBay, Facebook etc) use a separate webserver for big static assets like images.  Delivering them takes much longer than the page’s dynamic content and can be done much more efficiently using simpler types of webserver.

5) Advanced: Caching

Probably the page getting hammered is being served using some third-party application you installed from the Internet.  Examples here are WordPress, Gallery, Drupal and Joomla.

Caching at the app level

Bad news: in their default states these applications are rarely efficient.

Good news: for most of the common web applications someone else has already noticed the problem and written a plugin to fix it.  See if you can find a caching plugin that’ll store generated content and regurgitate it for future hits.  I did this for my WordPress installation and won at least 10x extra performance for a few seconds work.

I won’t attempt to list the caching plugins for every big web app but here are a few popular ones:

• WordPress: WP-Cache
• Drupal: Cache
• Joomla: System cache plugin
• Gallery: built-in, just enable it

Caching at the PHP level

Is the app you’re serving written in PHP?  There’s an easy win to be had by installing one of the code caches.  I favour xcache but it’s a matter of personal choice.

# Ubuntu/Debian
root@my.slashdotted.machine:~$ apt-get install php5-xcache
root@my.slashdotted.machine:~$ service apache2 restart

# RedHat/CentOS/Suse
root@my.slashdotted.machine:~$ yum install php-xcache
root@my.slashdotted.machine:~$ /etc/init.d/httpd restart

Tweak the settings if you like but in both cases the cache modules should give a big improvement straight out of the box.

What does it do?  Two things:

1. By default for every hit to a dynamic page PHP will be loading the code from disk, interpreting it and then executing it.  A code cache skips the first two steps – once the code has been loaded and interpreted it’ll be stored in memory for later use.  Unless you’ve changed your code (by default the module calls stat() on the file to check) subsequent hits should mean a lot less work for PHP.
2. Provides a special area of memory shared between all processes serving your data.  If written to do so (look for a plugin) this can be used to cache generated page content and other commonly loaded data.

6) Conclusions

If you’re running a stock install of any OS on minimal hardware as your webserver it’s common to run into problems under load.  There are plenty of simple, tried-and-tested techniques for reducing it – with a few minutes and a little knowledge of your server’s internals you can squeeze an awful lot more performance out of a simple machine before needing to buy more hardware.

A month later they still hadn’t gotten back about my license application – even though I’m aware of several web developers who have been offered licenses to use the data feeds for their free sites.  So I sent Derek Parlour a polite email prodding him about the lack of communication:

Hi Derek,

Hope you’re well.  Has there been any progress on my application for a license?

Regards,
Alex.

Eight days later Derek replied to say they would not be licensing my use of this “open” data feed, and that this is solely due to the negative publicity caused by my posts about National Rail Enquiries greedy behaviour around its data feeds.

So much for free speech.  Even though NRE’s participant organisations (& therefore NRE itself) are funded by passenger fares and government subsidy if you criticize them you get punished: they won’t let you use the feed of UK train information.

Fortunately National Rail Enquiries are obliged by their code of practice to give a timely, written response containing their reasons for denying my website a license.  Unfortunately the person I have to appeal to is Chris Scoggins, Chief Executive of National Rail Enquiries – the very man who took umbrage last month when I quoted the three lines of the email where NRE said my free application was likely to face a license fee.

Here’s my appeal to Chris Scoggins in full:

Thursday 2nd December

Dear Chris,

We spoke via email a month ago about NRE’s decision to close their “open” data feeds of train times and provide data on train times only to entities for whom NRE choose to grant a license.

On October 27th I applied to your colleague Derek Parlour for a license to use the service.  As you know he responded somewhat negatively but did not answer my request for a license.

On November 22nd – almost a month later – I emailed Derek again asking whether there had been any progress.  After a a pause of 8 days Derek replied: he said clearly that:

• NRE have a responsibility to its shareholders and licensees are required to work with it to “promote the industry”
• I have made a number of public comments which make him think I will not work with NRE in its aims.

No doubt Derek is referring to the blog posts I made after he insisted I shut down my free, public train times application and the subsequent media coverage when you emailed me what looked like a veiled threat for quoting three lines of Derek’s email.

Derek did not supply any other reasons in his email – thus he’s stated quite clearly that you’re refusing the license because I made comments critical of your organisation and not on any technical or other practical grounds.

I responded to Derek asking him to clarify what these “aims” are; while he replied to this email he neglected to answer the question.  As we can see from previous communications Derek seems to have problems answering a lot of simple questions.  May I ask why he is so circumspect in answering simple queries from a member of the public?  Since they form a part of the refusal I do now have a right to know what these aims are.

NRE’s denial of a license seems extremely childish.  It smacks of a punishment for speaking out publicly about your organisation’s intention to charge free application developers for train time information.  Is it in fact the case that “promoting the industry” means licensees are not permitted any freedom of speech about NRE and its conduct, and that those who publicly criticize you will be denied access to the data feeds?

As per the Code of Practice NRE must follow regarding its real-time information systems I would like to appeal this decision.  I understand that such appeals should be addressed in writing to you.  You are obliged to give me a “fair and impartial” written response explaining the decision “with full reasons” and that if I am not satisfied with this I can escalate it to the board of National Rail Enquiries, and presumably thence to the Office of Rail Regulation and my MP.

I look forward to your fair and impartial response.

Yours Sincerely,

Mr A. Hewson

I am truly fascinated to see how they respond.  More flannel?  Legal threats for exposing their shady practices?  Let’s wait and see.

Edit: “Your item with reference AH209917311GB was delivered from our WEST CENTRAL LONDON Delivery Office on 06/12/10.”.  No response yet as of 2011-01-02.