This week we ran into a little bit of an unknown load problem with the Livecloud/Questionshark network, and it reminded me of a good rule of thumb when writing performant code- expect the unexpected.

The issue we had presented itself as a sudden load spike on our servers. Load is a semi-ambiguous term in the server world, it translates to stuff to do divided by processors available to do it. Generally speaking we aim for a low number. We were able to determine through various logging that the high load boiled down to one quite simple and quite fast query that shouldn't cause any problems whatsoever. The query in question is a highly cached lookup of what resources to load for an incoming web request.

Enter spammers... Someone sent a mass spam email containing a bogus address that pointed to our servers.

Our problem? We didn't cache bad data.  And, honestly, why would we?  By definition it is BAD. But we needed to cache the bad data in order to catch the spam requests so we can prevent any load issues. Once we figured out the problem, it took a few minutes to release a patch to our production environments, but it reminded us... Expect the unexpected.