Debate: the enemy of effective incident response?
04:30 Monday, 20 September 2021
UK Cyber Security Council
As with many things in life, Incident Response (IR) is something you get better at with practice. I have yet to experience an IR from which I did not learn something, and incident response simulations have further enhanced my understanding of how to conduct an incident response.
I have probably worked on a dozen real or simulated incidents over the last few years, and one simulation sticks firmly in my mind. My organisation rented two rooms in a remote location for business continuity purposes: one had about 45 workstations and was populated by the key business functions, while the other seated a dozen and was the “command centre” from which my colleagues and I would direct the response.
We employed a specialist third party to run our simulations and, once we had convened, powered up our PCs and ensured a stable supply of coffee (note: the authors of incident response plans often forget that IRs can take hours and the responders need fluids), the organiser read out the scenario. A few sentences into the oration, we were informed that the entire executive team and Board were at an offsite meeting in the Caribbean, that bad weather had knocked out the telecoms links to their island, and that hence they were uncontactable. It felt like we were supposed to consider this a bad thing, but the small cheer that went up told me that everyone in the room thought just the opposite.
And here’s the thing: in day-to-day operations, the executive team run the show. And rightly so: they are immersed in the strategic direction and planning of the organisation and know more about the “big picture” than the people at the coal face - hence their business decisions are more likely to be right than those further down the hierarchy. But in an incident response the opposite is true, as IR is mostly a tactical exercise. Yes, you must have an incident response framework and a plan for how to respond, because that’s what enables you to mobilise the right people and support functions and to hand out responsibilities such as PR, stakeholder communication, legal considerations and note-taking. But as it is impossible to plan for every potential scenario, much of an incident response is done by people in a room talking to each other and making quick decisions.
And sometimes the decisions made will be bad. Not bad in the sense that they are wrong, because except in extreme circumstances there is no such thing as “wrong” in an incident response, but bad in the sense that they do something negative to the organisation - disable the main web site, perhaps, or take the call centre off line, or incur a hefty cost. Taking such decisions can be tough, but what’s essential is that they are taken.
Our team’s joy at being told we had no executive help on hand was daunting at first, but everyone in the room read it the same: we’re accountable for how things go, but we’re empowered to do our best to deal with the incident. Yes, it was only a simulation, but anyone reading this who has taken part in one will know that once these exercises get going, it’s easy to forget that you’re not part of a real incident. The team performed well over the course of a very tiring day and emerged with a good result, albeit with a little bit of adverse (pretend) PR thanks to one sub-optimal call.
Just as this example sticks in my mind, though, so does another – and one whose positive outcome was more by luck than by judgement. I received a call one Saturday morning from a friend who was a middle-ranking IT manager, asking for advice: in less than a minute it was clear that his company was subject to a ransomware attack, so my advice was to activate the organisation’s incident response plan. It’s what the policy said should happen (another note: always have a policy that says you won’t be shot for activating an IR in good faith if it turns out to be a false alarm) but his boss’s boss revoked the instruction as he thought (=guessed) it was a minor issue that didn’t warrant disturbing people at the weekend. Happily, my friend was a hardy chap and went directly to the boss’s boss’s boss, the IR team was called, and the attack was handled with minimal damage. I gather also that the boss’s boss received something of a talking-to from on high the following Monday morning.
Incident response is an inexact science, which means we need to make decisions and run with them. This doesn’t mean we have to be rash and leap to conclusions without proper consideration, but it does mean that we need to discuss and consider enough but no more, and then act decisively. Excessive debate, or being second-guessed by management will always end in tears: it is almost always better to act sub-optimally than to not act at all.