Logging security activity is one thing. Actually reading the logs is another.
12:00 Monday, 16 November 2020
UK Cyber Security Council
Technology is based on science. Microprocessors are manufactured in sterile factories, storage engineers do funky atom-level physics to cram yet more data on a magnetic disk or tape, mathematicians perform calculations to establish the computational complexity of an algorithm and so optimise performance.
Yet logging seems to be more an art than a science – there seems to be no right answer.
When I was a fresh-faced graduate of 21, I managed a fleet of Sun-3 servers. Logging was the bane of my life, because the available disk space at any point could only be described as “not very much”. To stand any chance of figuring out, after the event, why something broke, one had to walk the tightrope of cranking up the log activity just enough to be informative but not enough to kill the machine by filling up the disk.
And if something went wrong: all bets were, generally, off. It would die, start to come up, write a few thousand “hey, I broke, but I’m nearly back now” messages, fill the disk up, kernel-dump and die. Rinse and repeat, until I could get my timing right when stabbing the right keys to make it boot into single-user mode and thus have some modicum of control over the start-up process.
Wind the clock forward, and we have infinite amounts of storage and we can log anything we like. We also have infinite amounts of CPU, so we don’t have to worry about slowing processing down with excessive logging. At last, we can be sure that we are able to log to a helpful extent.
Why, then, do we not bother looking at the logs until something has gone catastrophically wrong and we have the incident response team looking expectantly at us? We have more log data than ever before, but we don’t make the time to look at it until it’s too late.
And don’t try to say that logs of the past can’t predict hacks of the future, because that’s simply wrong. For example, the predecessors of SMART (the technology that monitors hard disk performance and flags behaviour that could indicate a future failure) have been around since 1992. And if, after the event, you are looking in the logs to see what happened that relates to the security incident that just hit you, you’re kidding yourself if you think you couldn’t have used that data to predict a future incident. In most cases you look back after the event and see a sequence of log entries that, if played forward, would inevitably lead to something bad.
So make the most of comprehensive logging. But look at the logs – either manually, or through scripts, or via a SIEM product.
Apply some science. Because it means that you’ll get value from the logs. And that you may well see the incident before it happens, not after.