Well, it’s certainly been a while since I’ve been able to post anything and so I offer my apologies to my frequent readers. Let me begin my return with a post on why I took a break at the beginning, and I’ll work to get back to regular postings again going forward.
Is this the threat actor, or that weirdo in Information Security?
I’ll leave it to you to decide.
My hiatus began with a particularly rough patch at work. While I won’t get into specifics, I will simply state that I have the distinct privilege of working with some incredibly talented IT professionals, and that I was never prouder to work alongside them than during the summer of 2021. Since then, I’ve mostly just been busy with increased demands in the office, a resurgence of activity in my family as the kids returned to in-person school, and of course preparing for my transition out of the President role at VMUG (more on that in a later post). For my first post back, I thought that I would focus on some lessons learned in the summer during the cyber security event.
I think my current vantage point of having come out the other side of a cyber security event gives me an unfortunately not unique and increasingly common perspective. Still, as someone who believes firmly in the value of a technology community, I wanted to share some of the thoughts and lessons that I’ve had during and since the event. I offer the following group of possibly disconnected thoughts in the hope that they provide you some useful suggestions and color commentary on an all-too-real threat.
You’re gonna be hacked.
It’s been said so often it’s almost a trope at this point. You’ve seen the security experts tell you that “it’s only a matter of time,” but you don’t want to believe them – can’t believe them. Believing them means that you need to be ready and that seems like an insurmountable challenge. Better to roll the dice and have the resume ready, amiright? Well, actually, you can prepare and be ready but it requires your organization to acknowledge the threat and resource you appropriately. Your organization probably carries insurance for events like fire, flooding, crop loss (if you’re General Mills or something), and other business risks. Do they carry cybersecurity insurance? If not, they probably aren’t taking the threat seriously.
If you’re having trouble convincing your executive management (CEO, CFO, etc.) on the value of having adequate detection and remediation tools, or writing and enforcing the business policies that make such tools useful, I would recommend Cyber Crisis by Eric Cole. It’s a fast read and written for people that think Python is a snake at a zoo - you know, like your CFO. It gives them a very high-level briefing and then a framework. Some of it borders on scare tactics, but I’m not above that to make a point.
If, on the other hand, you are having a hard time convincing your CIO to be ready: get a new job. Nope, I’m not kidding. That person is getting ready to steam their career (and yours) into an abyss. Jump ship. If in 2021 your CIO isn’t losing sleep at least occasionally because of the constant risk of breach and ransom, they’re totally divorced from reality and you have better options. Go find them.
The bottom line is that your business needs to be ready to identify and recover from these events. I think the insurance is a good step (this stuff gets expensive), but you also need to have some things in place to make sure you can recover from such an event. How? Well, I have some other thoughts. Let’s keep going.
Your backups are everything.
Is this a bunch of backup nodes? Probably not, but it was free stock photography so let’s just pretend it’s a bunch of backup nodes.
So why are they all in the same place?!
You’ve seen the movies. An elite team of assassins is outside the mansion of their target. They find the control panel to the alarm system (which is outside because Hollywood) and they cut the wires, but not before the sarcastic hacker in “the van” scrambles cell service and also gets the video feed from inside. With no chance for help to find out or respond, they enter the building. Well, real-life cybercrime isn’t a lot different, except that your backups are that “help.” When a cybercriminal gets into your network, the very first thing they do is try to find your backup copies and destroy them. Why? Because when they send you a ransom, they want to know that you can’t just restore the data without paying them. If their whole reason is to make a profit, they want to ensure their ROI by making sure you pay.
So, what do you do about that? My advice is to make sure that at least one copy of your backup data is air gapped from your network. There’s a lot of ways to do this. You could use tape, or copy your backup data to a write-locked cloud, or use immutable volumes. Or - if you’re paranoid - do more than one of those for extra security. There are some great options from companies like Cohesity and Rubrik that help you check some of these boxes without a lot of engineering. The important thing is to make sure that you have safe offline backups that go back far enough to predate an incident.
The bottom line is to think through the nightmare scenario: if someone gets root access to your backups and destroys them. Put yourself in that scenario mentally. What have you done to be able to recover from that? How will you get back up and running? This is the kind of thing that is an existential threat to many businesses. It’s your job as the IT pro to ensure that your organization isn’t one of them.
Hackers don’t take holidays
Does your company have 24-hour security monitoring? If not, you need it.
It turns out that cybercriminals don’t care if it’s Thanksgiving or Christmas or the 4th of July. They know that’s when you’re paying the least attention – and they will take advantage of that. Make sure that someone - whether it’s your colleague Jill (hey, Jill!) or an outsourced security monitoring firm - is watching around-the-clock. Also, make sure that you have a response plan for when something gets escalated by the monitoring team. It’s not enough to be watching something; you need to be able to respond quickly to prevent or contain the damage.
This is one that sometimes requires re-evaluating your staffing mix. Maybe you need to think about folks working different shifts or having defined on-call rotations to ensure that you have coverage for when the attack happens. Make sure this is a defined, documented procedure; it’s not enough to figure it out when things go south on you, and by the way – you’re going to be so flooded with adrenaline that you will immediately lose some IQ points so having it pre-defined will help. Think this through and develop it before you need it.
MFA all the things
There is no magic or silver bullet to prevent breaches, but multi-factor authentication is an incredibly powerful tool. Passwords alone are just too susceptible to phishing or brute force attacks. It’s like protecting the company vault with one of those keys that comes with your server rack. It’s technically locked, but anyone who wants to can break through. Having a strong MFA solution in place helps you ensure that the people accessing your systems are the people they say they are.
Laugh all you want. In the 1980s, this was the pinnacle of anti-theft. Do you want to hacksaw through that in broad daylight? Didn’t think so.
Go steal someone else’s 1987 Toyota Camry.
My recommendation is that you allow absolutely no access to any of your company systems from offsite without MFA protection. Most cybercriminals aren’t going to show up in your office, so you might be willing to relax a little within your locked facility; but with today’s anytime, anywhere access to apps and data you need something to make sure that the person accessing it is your employee working remotely and not a foreign actor cybercriminal. MFA helps you do that. While there are ways to defeat MFA, it’s like the “The Club” was for cars in the 1980s. It’s not perfect, but it makes the car next to you (or the company down the street) an easier target.
As a simple tip, if you have apps that you can’t get working via MFA, put them behind a firewall that does work with MFA. With today’s authenticator apps, using a one-time password or a simple “approve” button on someone’s mobile device takes seconds and will help you up your security game in a big way. If you’re not the team in IT that sets this up – please don’t complain about it. It’s making your company’s data substantially safer than without it.
Work with great people.
This advice is probably more for everyday rather than just during a cyber incident, but I was thankful throughout my journey that I am fortunate to work with people that I genuinely like as colleagues. Simply put, life is too short to surround yourself with jerks. When you start pulling 18-hour days for multiple weeks (including weekends), you had better be okay spending that kind of time with the folks in the office.
I don’t think this blog entry is the right place to go over a detailed discussion about how to ensure you’re working with people you respect and can learn from; but if you spend any amount of time wishing that a plurality of people in your office would cease to be there, take that as a serious warning sign. After all, if you’re going to be involved in a serious life-or-death event for your company, you shouldn’t waste time being cranky with your colleagues. And honestly, even if you never have that do-or-die moment, wouldn’t you rather spend time with people you like and admire?
If you’re leading people, show you care.
This is very similar to not working with jerks. If you have people that report to you, you need to recognize that this is an incredibly stressful time for them and you need to acknowledge that your success is entirely dependent upon theirs. Then figure out how to show them how much you appreciate what they’re doing for your company.
I’ve written other posts that talk a bit about leading people, but the groundwork for this happens well before a crisis. You should be communicating to your team that you appreciate their contributions and celebrating them as individuals regularly – and not just when you need them because the chips are down.
Listen, when things are going really bad the last thing people need is you barking at them. Make yourself useful and go get some sandwiches and soda.. and not that Sam’s Choice crap.
More practically, be sure to take care of their immediate needs. When you’re asking them to move mountains on a tight deadline and be in the office from 7 am to midnight or later for days on end, they shouldn’t have to worry about simple things like meals. Of course, despite being “in the thick of it”, your team will continue to have families and lives – and you need to give them the flexibility to live it, and in some cases you may need to insist they live it. Trust me when I say that when you look someone in the eye and tell them to go to their kids’ event or take that family trip - even when the fire is burning around your team - they won’t forget that. And really, if you can’t cut someone free for a bit, shame on you for not staffing and developing your team properly. One of the ways you show you care is by building a team where people don’t need to be “on the line” continuously.
Consider Resiliency
While you identify, isolate, and remediate a cyberattack, it’s very likely that one or more of your critical systems are going to be down for a period of time. You need to plan ahead how your company will operate without those systems. Some things that I would suggest you pay special attention to would be:
Your website. It’s a critical tool to communicate with customers and employees during a crisis, so if you take it offline how will you communicate with them? Do you have a backup plan that you can publish ASAP and keep updated?
Your company’s communications and collaboration tools. Don’t underestimate the importance of email, telephony, and persistent chat tools. These are often the lifeblood of an organization and if they’re not available – a huge portion of your company is shutdown.
Any transactional systems such as point of sale, financials, or other applications that are essential for getting revenue. Ultimately, if you are down for any length of time where you can’t book revenue, the costs and pain add up quickly. You need a plan to not let that happen.
Finally, any applications that your customers are actually paying for. For instance, if you are a brokerage and customers can’t access the online trading tools that they are paying for, you’re effectively depriving your customers of a critical service. That’s a really bad look.
Bonus from my wife
One of the things that finally got driven home by living through something like this for my family was to ensure that you are using long, insanely complex passwords that are unique to each website and use MFA on as many sites as possible.
If you don’t know this movie, stop reading this and go watch Spaceballs.
It’s funny how many people I’ve talked to think they’re being really clever because they use the same password like “DingBat2021!” on a bunch of websites. It’s complex and it’s kinda long and so how could it ever be hacked, right? Well, there’s just too many patterns there. Years are easy to try out, the exclamation mark at the end is one of the first things tried in a brute force, and well... you also picked dictionary words. Not so great. Instead, you use something patently obscene like “M7lHiZOC#G*tXMF8CtmML*#3zgE8KZB#vio@KmB8b?”. The use of a password manager like LastPass will go a long way to making this a more viable reality. You never have to type it in – just select it from the list and your password manager will do the rest.
And back to MFA for a bit. We consider it essential for any website that has our financial information. Yes, it can be a little bit annoying when you’ve left your phone on the other side of the room, but the extra security means that I feel a whole heck of a lot better than someone can’t get access to my checking account. I want to know it’s me who overdrafted the $1.58 in it, and not someone pretending to be me.
When something like this happens you may never feel like you were truly prepared, but take it from someone who has been there: being partially prepared is infinitely better than being totally unprepared. Every hour you invest in prevention and planning a recovery will save you hours or days when it comes time to ultimately recover, not to mention potentially hundreds of thousands (or millions?) of dollars in a potential ransom.
We’re in an arms race against the threat actors and unfortunately there is little room for error. IT has to get it right every day, day in and day out. It has to be a string of uninterrupted successes, because the bad guys are always there, always trying to find a way in. They’re attacking your systems, your employees, and your customers to find a way to access your critical systems and data. It’s not fair, but they only need to get it right once to achieve their goals. So, spend the time to make your company as unattractive as possible to them and make sure you have a plan for how to recover and stay operational throughout an incident.
Questions for reflection:
Are you prepared for a successful cyber attack? If someone gets root on your servers, can you survive?
How do you feel about the company you work for and your colleagues? Do you feel like you have each other’s backs when the chips are down?
Do you have enough defensive technologies to make you a harder target than other companies?
