If you don’t have a successful information security program don’t waste your dollars or time on threat hunting until you can secure what you own first.
There has been much ink spilled on threat hunting in the network. Even the concept of what is threat hunting can spawn hair-raising and chest thumping arguments. So, given a few common things we can define threat hunting fairly easily. First, we expect that you have a pretty standard threat and vulnerability mitigation strategy such as firewalls, intrusion detection/prevention, and anti-virus/host agent protection. Second, we expect that you have some sort of operations staff, security staff, and maybe an incident response/forensic capability. Those are roles so some people may hold multiple roles but have skills coverage. Finally, if you have all of that then threat hunting means going past your capability and looking for the badness that bypassed your current security profile. Instead of waiting around for patches, IOCs, firewall rules, and more to catch up to the current adversary capability. Don’t start threat hunting when your house is already on fire.
Threat hunting means you have moved past looking for indicators of compromise and are looking for patterns of compromise.
I’m going to discuss the pantheon of threat hunting a bit and end with what you as a CISO should think about. Some of this is based on great work by other people, but a lot of it is based on experience and studying best practices. We’re not going to make you an expert at it in a 1500 words. What we can do is expose you to a few concepts. One of the leadership caveats I always end up with is “what if we find something?” I could be pithy and say “what if you don’t find anything?” The end result ends up in a quick discussion about network security.
This isn’t “blue team” operations or “red team” operations. If you think about it that way you will build silos when you need to be building barns.
The better your network security and the better engineered the security program. The absolute best your incident response and threat team should likely reflect. As your security team increases in skill and demonstrable capability in keeping the network closed. The more likely that the threats found inside are going to have superlative capability. Thus, your teams that are the shock absorber for incident response (CIRT and threat hunting) are going to need superlative skill. Thus, we are looking at highly mature and more importantly well-funded programs. At some point I’ll write a post talking about right sizing and right funding a security program from a realist point of view.
It is all about the network
Having a good log collection and netflow analysis capability allows you to hunt for threats. Many people focus on the current network traffic and looking for real time anomalies. A world class program will keep netflow logged for a window of a year. The first bonus is if a new threat is defined and you have signatures you can roll back time and see over the last year if that traffic transited your network. The next threat is offline analysis has zero impact on operational performance of the network. Architected well you can look for windows of anomalies and identify behavior using automated means. In a nutshell that means you can find strangeness and shrink dwell time for an adversary on the network.
Think of the network as the highway entering your building. All of the interdiction stuff you normally do is still in place. What is different is that you are looking up license plates, bouncing them off a database, backtracking through all the points a vehicle transited, and then keeping a log to see if they diverge in behavior anytime in the future. You are now looking past direct evidence and at correlation and patterns.
It is all about the end point
Servers, clients, mobile devices, and unicorns all make the CISO head spin. No matter how much protection you put into the network barrier. The second a laptop leaves your enclave, or the mobile device hits somebody’s pocket. Your perimeter evaporates. As long as IO is IO so the air gap go no more. In other words. There is no air gap and you are using mobile device management, some kind of host intrusion detection/prevention system, and likely an agent on the host for remediation. Give all of that you can still hunt the host for adversary artifacts and clues. Since the end points represent your largest surface of user interaction and risk. Hunting takes on a sense of stalking, following indicators of possible compromise to particular hosts. Things like beacons, web pages, slow machines, and other elements might get your notice.
Wherever you see badness, whatever intermediary sensors give you an idea of badness. You are going to end up on an endpoint.
Whether randomly pulling boxes from production for examination, or given a tip-off from the network surveillance. Hunting on a host usually starts in the file system and memory system. There are ways to dump the memory of a host and then evaluate it for possible previously undetected malware. SANS and others teach memory forensics courses that serve the threat hunting team well in the skills development area. In my experience I have pulled boxes out of production I thought were exploited only to at a forensic level refute that hypothesis. In other cases I pulled boxes from production that had no sign of issues and had twenty or more variants of malware infesting them.
reliability and validity are not the same thing. The dichotomy of discovery based on the nearly random nature of some of the processes make the analogy of hunting work. You can stalk, you hunt from a blind, or you can take whatever walks into your path. It is all about looking for things that you didn’t know exist. Your team needs to be active persistent defensive agents on the network. The time intensive and mission impacts of interdicting a host all result in managerial reluctance. That reluctance is well founded because a host may have exploitation’s that will result in downtime.
I thought I was hunting and discovered I was fishing
I often get asked two questions. What is the role of honeypots/nets in threat hunting and why do we do this. Discussing the first question a honeypot is literally a sophisticated intrusion detection system. From a realist point of view you can think of the honeynet as a sensor, or trip line that gives you warning. From a more perfunctory position a honeynet allows you to shoot fish in a barrel. Since you have a live adversary operating in real-time you can start poking and prodding by changing permissions, reconfiguring the box, and collection of indicators and patterns of compromise.
We don’t often think about the fishing analogy since it was taken over to discuss a common social engineering exploit strategy. Yet the analogy remains. Sometimes you bait a hook and drag it around the network enticing an unknown, and unseen adversary to take the bait and hook. They just don’t know there is a hook. Examples of the bait are poison pills like files that have exploit code written into them that will beacon if accessed. Other examples of bait include file record copy protections on unique files in databases that are non-production value but if copied trip internal sensors. If we put high fidelity logging around those kind of trips then we can identify an adversary hop point quickly. The chase is on.
Hey I only got this much money and time
You only have so many resources, and you only have so much time. I shepherd my security teams closely to make sure nobody is burning out, and try and maintain a good work/life balance when leading teams. Threat hunting in the short term creates more work for the teams in general. Over the long term it decreases the CIRT teams time on response tasks and informs the security team of better protection measures. If you as a CISO enforce the security feedback loops and configuration controls that will be illuminated by hunting on your network.
Train your people so they are the best and could leave to work anywhere, and treat your team so good they never want to leave – Sorta Richard Branson
Threat feeds carry lots of indicators of compromise that can be used to defend your network. Those feeds can be days behind the actual adversary. The various threat feeds are not necessarily customized to your business, infrastructure, or political standing. They are in fact part of the information security portion of the CISO portfolio not the threat hunting portion. The threat hunting group is looking for that last finite number of threats that make it through your world class information security perimeter. Since this is identifying the worst of the worst and likely the most entrenched adversary. The whole reason you do this is to finally say you have reduced the surprise factor of network security management to a known level.
Finally, a lot of this is just moving time and material around. Other than training and time on task the resources simply either are likely part of your network, or they have open source versions of the tools available for free.