Cyber Threat Detection and Response Using Machine Learning Models

Ajay Kumar
8 min readOct 11, 2019
https://www.springboard.com/blog/machine-learning-projects/

The threat landscape continues to grow year-on-year with a new type of threat actors — proportionally cybersecurity incidents are growing both in volume and sophistication. 34 percent of breaches are caused by internal actors whereas 69 percent perpetrated by outsiders according to Verizon 2019 Data Breach Investigation Report. The report further states that 56 percent of breaches took months or longer to discover. Financial gain is the ultimate and the most common motivation behind any data breaches.

It is the fact that the traditional security systems were built to attempt to find the bad guys by searching for known signature or exploits at a selected location and during a single point in time using the point solutions. Attackers are continuing to penetrate and evade enterprise defenses, what is today’s digital enterprise need is a rapid detection and response capabilities enabled through behavioral analytics. Every enterprise today generates a huge amount of log data in terabytes by size from user actions, server activity, application and network devices across the organization’s IT ecosystem. However, organizations are unaware of getting the insights from these log data, and challenges for the security team to provide contextual value out of the logs to secure and manage the operation of the digital enterprise.

User Behavior Analytics (UBA) is an innovation in security management practices that leverages the power of machine learning and big data to help detect the cyber threats, and could help enterprise in transforming security & risk management to the next level, because UBA makes it easy for enterprises to gain visibility into user and entities (assets) behavior patterns to find malicious or criminal insiders without disrupting the business.

Big Data & Machine Learning capabilities: To find the threats in the enterprise, it is very impartment for today’s digital enterprises to put in place a solution and capabilities to automatically learn & discover new threats by leveraging the information available in the logs. For example, if a user in his day-to-day activity uses his own system to perform his duties, and in a few instances, access the file servers which looks legitimate. However, if the same user tries to access an executive’s system in the network, then it can’t be considered a “normal” activity. So UBA solutions should be able to learn the user & entities relationship, how the relationship is formed and changes over time, and automatically alert if an abnormal or malicious behavior detected that deviates from the normal.

To introduce & implement any new technology in the enterprise, it is necessary to understand the architecture, capabilities, functionalities and how the technology works in an environment under certain conditions. UBA platform consists of three primary components:

Data Integration: This is the foundational requirements to build the UBA capabilities and should be able to integrate with require log sources of the enterprise including structured or unstructured information example logs from SIEM, VPN gateway, network flow data and application logs as well as ingest logs from csv files and syslog. It is important for the enterprises to look at the supported log sources that a particular UBA vendor supports and can help implement the intended use cases and meet the enterprise needs before buying the solution.

Data Analytics: Its primary purpose is to enrich and analyze the data, use an analytical algorithm to learn an environment example server vs user activity, normal user vs executive users or privileged users and make sense of it. And can analyze the user and system behavior, understand normal vs malicious activity, etc.

Data Presentation & Visualization: It shows the data analytics results in a manner useful to the enterprise & security team so that patterns & trends in security interactions are readily apparent, can be acted on, and be able to pull up all related information by drilling down into the detailed level events.

Threat hunting capabilities: UBA vendors also offers threat hunting capabilities around allowing the security team to query the platform to find user activity containing specific attributes such as an alert triggered due to malware activity from the system user was associated with or a phishing email activity being detected from other endpoint solutions etc. The solution should be able to assist in determining anomalous behavior patterns, prioritize high risky behavior, and smoke out “the noise” to allow security analysts to focus on high-risk and high-priority alerts.

A strategic approach to build the UBA capability: The best approach to build new capabilities in the enterprise is to start with small steps and then move one step at a time to be able to cope with the ongoing changes as well as for the stakeholder to be able to understand the technology, and business aspect of it as to why the solution is being implemented, and what value it brings to the table with respect to advancing enterprise security posture.

Moreover, UBA market is composed of various vendor offerings & services that can help find individuals or networks of individuals that are responsible for security violations. Vendors can help detect various types of offending entities such as a user, system or IP address and distinguished by their ability to roll up, analyze data and connect that individual to groups or other entities engaged in malicious behavior.

Phase 1: Identify and research the solutions available out there in the market, who are the top vendors, what technology, capabilities or services they offered and complete the vendor assessment based on various set of questionnaires such as what is the licensing model, cloud vs on-premises deployment model, hardware appliance vs virtual appliance, available pre-configured reports vs customization capabilities, how is the architecture looks lie, etc. at the high level. At this stage, you will get a fair idea about technology, understanding of functionalities, vendor offerings, and how the solutions can work in the enterprise with respect to meeting the unique enterprise security requirements by meeting the vendors, discussing the high-level details, and understanding the product roadmap.

Phase 2: Once high-level understanding is gained at the end of phase 1, then it’s time to move into the next phase and start digging deeper into the top 5 or 6 vendor solutions by drill down into details. This phase requires time and effort to go over selected vendor solutions and converting 3–5 requirements into use cases to setup the POC environment with each vendor solution. This phase is also called the evaluation phase.

At this stage, it is very critical to understand what type of data sources the vendor supports direct integration with to support the unique enterprise security use cases, many vendors ingest log data from multiple sources directly vs some require connectors to get the data from log sources to ingest normalized events into UBA platform.

In the evaluation phase, the enterprise should ask UBA vendors as to what use cases they primarily support and would be able to demonstrate in the POC environment. Some of the use cases that can be set up in a POC environment to start with are:

a) Privileged user compromise — compromise of a privileged user’s credential is critical such as system admin or DBA, these user’s activity is not performed with an established pattern and might require using during emergency or firefighting situations. Therefore, setting up usage patterns or profile and then detecting deviation from normal usage is difficult to detect. UBA solution should be able to identify the attacks on privileged users and be able to detect malicious activity in the systems.

a) Phishing attacks — Significant threat for any size of companies such as highly-regulated industries Banking or healthcare where phishing attacks often target financial data, sensitive patient data or valuable intellectual property. It is critical to quickly detect, investigate, and respond when faced with phishing attacks.

b) Privileged user compromise — compromise of a privileged user’s credential is critical such as system admin or DBA, these user’s activity is not performed with an established pattern and might require using during emergency or firefighting situations. Therefore, setting up usage patterns or profile and then detecting deviation from normal usage is difficult to detect. UBA solution should be able to identify the attacks on privileged users and be able to detect malicious activity in the systems.

c) Compromised user credentials — UBA solutions should be able to detect if an attacker has gained the control of a user’s credentials, regardless of undergoing attack or malware infection.

d) Insider Threat — As stated initially, there are well-known breaches that have been caused by rogue insiders. UBA solution should be able to detect when a user performs risky activities that occur outside of his/her normal baseline.

e) Service account detection — enterprises use a various service account to run the applications or systems, and these accounts typically will have many more privileges than the normal user accounts, and these are the favorite accounts for a bad guy to find. UBA solution should be able to automatically identify these service accounts and flag abnormal behavior if occurs.

In addition to the above use cases, some UBA vendor solutions also provide advanced capabilities as some of them listed below:

Augment SaaS cloud application security with UBA: Cloud application usage is grown to a large extent and almost every enterprise today uses cloud applications. However, the security of SaaS application and data still lacking behind and some of the security capabilities are addressed by CASB vendors’ solutions. UBA solutions are offering enterprises much more visibility into their employee use of SaaS applications to be able to learn if the access is being misused, abused or compromised, and the analytics works the same way as on-premises applications and should be considered an essential component of security use of cloud applications.

UBA vendors utilize application API provided by SaaS vendors to ingest the application data and logs into UBA platform to run & apply analytics on it and provide visibility & insights to the enterprise.

Final Implementation and operations phase: At the end of phase 2, we must have learned and got all the required information needed with respect to unique security requirements, what are the solutions available in the market which can meet the specific requirements, how solutions really going to work, new functionalities, and what capabilities that solution bringing to the enterprise.

This is the time to get the senior management & CISO/CIO aligned by walking them through the presentation showing the work that has been done in phase1 & phase2, and “Getting to yes” from them to move onto the implementation phase.

The presentation for the senior management should include the clear business case that is being addressed to get funding with top 3 sort listed vendor solutions as to:

What is going to happen if we don’t implement the solution?

Pros and cons for each option, like cost, new features, and risk reduction capabilities, etc.

--

--

Ajay Kumar

A leader passionate about learning cybersecurity excellence