Sophos excels in the 2024 MITRE ATT&CK® Evaluations: Enterprise

Sophos has once again achieved exceptional results in the latest 2024 MITRE ATT&CK Evaluations for Enterprise. In this round, Sophos XDR achieved:

The highest possible (‘Technique’) ratings for  100% of adversary activities in the Windows and Linux ransomware attack scenarios
The highest possible (‘Technique’) ratings for 78 out of 80 total adversary activities across all three comprehensive scenarios
‘Analytic coverage’ ratings for 79 out of 80 total adversary activities activities

The eagerly anticipated results of the sixth round of MITRE ATT&CK^® Evaluations for Enterprise have been released, assessing the ability of nineteen endpoint detection and response (EDR/XDR) solutions to accurately identify and report the malicious activities of sophisticated threat groups.

Watch this short video for an overview of the evaluation:

What are MITRE ATT&CK^® Evaluations?

MITRE ATT&CK® Evaluations are among the world’s most respected independent security tests. They emulate the tactics, techniques, and procedures (TTPs) leveraged by real-world adversarial groups and evaluate each participating vendor’s ability to detect, analyze, and describe threats, with output aligned to the language and structure of the MITRE ATT&CK^® Framework.

There is no singular way to interpret the results of ATT&CK Evaluations, and they are not intended to be competitive analyses. The results show what the evaluation observed and do not result in a “winner” or “leader” – despite what some vendors might like you to think!

There is nuance in the ways each vendor’s tool works and how it presents information to the analyst using it, and your individual needs and preferences play a vital role in determining which solution is best for you and your team. Learn about Sophos Extended Detection and Response (XDR)

Evaluation overview

This was the sixth round of ATT&CK Evaluations for Enterprise — MITRE’s product-focused evaluation — designed to help organizations better understand how endpoint detection and response (EDR) offerings like Sophos XDR can help them defend against sophisticated, multi-stage attacks.

This round focused on behaviors inspired by three known threat groups:

Democratic People’s Republic of Korea (DPRK). The evaluation emulated DPRK’s adversary behaviors targeting macOS via multi-stage operations, including elevating privileges and credential theft.
CL0P and LockBit Ransomware. The evaluation emulated behaviors prevalent across campaigns using CL0P and LockBit ransomware targeting Windows and Linux platforms, including the abuse of legitimate tools and disabling critical services.

Evaluation participants

Nineteen EDR/XDR solution vendors participated in this evaluation round (in alphabetical order):

Understanding the results

Each adversary activity (called a ‘sub-step’) emulated during the evaluation received one of the following ratings, indicating the solution’s ability to detect, analyze, and describe the adversary activity, with output aligned to the language and structure of the MITRE ATT&CK® Framework.

Not applicable — a “miss”: The adversary activity was not detected or the evaluation for the sub-step was not completed.
None: Execution of the sub step was successful; however, evidence provided did not meet the documented Detection Criteria, or there was no evidence of Red Team activity provided.
General: The solution autonomously identified that the malicious/suspicious event(s) occurred and reported the What, Where, When, and Who.
Tactic: In addition to meeting the criteria for a ‘General’ rating, the solution also provided information on the attacker’s potential intent; the Why, aligned to MITRE ATT&CK Tactics.
Technique — the highest possible rating: In addition to meeting the criteria for a ‘Tactic’ rating, the solution also provided details on the attacker’s method for achieving a goal; How the action was performed.

Detections classified as General, Tactic, or Technique are grouped under the definition of Analytic Coverage, which measures the solution’s ability to convert telemetry into actionable threat detections.

How did Sophos perform in this evaluation?

Throughout the evaluation, MITRE executed three discrete attack scenarios (DPRK, CL0P, and LockBit), comprising a total of 16 steps and 80 sub-steps.

Sophos XDR delivered impressive results, achieving:

The highest possible (‘Technique’) ratings for  100% of adversary activities in the Windows and Linux ransomware attack scenarios
The highest possible (‘Technique’) ratings for 78 out of 80 total adversary activities across all three comprehensive scenarios
‘Analytic coverage’ ratings for 79 out of 80 total adversary activities activities

Attack scenario 1: DPRK (macOS only)
North Korea has emerged as a formidable cyber threat, and by expanding its focus to macOS, they have gained the ability to target and infiltrate additional high-value systems. In this attack scenario, the MITRE team used a backdoor from a supply chain attack, followed by persistence, discovery, and credential access, resulting in the collection and exfiltration of system information and macOS keychain files.

This scenario comprised 4 steps with 21 sub-steps on macOS only.

Sophos XDR detected and provided rich ‘analytic’ coverage for 20 out of 21 sub-steps (95%) in this scenario.
19 sub-steps were assigned ‘Technique’ level categorization — the highest possible rating.

Attack scenario 2: CL0P ransomware (Windows)
Active since at least 2019, CL0P is a ransomware family affiliated with the TA505 cyber-criminal threat actor (also known as Snakefly) and is widely believed to be operated by Russian-speaking groups. The MITRE team used evasion techniques, persistence, and an in-memory payload to perform discovery and exfiltration before executing ransomware.

This scenario comprised 4 steps with 19 sub-steps on Windows only.

Sophos XDR detected and provided full ‘technique’ level coverage — the highest possible rating — for 100% of sub-steps in this scenario.

Attack scenario 3: LockBit ransomware (Windows and Linux)
Operating on a Ransomware-as-a-Service (RaaS) basis, LockBit is a notorious ransomware variant that has gained infamy for its sophisticated tools, extortion methods, and high-severity attacks. The MITRE team gained access using compromised credentials, ultimately deploying an exfiltration tool and ransomware to stop virtual machines and exfiltrate and encrypt files.

This scenario comprised 8 steps with 40 sub-steps on Windows and Linux.

Sophos XDR detected and provided full ‘technique’ level coverage — the highest possible rating — for 100% of sub-steps in this scenario.

Learn more at sophos.com/mitre and explore the full results on the MITRE website.

How do Sophos’ results compare to other participants?

As a reminder, there’s no singular way to interpret the results of ATT&CK Evaluations, and you will see different charts, graphs, and other visualizations created by participating vendors that frame the results in different ways.

Detection quality is critical for providing details on the adversary’s behavior so analysts can investigate and respond quickly and efficiently. Therefore, one of the most valuable ways to view the results of ATT&CK® Evaluations is by comparing the number of sub-steps that generated a detection that provided rich detail on the adversarial behaviors (analytic coverage) and the number of sub-steps that achieved full ‘technique’ level coverage.

How to use the results of MITRE ATT&CK Evaluations

When considering an EDR or extended detection and response (XDR) solution, review the results from ATT&CK Evaluations alongside other reputable third-party proof points, including verified customer reviews and analyst evaluations. Recent third-party recognitions for Sophos XDR include:

As you review the data available in the MITRE portal for each participating vendor, consider the following questions as they pertain to you, your team, and your organization:

Does the evaluated tool help you identify threats?
Does it present information to you the way you want it?
Who will be using the tool? Tier 3 analysts? IT specialists or Sysadmins?
How does the tool enable you to conduct threat hunts?
Are disparate events correlated? Is that done automatically, or do you need to do that on your own?
Can the EDR/XDR tool integrate with other technology in your environment (e.g., firewall, email, cloud, identity, network, etc.) including solutions from other vendors?
Are you planning to use the tool by yourself, or will you have the support of a Managed Detection and Response (MDR) partner?

Why we participate in MITRE ATT&CK Evaluations

MITRE ATT&CK Evaluations are among the world’s most respected independent security tests due to the emulation of real-world attack scenarios and transparency of results. Sophos is committed to participating in these evaluations alongside some of the best security vendors in the industry. As a community, we are united against a common enemy. These evaluations help make us better, individually and collectively, for the benefit of the organizations we defend.

Get started with Sophos XDR

Our results in this latest evaluation further validate Sophos’ position as an industry-leading provider of endpoint detection and response (EDR) and extended detection and response (XDR) capabilities to over 43,000 organizations worldwide.

Visit our website or speak with an expert to see how Sophos can streamline your detection and response and drive superior outcomes for your organization today.

Source: Sophos