CrowdStrike and Meta have launched CyberSOCEval, an open-source benchmark suite to assess AI models’ performance in security operations centers, addressing the challenge of selecting effective AI-powered cybersecurity solutions amid a rapidly evolving threat landscape.
The partnership between CrowdStrike and Meta tackles the issue of numerous AI tools with varying capabilities and costs, making it difficult for organizations to determine which systems deliver a true AI advantage against real-world attacks. As CrowdStrike stated, “Without clear benchmarks, it’s difficult to know which systems, use cases, and performance standards deliver a true AI advantage against real-world attacks.” CyberSOCEval tests large language models on essential cybersecurity tasks, including incident response, threat analysis comprehension, and malware testing, providing empirical evidence of AI systems’ strengths and weaknesses.
Beyond evaluation, the framework offers insights into how enterprise clients deploy LLMs in cybersecurity contexts, equipping developers to create more specialized and effective models. This could accelerate advancements in AI tailored for security, ultimately strengthening defenses in high-stakes environments like financial services. The launch highlights the escalating “digital arms race” in cybersecurity, where AI empowers both attackers and defenders. Malicious actors are leveraging AI for novel threats, such as automated password brute-forcing, while SOCs integrate AI into their operations to detect anomalies, analyze threats, and automate responses.
Real-world evidence demonstrates the potential rewards of AI-powered cybersecurity tools. A recent survey by Mastercard and the Financial Times’ Longitude revealed that financial services firms have saved millions of dollars by deploying AI-powered tools to combat AI-enabled fraud. However, without reliable benchmarks like CyberSOCEval, organizations risk investing in underperforming tools that fail to counter emerging risks. Meta’s commitment to open-source principles is central to this project, allowing developers to access model weights and source code, fostering collaborative innovation.
Vincent Gonguet, Director of Product for Generative AI at Meta’s Superintelligence Labs division, emphasized the collaborative potential: “With these benchmarks in place, and open for the security and AI community to further improve, we can more quickly work as an industry to unlock the potential of AI in protecting against advanced attacks, including AI-based threats.” The timing of CyberSOCEval is particularly relevant as businesses prepare for 2025’s projected surge in AI-driven cyber threats, including deepfake-enabled phishing and AI-optimized malware.
CyberSOCEval is freely available on GitHub, inviting contributions from the global security and AI communities. Additional resources, including detailed benchmark specifications and usage guides, are hosted on the project’s dedicated website. This open model contrasts with closed ecosystems, potentially democratizing access to high-quality AI security tools and reducing barriers for smaller organizations. As AI’s role in cybersecurity deepens, initiatives like CyberSOCEval represent a pivotal step toward responsible innovation, promoting a shared understanding of AI’s limits and possibilities in defending against an increasingly intelligent adversary.
In summary, the collaboration between CrowdStrike and Meta signals a maturing AI cybersecurity market, where benchmarks drive accountability and progress. With threats multiplying, tools like CyberSOCEval are essential for ensuring that AI serves as a shield rather than a vulnerability.
