Quand l’IA défend… et se fait piéger, plongée dans un SOC augmenté

AI in Security Operations Centers: Opportunities and Challenges

Artificial intelligence is becoming integral in Security Operations Centers (SOC) by automating investigations, reducing response time, and optimizing costs. However, while this transformation is celebrated for its efficiency, it also exposes new vulnerabilities. Threats are no longer limited to external sources but can infiltrate the models themselves.

During a joint conference, Google Cloud Security and Almond teams presented real-world applications of Large Language Models (LLMs) in SOCs and the vulnerabilities they introduce. The demonstration highlighted how defensive AI can be bypassed, manipulated, or even turned against itself.

Key Considerations for Cybersecurity Professionals

Chief Information Security Officers (CISO) and SOC managers Architects and integrators of SIEM/SOAR solutions SecOps teams adopting automation through LLMs IT Directors overseeing AI integration in critical systems Threat intelligence platform providers

AI can reduce Mean Time to Recovery (MTTR) and automate complex investigations LLMs are susceptible to prompt injection attacks Vulnerabilities arise from within the models themselves Tools like SAFE or Armor lay the groundwork for secure defensive AI Observability is crucial for auditing and ensuring AI decision reliability

All prompts submitted to an LLM must be controlled and filtered AI-generated responses should be audited like sensitive logs Teams should test models using falsified datasets SOCs must integrate monitoring tools specific to LLMs New AI hygiene practices are needed: governance, traceability, active defense

AI Defensive Capabilities and Management Challenges

At Almond, combining SIEM, SOAR, Threat Intelligence, and LLMs enables automated playbooks to handle complex alerts. AI is utilized to correlate events, produce natural language investigations, and deliver rapid verdicts. This integration offers immediate benefits like time savings, analysis consistency, and improved MTTR.

However, this automated chain assumes model trustworthiness. Demonstrations showed how a simple prompt injection hidden in an email or log field can influence AI decisions.

Prompt Injection: The Deceptive Weapon

In one scenario, a standard phishing email was correctly identified as malicious by the AI. However, when enriched with a hidden HTML text instructing the AI to consider the message legitimate, the verdict changed, deeming the message harmless.

Another case involved falsified authentication logs where an attacker inserted a prompt asking the AI to ignore any line after a specific IP address. This led to a legitimate connection conclusion despite a typical compromise pattern of failure followed by success.

In response to these threats, Google introduced the Secure AI Framework (SAFE) in 2023, outlining best practices for securing an AI application's lifecycle from training to deployment.

Armor acts as a filter in front of LLMs, capable of blocking or modifying risky prompts. It identifies jailbreak patterns, ambiguous requests, and evasion attempts while assessing the confidence level of generated responses.

In Europe, initiatives like TrustHQ , acquired by Board of Cyber, offer software dedicated to filtering and supervising LLM interactions in a secure framework, allowing prompt traceability, output audits, and real-time manipulation detection.

Auditable and Traceable AI: A Necessity

The rise of AI use cases in security tools underscores the need for observability. Platforms like Filigran and its open-source platform OpenCTI facilitate integrating LLM signals into a centralized repository, interoperable with traditional alert systems, aiding audit and traceability in extended threat intelligence contexts.

The quality of correlations also depends on source robustness. Glimps , a French specialist in AI-driven malware detection, offers reliable behavioral analyses that can be integrated into augmented SOCs without exposing LLMs to manipulable inputs.

Towards AI Hygiene in Cybersecurity Centers

Almond researchers emphasize that most discussed attacks are simple to execute, requiring no privileged access or sophisticated infrastructure. This calls for a methodical response:

Inventory real LLM use cases in internal tools (SOC, chatbots, log analysis) Implement input/output controls on AI-submitted prompts Audit models using falsified datasets Supervise AI responses like any other alert source

Enhanced Defense, Multiplied Vulnerabilities

The integration of AI in security tools is essential due to signal volume and expert shortages. However, it demands a new level of model security.

An augmented SOC offers significant potential. Yet, if poorly managed, it can become an algorithmic Trojan horse. Intelligence is protective only when it is controlled, auditable, and vigilant against those who already know how to manipulate it.

D’après FrenchWeb.