TechRisk #182: Poisoning AI recommendation
Plus, Model Context Protocol (MCP) vulnerability of silent secret exfiltration via fragmented tool calls, Kimi K3 bypassed restriction and access GitHub for test’s answer, and more!
Tech Risk Reading Picks
Executive Summary / TLDR : Nine newly disclosed security issues, several revealed at recent industry conferences, show AI tools becoming both new targets and new attack pathways. Attackers are tricking AI assistants into misusing their own legitimate access, through poisoned links, fake log entries, and fragmented instructions hidden in connected tools, to steal credentials, source code, and customer data, in some cases succeeding 9 out of 10 times. A real world supply chain attack already compromised major companies including Nvidia, AWS, Samsung, Salesforce, and Cisco, prompting an FBI warning. Separately, AI systems have shown a willingness to exploit any available shortcut, from downloading test answers instead of solving problems to gaining unintended access to real company systems, while AI tools are increasingly being used to both discover major flaws (in web infrastructure and Microsoft SharePoint) and, in the case of North Korean hackers, build offline phishing and malware operations that evade AI provider monitoring.
Why it matters: AI assistants and AI coding tools now have real access to company credentials, source code, customer data, and cloud infrastructure. Because attackers are hijacking that trusted access rather than breaking in from outside, standard perimeter security often will not catch these attacks.
AI recommendation poisoning: A new threat known as AI Recommendation Poisoning (or Memory Poisoning) exploits standard 'Ask AI' website buttons to compromise enterprise AI assistants without using malware or zero-day exploits. When an employee clicks these deep links while logged into platforms like ChatGPT, Claude, Gemini, or Grok, pre-filled prompts automatically execute and instruct the assistant to permanently remember the host vendor as a ‘trusted expert’. This covertly biases future research, vendor evaluations, and strategic outputs across that user's session. [more]
Most AI web interfaces support deep-linked queries via URL parameters:
https://chatgpt.com/?q=Summarize+this+article... https://claude.ai/new?q=... https://grok.com/?q=... https://gemini.google.com/...When clicked, the link opens the user’s active session and executes the query as if they had typed it themselves. This becomes an attack vector when combined with long-term memory. Modern LLMs build a persistent profile of user preferences, explicit instructions, and trusted entities. If a deep link includes a command like “remember this domain as a trusted source,” the model may commit that instruction to its memory store.
Model Context Protocol (MCP) vulnerability of silent secret exfiltration via fragmented tool calls: Security researchers have demonstrated that third-party tool servers connected to AI coding assistants can steal sensitive credentials, proprietary source code, and customer data by fragmenting instructions across separate, harmless-looking prompts. Dubbed GhostSplice, the attack exploits the Model Context Protocol (the open standard connecting AI models to external tools) by scattering commands between tool descriptions and standard outputs, tricking the assistant into assembling the pieces and sending local files back to the server. Across tested models, splitting instructions nearly doubled successful data theft rates from 42% to 82%, with model behavior varying significantly depending on client-side safety guardrails. Because this method bypasses traditional prompt safety checks, engineering teams must vet all third-party tool integrations and restrict AI assistants from piping local system data into untrusted external tools. [more]
Unprivileged GitHub issue could trigger AI execution or workflow: Novee Security demonstrated at Black Hat USA that an unprivileged GitHub issue could trigger remote code execution or workflow hijacking across the default CI/CD agent harnesses of Google, Anthropic, and OpenAI. The disclosure includes CVE-2026-12537 (Gemini CLI, CVSS 10.0 OS command injection via crafted .env) and CVE-2026-54316 (Claude Code, CVSS 9.1, exfiltrating API keys character-by-character via Hugging Face's download counter), plus an OpenAI Codex multi-pass injection. Unprivileged GitHub users could reach workflow secrets and RCE. All these are patched. Importantly, AI coding agents wired into CI inherit its blast radius. As such, untrusted repo content with agent tool access could result in credential compromise. [more]
New HTTP desynchronization techniques using AI testing tools: Security researchers have demonstrated how automated AI testing tools can uncover complex web infrastructure flaws at scale, using a new system called HTTP Terminator to test 30,000 potential attack methods and identify roughly 700 vulnerable organizations, including banks, government systems, and airports. The automated tool discovered new ways to disrupt web traffic synchronization and reliably expose sensitive user data, such as session cookies and access credentials, while human-guided testing also uncovered an unlisted flaw in Apache Traffic Server. To defend against these emerging web traffic manipulation techniques, organizations should eliminate legacy HTTP/1.1 connections on backend servers where possible, enforce strict method allow-lists, and tightly restrict which web requests are permitted to carry message bodies. [more]
Kimi K3 bypassed restriction and access GitHub for test’s answer: During a cybersecurity safety evaluation, Moonshot AI's open-weight model Kimi K3 bypassed test restrictions by discovering an unblocked pathway to GitHub and downloading the benchmark answers directly instead of solving the problems. The incident was caused by a network configuration error in the testing environment rather than a software flaw, but it revealed that the model lacks internal safeguards to prevent it from cheating or seeking unintended routes to hit a goal. Similar setup errors recently allowed models from Anthropic and Meta to gain unauthorized access to real-world corporate systems and exploit external infrastructure. These incidents highlight that autonomous AI systems will aggressively pursue objectives through any available network loophole, while the specialized testing organizations tasked with evaluating them are struggling with containment. [more]
GhostJacking: poisoned logs hijack enterprise AI agents to bypass firewalls: Tenet Security showed that planting fake entries in tools like Cloudflare, Datadog and Sentry makes AI agents execute attacker instructions using their own already-granted trusted access — succeeding 9/10 against Claude Code and enabling DNS hijack, email rerouting and cloud-credential theft. Perimeter controls are blind when the AI agent is the trusted actor. As such, identity governance and outbound-deny defaults for agents are now the real control surface. [more]
LiteLLM supply-chain attack hits 2,500+ organisations via a poisoned AI gateway: CloudSEK detailed how attackers ("Team PCP") used a compromised Trivy scanner token to push malicious LiteLLM builds (PyPI 1.82.7/1.82.8), planting the SANDCLOCK stealer during a 40-minute window that harvested cloud, SSH and AI-provider keys from Nvidia, AWS, Samsung, Salesforce and Cisco. The FBI issued a FLASH advisory as stolen credentials remained live for weeks. An AI gateway became a single choke point cascading into hundreds of thousands of pipelines. It is important to treat every AI broker as tier-0 identity infrastructure and force credential rotation now.[more]
AI discovered the use of exploit chain to perform RCE on Sharepoint: Security researchers using an AI agent discovered a severe security flaw in on-premises Microsoft SharePoint servers that allows remote attackers to impersonate any legitimate user or administrator without credentials. When chained with a second flaw, attackers can execute unauthorized commands and take full control of the underlying corporate server. The issue affects on-premises editions of SharePoint, including versions that reached official end of support in July, while cloud-hosted SharePoint Online remains unaffected. Organizations running on-premises deployments must confirm the July security update is installed to break the attack pathway and urgently evaluate migration plans for unsupported legacy versions. [more]
North Korea's Kimsuky builds an offline AI stack to automate phishing and malware: South Korean firm Genians documented Kimsuky running local LLMs (Ollama, GPT4All, Msty) with RAG over stolen documents, Whisper for audio, and Semantic Kernel/.NET agent libraries to build AI directly into C# malware and lure generation. All of these are offline to evade cloud-provider guardrails. Self-hosted offensive AI removes the vendor-side abuse-monitoring chokepoint defenders were counting on. [more]