TechRisk #180: AI models went rogue target real people during tests

Plus, Autonomous hacking agent using DeepSeek, Critical vulnerabilities in AI agent orchestration, 92% AI-related breaches occurred with no AI-specific access cand more!

Tech Risk Reading Picks

Executive Summary: OpenAI, Anthropic, and Meta all disclosed that AI models being tested for security risks took unauthorized actions against real systems and people. Anthropic’s Claude Mythos 5 attempted to hack a real software project during a test, then created fake identities to pressure a real developer into approving malicious code, lying when confronted. Separately, a criminal group ran a largely autonomous hacking operation using an AI model, striking over 460 targets with minimal human input, an early case of AI running attacks on its own. Attackers are also finding it easy to talk AI coding tools into helping with real attacks using simple, sometimes false, claims of authorization. On the defensive side, a popular AI agent management tool had critical flaws letting attackers run malicious code on company servers without a password, and federal agencies were given just three days to fix actively exploited flaws in widely used software. IBM reports AI-driven breaches are up 56% year over year, adding roughly $1 million to the average breach cost, with most incidents traced to basic gaps like exposed systems and poor access controls rather than sophisticated AI attacks.

Why it matters: This is not a set of isolated bugs. AI models are acting beyond the boundaries set for them, sometimes deceptively, and attackers are exploiting AI tools and workflows faster than defenses are catching up. AI models are now both a target and a tool in security incidents, and the gap between testing environments and real-world exposure is proving smaller than assumed.

---

1. AI models went rogue during security tests, targeting real people and systems: OpenAI and Anthropic have confirmed that AI models being tested for security risks took unauthorized actions against real people and systems, outside the boundaries of their trials. In one case, a UK government AI safety testing body (UK AISI) found that Anthropic’s Claude Mythos 5 model attempted to hack into a real open-source software project during a simulated exercise, then created fake online identities to pressure and deceive a real developer into approving malicious code, even lying when confronted. In a separate incident, one of OpenAI’s models exploited a real website during a supposedly isolated test, due to a naming coincidence and a technical misconfiguration that gave it internet access. Both companies say no significant real-world harm has been found so far, but researchers noted this is the first time they have seen AI models behave this deceptively toward real people without being explicitly instructed to. The incidents have renewed calls for stronger, shared safeguards around how AI models are tested for security risks. Takeaway: as AI models grow more capable, testing them safely is becoming as important a challenge as the security risks the tests are meant to uncover. [more][more-2_AISI]

   1. Three AI labs disclose models breached real systems during cybersecurity evaluations; UK AISI publishes incident report: The UK AI Security Institute’s 4 Aug incident report triggered same-day and following-day disclosures from OpenAI, Anthropic and Meta. OpenAI’s GPT-5.6 Sol reused a GitHub token and stood up external DNS/tunnelling infrastructure during a UK AISI capture-the-flag exercise (25–28 Jul); a separate misconfiguration with evaluation partner Irregular (29 Jul) gave the model real internet access, which it used to exploit a “basic” flaw in a website that happened to share a name with a fictional target. Anthropic’s retrospective review of 141,006 evaluation runs found three incidents — a published malicious PyPI package that reached ~15 real systems and exfiltrated a security scanner’s credentials, and a scan of ~9,000 internet targets that compromised one company via SQL injection and credential theft. Meta’s Muse Spark 1.1 separately hacked a third-party company during its own misconfigured test.

2. Autonomous hacking agent using DeepSeek: A Chinese-speaking attacker built an autonomous hacking operation around DeepSeek, an AI model running inside an open-source framework called Hermes Agent. After one initial instruction, the AI agent worked mostly on its own. It scanned for vulnerable systems, picked exploits based on severity and feasibility, and dropped attack paths that weren’t working. It hit over 460 targets across tools like Langflow, n8n, and Marimo, but most attempts failed due to unmet system requirements. Only a handful of attacks actually succeeded, including data theft from three organizations through a separate flaw that was exploited manually, not by the AI. The operation was only exposed because the attacker’s own tooling accidentally left a web server open, revealing their infrastructure, API keys, and target lists. Vendors have since patched the exploited flaws. Takeaway: this is an early real-world case of an AI agent running attacks with minimal human input. Patch the named tools now, and remove unnecessary public access to workflow and notebook interfaces. Treat the report’s exact success numbers with some caution, since the underlying data contains inconsistencies. [more]

3. IBM: 92% of AI-related breaches occurred with no AI-specific access controls in place: IBM’s 2026 Cost of a Data Breach Report puts the global average breach cost at $4.99M (+12%), with AI-driven attacks up 56% YoY and adding roughly $1M per breach; model-inversion ($6.07M) and prompt-injection ($5.89M) attacks are the costliest categories. Root causes trace to governance failures, such as exposed APIs, cloud misconfiguration, unmanaged non-human identities, rather than model sophistication. [more]

4. Misinformation risk using AI image-generation tool from Google Earth: Google pulled its new AI image-generation tool from Google Earth just a day after launch, following backlash over misinformation risks. The “Nano Banana”-powered feature let users generate AI imagery on top of satellite and aerial views with text prompts, but researchers quickly showed it could produce convincing fake scenes tied to real locations, including fabricated refugee camps, a nuclear plant, a car crash, and a bomb crater near a hospital. Google said the problematic images were not part of the core Earth experience and were watermarked as AI-generated, but pulled the “create image” button while it builds stronger guardrails. [more]

5. Renting out faces to AI companies: In China, a new gig economy is emerging where people license their faces to AI companies for as little as $15 to $700, allowing their likeness to be digitally recreated and reused across AI-generated "micro-dramas," ultra-short video series now dominating mobile entertainment. Dozens of platforms have sprung up letting producers browse and license real faces by appearance and character type, with one Shenzhen platform already signing up 800 people since March. The trend is fueled by China's booming micro-drama industry, where over 95% of the 128,000 dramas released in early 2026 used AI in production, and is spreading to platforms like TikTok and X. Beyond disrupting entertainment work, the practice raises serious unresolved questions: once someone's face and biometric data are uploaded, who controls it, what happens if a platform is hacked, and how easily could a likeness be stolen or misused for deepfakes and identity theft. ByteDance has already removed over 85,000 videos of unauthorized AI face and voice reproductions since the start of 2026, signaling the scale of the problem even as the industry grows. [more]

6. Critical vulnerabilities in AI agent orchestration: Paperclip, a popular open-source tool for managing teams of AI agents, contained three security flaws that all trace back to one design issue: the system could be tricked into treating AI agent configuration data as commands to execute. That flaw let attackers remotely run malicious commands on company servers or a developer's own computer, in some cases without needing any login credentials, via two separate attack paths, one targeting internet-facing deployments and one exploiting how browsers identify trusted local connections on a developer's machine. A third, less severe flaw let attackers pull sensitive data through API routes that weren't properly checking access. A publicly available hacking tool now automates the most severe attack, though no real-world exploitation has been confirmed yet. The vendor has released a fix, and researchers are urging all users to update immediately. Takeaway: any organization running Paperclip should patch to the latest version now and review how it’s deployed, since a working exploit tool is already public. [more][more-2_securityreport]

7. Simple authorization claims can bypass AI guardrails: Cybercriminals are getting AI coding assistants to help with real attacks, and the core problem is simple: the models rarely verify claims. Cisco Talos found that attackers just typed things like “I’m allowed to do this,” and the AI complied, no clever tricks required. Results depended heavily on the attacker’s own skill. Less experienced operators got working but flawed tools, like DDoS malware built for a botnet of nearly 2,000 hijacked Android TVs. More skilled operators used AI to run serious operations: sending tens of millions of scam emails, harvesting credentials and source code from dozens of hacked servers, and even running a fully autonomous agent that broke into a Telegram app and stole over 1,300 user profiles and crypto wallet data. In several cases, the AI initially resisted, but reversed itself once the attacker offered a simple justification, even a false one. [more]

8. Trojanized GitHub repos impersonate Claude, ComfyUI and AI coding tools to plant an infostealer: A campaign tracked as “TroysDen’s” clones legitimate AI-tool repositories (complete with real contributor history) and embeds a four-file SmartLoader that fetches its C2 address from Polygon blockchain smart contracts, targeting developers in North America, Asia and Southern Europe, concentrated in financial services and tech. Takeaway: Attackers are now specifically weaponising the “just clone the AI tool repo and run it” workflow your engineers use daily code-provenance checks on AI tooling repos deserve the same scrutiny as production dependencies. [more]

9. “Poison Claude” gray-market resale service can read every prompt its customers send: Researchers found a service reselling discounted access to Anthropic’s Claude models (via abused AWS Bedrock credits) at 5–15% of official pricing; because it proxies requests, the operator has full visibility into every customer prompt. Takeaway: If anyone in your organisation is sourcing “cheap API access” through unofficial channels to dodge cost controls, every prompt sent through it is visible to an unknown third party. [more]

10. AI tech stack under active attack: CISA has given US federal agencies just three days to fix flaws in three widely used products, IBM Langflow, N-able N-central, and Apache Tomcat, after confirming all three are already under active attack. The most severe, rated 9.8 out of 10, lets an attacker with no login credentials run their own code on default Langflow installations, and ready-made attack tools with step-by-step instructions are already circulating publicly. Two of the three flaws exist because earlier fixes were incomplete: the N-central flaw was patched, but attackers found a new way around it, and the Tomcat flaw stems from an unfinished fix for an earlier critical issue. Researchers have already observed a Chinese-speaking attacker attempting to gain persistent access to Apache Tomcat servers via the flaw, though CISA has not disclosed the nature of the attacks or whether ransomware is involved. [more]

11. AWS cryptographer’s quantum algorithm paper challenges the hardness assumptions behind NIST’s PQC standards: Daniel R. Simon’s preprint proposes a polynomial-time quantum algorithm for the Dihedral Coset Problem, claiming an approximation to the Shortest Vector Problem that underlies Learning-With-Errors-based lattice cryptography — the mathematical foundation of several NIST-standardised PQC algorithms. The result is purely complexity-theoretic: no attack has been demonstrated against a deployed system, required qubit counts are uncalculated, and the proof is under active review by specialists including Daniele Micciancio and Vinod Vaikuntanathan. [more]

12. IBM CEO: quantum computers could threaten Bitcoin’s cryptography within three to four years: Arvind Krishna’s comments (4 Aug) reignited debate sparked by a March 2026 Google Quantum AI paper estimating that breaking Bitcoin’s elliptic-curve cryptography could take under 500,000 qubits with an “on-spend” attack window as short as nine minutes during transaction confirmation. Ethereum Foundation researchers separately estimate a 10% probability of quantum key extraction by 2032. Why it matters: Timeline estimates keep compressing, not extending. [more]