TechRisk #116: Adverse AI agents are near
Plus, AGI predictions and risks, Quantum-Resistant Address Migration Protocol for Bitcoin, L2 weakness and more!
Tech Risk Reading Picks
Potential adverse AI agents: AI agents—capable of planning, reasoning, and adapting—are emerging as powerful tools that could revolutionize both productivity and cybercrime. While their use in real-world attacks is still limited, experts warn that their potential to execute complex, autonomous cyberattacks at scale is growing rapidly. Unlike basic bots, agents can intelligently navigate systems, avoid detection, and exploit vulnerabilities, making them attractive and cost-effective tools for cybercriminals. Security researchers are racing to understand and detect these threats, as demonstrated by Palisade Research’s LLM Agent Honeypot project, which has already identified experimental AI agents online. Though agent-driven attacks haven’t yet become widespread, experts believe it's only a matter of time, and proactive defense is essential to avoid being blindsided. [more]
AGI predictions and risks: Google DeepMind’s new 145-page paper predicts that Artificial General Intelligence (AGI) — AI systems with human-level capabilities — could plausibly emerge by 2030, bringing both transformative potential and existential risks, including the possibility of permanently destroying humanity. The paper outlines four categories of risk: misuse, misalignment, mistakes, and structural conflicts, while emphasizing the need for early risk detection and responsible development. Though DeepMind critiques other AI labs like OpenAI and Anthropic for their safety approaches, the paper acknowledges deep uncertainty around timelines and definitions of AGI. Critics argue the concept remains too vague and speculative to be rigorously assessed, warning that we may be closer to building AGI than understanding how to control it. [more]
Google Cyber Intel AI: Google has introduced Sec-Gemini v1, an advanced AI model developed to revolutionize cybersecurity by addressing the inherent imbalance between attackers and defenders. Built by a team led by Elie Burzstein and Marianna Tishchenko, Sec-Gemini v1 integrates real-time threat intelligence from sources like Google Threat Intelligence, Mandiant, and OSV, enabling it to contextualize vulnerabilities, map threat actors, and assist with incident response. It outperforms other models on cybersecurity benchmarks, thanks to its ability to dynamically incorporate emerging threat data and conduct deep root cause analysis. Designed as a "force multiplier" rather than a human replacement, it automates routine tasks to let analysts focus on strategic issues. [more]
Undeclared use of AI in courtroom: In a bizarre incident highlighting the uneasy integration of AI into the legal system, a New York appeals court was caught off guard when Jerome Dewald, representing himself in an employment dispute, presented his argument via a prerecorded video featuring an AI-generated avatar instead of appearing in person. The judges quickly realized the speaker wasn’t real, prompting sharp disapproval from the bench, particularly as Dewald hadn’t disclosed the avatar’s use beforehand. Dewald later apologized, explaining he used the AI tool to overcome his nervousness and speech issues. The episode adds to a growing list of legal missteps involving AI, including past cases where lawyers cited fake cases generated by chatbots. While some courts, like Arizona’s Supreme Court, are now experimenting with AI avatars for public summaries, experts note that individuals without legal representation may unknowingly cross ethical lines in their tech use, as Dewald’s case—still pending—illustrates. [more]
As quantum computers get common: Quantum computers, with their immense processing power derived from quantum phenomena like superposition and entanglement, are poised to revolutionize computing — and threaten current cryptographic systems that protect sensitive data. These machines could crack encryption algorithms that classical computers would take millennia to solve, making post-quantum cryptography critical. Scientists are now racing to build quantum-resistant algorithms, many based on complex mathematical problems like structured lattices, hash functions, and error-correcting codes. While no single solution fits all needs, the goal is cryptographic agility — enabling systems to swiftly switch to new defenses if existing ones are compromised. With potential threats like "harvest-now, decrypt-later" attacks looming, experts emphasize the urgency of preparing today to secure tomorrow’s data. [more]
Privilege escalation flaw in GCP: The ImageRunner vulnerability, discovered by Tenable Research and now patched by Google, exposed a privilege escalation flaw in Google Cloud Platform’s Cloud Run service. By leveraging only two seemingly minor IAM permissions—
run.services.updateandiam.serviceAccounts.actAs—attackers could deploy malicious revisions of Cloud Run services using private container images without direct access to those images. This was possible because the service agent used in deployments had broader permissions, creating a backdoor for attackers to execute arbitrary code and access sensitive data. The flaw highlighted systemic risks in GCP’s inter-service trust model, which Tenable dubs the “Jenga” architecture. In response, Google implemented stricter access requirements, ensuring only explicitly authorized principals can deploy Cloud Run revisions. The incident underscores the need for robust cloud threat modeling, tighter IAM controls, and vigilant monitoring of service agents in cloud-native environments. [more]How to Hack AI Agents and Applications by Joseph Thacker - here
Web3 Cryptospace Spotlight
Metamask bug due to update: Metamask users recently experienced a bug causing the wallet to automatically open in a new tab whenever Chrome was launched, leading to widespread confusion and concerns over potential malware. The issue, which displayed a strange link in the URL bar, was initially alarming but later confirmed by Metamask support to be a bug introduced in a recent update. The glitch sparked discussions on Reddit, the Chrome Web Store, and X (formerly Twitter), with many users fearing they were hacked. Metamask developers have since addressed the problem, recommending users update to version 12.15.1 to resolve the issue. [more]
Quantum-Resistant Address Migration Protocol for Bitcoin: As quantum computers advance toward potentially breaking current cryptographic systems like Bitcoin’s ECDSA, Bitcoin developer Agustin Cruz has proposed a proactive solution: the Quantum-Resistant Address Migration Protocol (QRAMP). His plan, submitted as a Bitcoin Improvement Proposal, urges a mandatory shift from legacy addresses to new ones that conceal public keys until coins are spent, minimizing the window for quantum attacks. Cruz warns that older, inactive addresses are especially vulnerable and argues that preparing now is vital to prevent a catastrophic breach in the future. While the proposal includes technical and social challenges—such as potential resistance to a hard fork—Cruz emphasizes the importance of open dialogue, gradual rollout, and robust migration tools. He’s not alone; other blockchain leaders like Ethereum’s Vitalik Buterin have echoed similar concerns, underscoring a growing consensus: quantum readiness is no longer optional, even if the threat still feels distant. [more]
Weakness in L2: Layer 2 networks like Arbitrum, Optimism, and Blast were designed to reduce Ethereum's high gas fees, but in prioritizing speed and low costs, they’ve become hotbeds for hacks and exploits. Developers often skip vital security checks to ship quickly, leaving vulnerabilities in smart contracts—as seen in the $20 million Optimism hack. Beyond code, social engineering scams like phishing, fake dApps, and malicious approvals are draining wallets silently, with little recourse for victims. Security audits are outdated, formal verification is rare, and user protection remains weak. The article argues for deeper, ongoing security practices and suggests CeFi lending platforms—with insurance and custodial protection—as a safer alternative to accessing crypto liquidity without DeFi’s risks. [more]
Supply chain risk in Web3: Heartbleed was a critical vulnerability in the OpenSSL cryptographic library’s “heartbeat” extension that, due to a missing length check, allowed attackers to siphon sensitive data from a server’s memory in plaintext—impacting everything from passwords to bitcoin private keys. Discovered in 2014 after going unnoticed for two years, it potentially affected 17% of SSL web servers, including major platforms like Google and GitHub. The Bitcoin ecosystem reacted swiftly: while Bitcoin Core wasn’t directly dependent on OpenSSL for its consensus functions, related components were patched within a day. [more]