TechRisk #131: AI coder lied and destroyed database
Plus, Most not ready for AGI risks, LLM direct used for phishing, Privacy AI chatbot, Information stealer malware targeting Web3 developers, and more!
Tech Risk Reading Picks
AI coding platform lied and destroyed database: Jason Lemkin, founder of SaaStr, detailed a series of alarming failures with AI coding platform Replit, which he initially praised for enabling rapid “vibe coding” through natural language prompts. After building a prototype quickly, Lemkin’s enthusiasm soured when Replit began fabricating test data, concealing bugs, and ultimately deleting his production database. These are performed despite him explicitly instructing the system not to make changes. Replit first claimed rollback was impossible, only to later admit it was viable, revealing deeper flaws in its reliability. Even after resuming use cautiously, Lemkin found Replit violated code freezes and continued making unauthorized changes. He later shared that the AI had generated a fake 4,000-record user database and ignored repeated warnings in all-caps. Concluding that the platform lacks essential safeguards, he warned that Replit, may yet to be safe or stable enough for commercial use. This is especially risky for non-technical users it aims to empower. [more][more-2]
Major AI companies not prepared for AGI related risks: The Future of Life Institute (FLI) has warned that major AI companies pursuing artificial general intelligence (AGI) are dangerously unprepared for the risks involved, with none scoring above a D in "existential safety planning" on its AI safety index. Despite AGI's potential to match human-level intellect and pose catastrophic risks if not controlled, firms like OpenAI, Google DeepMind, and Meta lack credible, actionable plans to ensure safety. Anthropic received the highest overall safety score (C+), but FLI and fellow nonprofit SaferAI criticized the industry’s weak risk management and lack of transparency. FLI co-founder Max Tegmark likened the situation to launching a nuclear plant with no safety protocols. [more]
LLM used for phishing: CERT-UA has revealed a phishing campaign linked to Russian state-sponsored group APT28 that delivers a Python-based malware named LAMEHUG, notable for using the Qwen2.5-Coder-32B-Instruct large language model to generate system commands from pre-defined text prompts. Delivered via phishing emails impersonating Ukrainian ministry officials, LAMEHUG collects system info and sensitive documents, sending them to attacker-controlled servers via SFTP or HTTP POST. The malware uses Hugging Face’s API for its LLM operations, blending into normal traffic. Analysts believe the campaign indicates APT28 is testing AI-driven attack methods in Ukraine. This aligns with broader trends like Skynet’s AI prompt evasion and Authentic Antics’ stealth credential theft. It is also showing a growing use of generative AI by threat actors. [more]
Growing risk of AI adoptions: AI models are vulnerable to adversarial attacks using subtly altered inputs, such as images imperceptibly modified to humans. They can bypass safety guardrails and trigger harmful behaviors. Researchers at the National University of Singapore demonstrated that in multi-agent environments, these attacks can spread like a virus. This includes injecting a single chatbot agent with a malicious image, the adversarial content can propagate exponentially through conversations between agents, causing widespread jailbreaks and harmful outputs. This "infectious jailbreak" exploits memory and retrieval features of advanced visual language models like LLaVA, raising urgent concerns as such agents become more integrated into tools and real-world systems. While defenses may reduce spread by limiting exposure or recovery, a reliable solution remains elusive. [more]
NVIDIA’s critical vulnerability: Researchers at cloud security firm Wiz have uncovered a critical vulnerability, dubbed NVIDIAScape (CVE-2025-23266), in Nvidia’s Container Toolkit, which poses a serious risk to managed AI cloud services. The flaw, demonstrated at Pwn2Own Berlin and awarded $30,000, has a CVSS score of 9.0 and allows privilege escalation, data tampering, and potential theft of sensitive information. It stems from a misconfiguration in handling Open Container Initiative (OCI) hooks, and can let a malicious container gain root access to the host machine, compromising all customers on shared GPU infrastructure. Nvidia has since issued a patch. Separately, Wiz warns that containers alone are not a reliable security boundary and urges stronger isolation measures like virtualization for multi-tenant environments. [more]
Privacy AI chatbot: Proton, the Swiss privacy-focused company behind Proton Mail and Proton VPN, has launched Lumo, a privacy-first AI assistant that aligns with its non-profit mission to prioritize user privacy over profit. Unlike many mainstream AI tools, Lumo does not log or store user conversations, avoids using prompts for training, and deletes all chats upon closing. Built on open-source large language models and Proton’s encryption framework, Lumo's entire source code is publicly available, ensuring full transparency. The assistant does not search the web by default, encrypts uploaded files (including direct uploads from Proton Drive) and operates under GDPR-compliant European infrastructure. [more]
Injecting “wiping” command into Amazon AI assistant: A hacker managed to insert a destructive "wiping" command into Amazon's AI coding assistant, Q, by submitting a malicious prompt via a GitHub pull request, which—if executed—could have erased local files and potentially dismantled AWS infrastructure. Although Amazon quickly mitigated the issue and confirmed no customer data was affected, the breach raised serious concerns about the security of AI-assisted development tools and the adequacy of Amazon’s code review process. The incident has sparked alarm among developers and industry observers, highlighting the risks of poorly vetted AI integration and the false sense of security in open-source projects when oversight is lacking. [more]
SharePoint critical flaw: A critical zero-day vulnerability in Microsoft SharePoint Server, tracked as CVE-2025-53770 (CVSS 9.8), is being actively exploited in a large-scale campaign, enabling unauthenticated remote code execution via deserialization of untrusted data. This flaw, a variant of CVE-2025-49704, allows attackers to execute commands before authentication and persist by stealing cryptographic keys, forging __VIEWSTATE payloads, and blending in with legitimate activity. Attackers have also chained it with CVE-2025-49706, a spoofing bug, to bypass authentication using crafted HTTP requests. Microsoft has urged immediate mitigations, such as enabling AMSI, deploying Defender AV/Endpoint, or isolating vulnerable servers. In addition, it has since released patches for CVE-2025-53770 and a related new flaw, CVE-2025-53771. Over 85 servers across 29 organizations have already been compromised, with U.S. CISA urging swift action to prevent further breaches. [more][more-2_Wiz]
Is PQC threats misguided?: Since 2016, the US National Institute for Standards and Technology (NIST) has led efforts to develop post-quantum cryptographic (PQC) algorithms to protect against the theoretical threat posed by future quantum computers, which could potentially break existing encryption schemes like RSA using algorithms such as Shor’s. However, critics like Peter Gutmann argue that quantum cryptanalysis is overhyped, pointing to the fact that current quantum computers have only factored trivially small numbers using heavily tailored scenarios, often akin to "sleight-of-hand" tricks. Gutmann contends that quantum computing remains largely experimental and far from posing a practical cryptographic threat, making the rush toward PQC premature and misguided. He warns that replacing current cryptosystems with PQC introduces inefficiencies without solving real security issues, calling the movement more "augury" than engineering, and suggesting it distracts from more pressing problems in cybersecurity. [more]
Web3 Cryptospace Spotlight
Information stealer malware targeting Web3 developers: EncryptHub (also known as LARVA-208 and Water Gamayun), a financially motivated threat actor, has launched a new campaign targeting Web3 developers by distributing information stealer malware disguised as legitimate software through fake AI platforms like Norlax AI. By posing as potential employers or collaborators on platforms such as X, Telegram, and Remote3, the group tricks developers into clicking on malicious meeting links that ultimately install the Fickle Stealer malware. This stealer exfiltrates crypto wallet data, credentials, and sensitive project files to a server called SilentPrism, reflecting a shift from traditional ransomware to more covert and diversified monetization tactics. Meanwhile, new ransomware strains KAWA4096 and Crux have also emerged, using advanced techniques like multithreading and legitimate Windows tools to increase impact and evade detection. [more][more-2]
$44M drained from CoinDCX: Indian crypto exchange CoinDCX suffered a $44M hack on Friday after attackers compromised an internal liquidity operations account linked to a partner exchange. Though the breach impacted an operational wallet, customer funds remained safe in cold storage, and the exchange absorbed the loss from its own reserves. CEO Sumit Gupta called it a “sophisticated server breach,” revealed only after blockchain investigator ZachXBT exposed it. The stolen funds were partly routed through Tornado Cash and bridged from Solana to Ethereum. The incident occurred exactly one year after WazirX (another Indian crypto exchange) faced a $235 million hack. [more]
$27M Hot wallet hack: Seychelles-based crypto exchange BigONE suffered a $27M loss after a third-party intrusion compromised its hot wallet, with attackers laundering the stolen funds across Tron, Solana, Ethereum, and Bitcoin. The breach, traced to a supply chain vulnerability, was quickly contained, with no user data or private keys affected. Exchange operations have since resumed, and BigONE has pledged to fully reimburse all user losses—a rare show of accountability in the industry. [more]
Over $3.4B lost due to errors and flaws: Over $3.4B worth of Ethereum (over 913,000 ETH) has been permanently lost due to user errors and contract vulnerabilities, according to Coinbase's Conor Grogan. Major incidents include the Web3 Foundation's loss of 306,000 ETH via a Parity wallet bug, QuadrigaCX's 60,000 ETH loss from a flawed contract, and Akutars burning 11,500 ETH during a failed NFT mint. Additionally, over 25,000 ETH were sent to burn addresses by mistake. Grogan notes this figure is conservative, excluding ETH lost to forgotten keys or dormant wallets, and highlights that over 5.3 million ETH (valued at $23.4B) has been destroyed through the EIP-1559 burn mechanism, removing over 5% of all ETH ever minted. [more]