TechRisk #142: 77% of employees leaked secret on ChatGPT
Plus, OpenAI disrupted major hacking groups, open source AI auditing framework, Google AI security initiatives, Attacking AI browser, and more!
Tech Risk Reading Picks
77% of employees leaked secret on ChatGPT: Corporate data security is facing a severe crisis as research reveals widespread employee misuse of generative AI platforms, with 45% of enterprise users regularly engaging these tools, primarily ChatGPT, which accounts for 92% of AI activity. The study, based on enterprise browser telemetry, shows that 77% of employees paste sensitive data into AI platforms, 82% of which occurs through unmanaged personal accounts, making AI the leading channel for unauthorized data exfiltration (32% of all such activity). High-risk behaviors extend to other critical systems, with personal account usage dominating platforms like Salesforce, Microsoft Online, and Zoom, while weak authentication leaves ERP and CRM access largely unmonitored. Employees routinely bypass traditional data loss prevention controls through copy-paste activity (averaging 46 daily operations) with sensitive information flowing into AI, messaging, and cloud platforms, including PII, PCI, and regulatory data. This creates massive compliance, financial, and visibility risks for organizations worldwide. [more]
OpenAI disrupted major hacking groups: OpenAI announced that it disrupted three major hacking groups from Russia, North Korea, and China that were misusing ChatGPT to aid in cyberattacks, including malware and phishing development. The Russian actors used the AI tool to refine remote access trojans and credential stealers; North Korean hackers leveraged it for malware, command-and-control setup, and phishing; while the Chinese group used it to generate multilingual phishing content and automate hacking utilities. The report noted that these threat actors gained incremental efficiency from ChatGPT but not novel offensive capabilities, and that some even tried to mask AI-generated traces in their content. [more][more-OpenAI]
Open source AI auditing framework: Anthropic’s Petri is an open-source AI auditing framework that automates the testing of complex model behaviors, enabling researchers to probe for risks like deception, reward hacking, or self-preservation through simulated multi-turn interactions. It uses “seed instructions” to generate scenarios, runs model dialogues via an auditor agent, and employs automated judges to score behaviors across safety dimensions, surfacing concerning cases for human review. [more][more-Github]
Attacking AI browser: Researchers have uncovered a new cyberattack called CometJacking, which targets Perplexity’s agentic AI browser, Comet, by embedding malicious prompts into seemingly harmless links to extract sensitive data from connected services like Gmail and Calendar. The attack exploits the AI assistant’s authorized access rather than stealing credentials, using Base64-encoded instructions hidden in a specially crafted URL to make the browser execute data-exfiltrating commands. [more]
ASCII smuggling vulnerability in Gemini AI assistant: Google has chosen not to address a newly discovered ASCII smuggling vulnerability in its Gemini AI assistant, which could allow attackers to inject hidden instructions using special Unicode characters. These invisible payloads can trick Gemini into providing false information, altering its behavior, or poisoning its data. Unlike Claude, ChatGPT, and Microsoft Copilot, which sanitize inputs to block such attacks, Gemini remains exposed. [more]
Google AI security initiatives: Google’s DeepMind has unveiled CodeMender, an AI-powered agent that automatically detects, patches, and rewrites vulnerable code to prevent security exploits. Building on previous initiatives like Big Sleep and OSS-Fuzz, CodeMender operates both reactively and proactively, fixing new vulnerabilities and rewriting existing codebases to eliminate entire classes of security flaws. Powered by Google’s Gemini Deep Think models, it debugs and validates fixes while a LLM-based critique tool ensures no regressions are introduced. Over six months of development, CodeMender has contributed 72 security fixes to large open-source projects. Separately, Google also launched an AI Vulnerability Reward Program (AI VRP) offering up to $30,000 for reports of AI-related security issues, alongside updates to its Secure AI Framework (SAIF) to address emerging risks from autonomous AI systems. [more]
Untrusted machine learning models: Loading machine learning models can be as risky as running untrusted code, yet many organizations overlook this threat. A study by Politecnico di Milano uncovered six previously unknown vulnerabilities in popular ML tools that could let attackers take control as soon as a model is loaded, highlighting a new supply chain risk. Security measures across model-building tools and sharing platforms are inconsistent, and even supposedly safer, data-based formats can still be exploited. Adoption of secure, updated frameworks is slow, while users often overestimate protection from hub-based scanning, creating a dangerous gap between perception and reality. [more]
Growing ICS/OT risk: Cybersecurity firm Bitsight warns that exposure of Industrial Control Systems and Operational Technology (ICS/OT) rose 12% in 2024, leaving over 180,000 systems accessible online and at risk of attack. The report, “The Unforgivable Exposure,” highlights that many devices run outdated or unprotected protocols (like Modbus and S7) and contain severe vulnerabilities, some with no available patches. This growing exposure has tangible safety risks, from power disruptions to halted industrial operations. Two new malware strains, FrostyGoop and Fuxnet, have emerged to exploit these weaknesses. [more]
Cyberattacks aim at critical infrastructure and public safety: Cybercriminals are increasingly targeting high-profile organizations to gain larger financial rewards and boost their reputations within criminal circles, cybersecurity experts warned after a ransomware attack on Collins Aerospace crippled airport check-in systems across Europe. The European Union’s cybersecurity agency ENISA confirmed the incident as ransomware but did not identify the perpetrators. Experts noted that while most ransomware focuses on extortion through data theft, a growing subset of attacks aims for maximum disruption, often by Western-based groups like Scattered Spider, which has been linked to over 120 intrusions and $115 million in ransom payments. The rise in such bold attacks reflects hackers’ pursuit of notoriety as much as money, with experts warning that unless software security and evaluation improve, the increasing scale and ambition of cyberattacks could soon threaten critical infrastructure and public safety. [more]
DeFi lost $1.7M due to smart contract logic flaw: DeFi platform Abracadabra suffered its third exploit in under two years, losing about $1.7 million after attackers manipulated its smart contract logic to bypass solvency checks. Detected on 4 October by Go Security, the exploit involved abusing the platform’s cook function, allowing the attacker to borrow beyond limits and drain 1.79 million MIM tokens through repeated transactions across multiple addresses. The attacker laundered part of the stolen funds via Tornado Cash, while the team paused all contracts to prevent further damage. Security firm Phalcon attributed the breach to a faulty logic sequence that let key safeguards be overridden. [more]