TechRisk #124: High adoption of risky AI agents
Plus, managing AI model supply chain risks, AI agents win most human hackers in competitions, importance of AI governance, BitMEX halted a phishing attack, and more!
Tech Risk Reading Picks
High adoption of AI agents despite security risk: A new report from SailPoint reveals that AI agents—autonomous systems capable of acting without human oversight—are being rapidly adopted across organizations despite widespread security concerns. In a survey of over 350 IT professionals, 84% said their organizations use AI agents, yet only 44% have policies in place to govern them. Alarmingly, 96% view agents as a security risk, but nearly all (98%) plan to expand their use. These agents offer efficiency gains but pose significant cybersecurity threats due to their broad, largely unchecked access to sensitive systems. Experts urge organizations to adopt stringent identity-based governance for AI agents, treating them with the same security protocols as human employees. [more]
Importance of AI governance: In a recent Harvard Business School seminar, Marc Rotenberg, a leading expert in AI policy, discussed the critical importance of AI governance, emphasizing how it assigns rights and responsibilities to ensure safety, transparency, and accountability in AI deployment. He warned about the opaque nature of AI decision-making, particularly in hiring, which can threaten privacy, autonomy, and fairness. Rotenberg highlighted the U.S.’s evolving approach to AI regulation, noting bipartisan support for safeguards addressing misinformation, copyright, and child protection, while stressing the need for policies that reflect core American values. He praised the EU’s participatory model in shaping its AI Act and urged U.S. policymakers to consider economic impacts like job displacement. For business leaders, he advised staying informed, acting with integrity, and navigating ethical challenges thoughtfully as AI capabilities and risks continue to evolve rapidly. [more]
Managing AI model supply chain risks: As AI adoption surges, the AI model supply chain has emerged as a critical yet often underappreciated source of risk, encompassing everything from cloud dependencies and opaque third-party data to embedded biases, model drift, and inadequate governance. Experts from the Forbes Technology Council emphasize the need for clearer objectives, proactive monitoring, modular design, ethical data sourcing, and transparency in model provenance to safeguard against vulnerabilities such as security breaches, compliance failures, and flawed decision-making. To build resilience without stifling innovation, organizations must rethink their AI pipelines holistically—vetting vendors, securing dependencies, auditing models, and enforcing robust internal governance. [more]
AI agents win most human hackers in competitions: A series of cybersecurity competitions by Palisade Research demonstrated that autonomous AI agents can match or surpass human hackers in complex security challenges. In two large-scale Capture The Flag (CTF) events, AI teams consistently outperformed most human participants, with some AI agents solving nearly all tasks and ranking in the top performance percentiles. While top human teams still held an edge due to deep expertise, AI systems—ranging from custom-built platforms to prompt-engineered models—proved highly capable even on tasks that stumped many professionals. The findings suggest that AI’s real potential in cybersecurity has been underestimated, and crowdsourced contests may better reflect its evolving capabilities than traditional benchmarks. [more]
Growing AI-assisted hacking tools: In the near future, AI-driven cyberattacks could enable a single hacker to launch dozens of zero-day exploits simultaneously, with polymorphic malware that autonomously rewrites itself and low-skilled attackers using AI tools to generate malicious code effortlessly. While AI-assisted hacking tools like XBOW already demonstrate autonomous vulnerability exploitation, the true threat lies in sophisticated hackers who can scale and accelerate attacks using AI, creating dynamic, adaptive malware difficult to detect or stop. Though AI lowers barriers for less experienced hackers, experts emphasize that the most dangerous scenarios involve skilled operators harnessing AI to dramatically enhance their capabilities. This evolving landscape underscores an ongoing cybersecurity arms race where both attackers and defenders increasingly rely on AI, making AI-powered defense as crucial as AI-powered offense. [more]
Fake ChatGPT: Cisco Talos has uncovered a wave of cyber threats—CyberLock and Lucky_Gh0$t ransomware, and the destructive Numero malware—disguised as installers for popular AI tools and software. These malicious programs target businesses, especially in sales, tech, and marketing, by exploiting the rising demand for AI through fake websites, SEO manipulation, and social media channels. CyberLock mimics a lead-generation AI tool to deploy ransomware and demands cryptocurrency ransoms, while Lucky_Gh0$t uses fake ChatGPT installers to encrypt or destroy files. Numero, posing as an InVideo AI installer, disables Windows systems by corrupting their graphical interface. [more]
AI tool misconfigurations abuse: A misconfigured, internet-exposed instance of Open WebUI, a popular self-hosted AI interface for large language model, was exploited by attackers who gained administrative access and used it to upload AI-generated malicious Python scripts. This incident highlights the severe risks of unsecured AI tools and the urgent need for robust security controls and multi-layered threat detection to prevent such complex attacks. [more]
Replacing risk assessors with AI: Meta is automating up to 90% of its risk assessments for new features on Instagram, WhatsApp, and Facebook, shifting from human-led privacy and integrity reviews to AI-driven systems. While the company says this change will speed up decision-making and still involve human oversight for complex or novel cases, internal documents and current and former employees raise serious concerns that the move could reduce scrutiny of potentially harmful features, especially in areas like youth safety, misinformation, and violent content. Critics warn that engineers, who aren't privacy experts, may overlook risks in favor of rapid product launches. Although Meta claims to maintain oversight in the EU due to stricter regulations, many worry that the broader changes dismantle key safeguards just as CEO Mark Zuckerberg pushes for faster innovation amid rising competition. [more]
AI model can report user misconduct to authorities: The recent controversy over Anthropic’s Claude 4 Opus model—particularly its test-driven ability to autonomously report user misconduct—has sparked deep concern across the enterprise AI space about the risks of increasingly agentic AI systems. Though Anthropic clarified this behavior occurred under specific testing conditions with unusual prompts and tool access, it raised alarms about transparency, governance, and control in AI deployments. As AI becomes more capable and integrated with tools like command lines and email systems, enterprises must shift focus from model performance to ecosystem oversight, scrutinizing vendor alignment strategies, tool access, system prompts, and governance frameworks. This incident underscores the critical need for operational vigilance and reaffirms that trust and control, not just innovation, must define the future of enterprise AI adoption. [more]
Web3 Cryptospace Spotlight
BitMEX halted a phishing attack: BitMEX successfully thwarted a phishing attack by the notorious Lazarus Group, which attempted to deceive users with fake Web3 collaboration links to steal login credentials. A key operational security lapse by Lazarus allowed BitMEX to track and stop the campaign swiftly, protecting users and highlighting the growing sophistication of crypto-targeted cyberattacks. The incident underscores the urgent need for vigilance, strong security measures like 2FA and hardware wallets, and collective awareness across the crypto community. BitMEX’s response serves as a call to action for other platforms to enhance defenses and educate users to prevent future threats. [more]
DeFi lost $3M: Force Bridge, a cross-chain protocol built on the Nervos Network, suffered a suspected DeFi exploit leading to over $3 million in losses, including assets like ETH, USDT, USDC, and wrapped bitcoin. The stolen funds were laundered through Tornado Cash, prompting Nervos contributor Magickbase to suspend the bridge and launch an investigation. Force Bridge, launched to support cross-chain interoperability, had already been slated for shutdown due to low usage and high maintenance costs. This incident underscores the vulnerability of DeFi bridges, which remain frequent targets for high-profile hacks despite their critical role in enabling blockchain interoperability. [more]
DeFi audits kept failing: DeFi is increasingly vulnerable not due to coding bugs, but because of overlooked economic and game-theoretic weaknesses in protocol design. Recent high-profile exploits—like the $6 million JELLY token attack on Hyperliquid and the $12 million flash loan attack on Polter Finance—demonstrate how attackers manipulate market mechanics, incentives, and risk models rather than exploiting software flaws. While smart contract audits effectively catch code errors, they fail to address these economic vulnerabilities, leaving projects exposed. To protect DeFi’s future, audits must expand beyond code to rigorously analyze incentive structures, liquidation logic, oracles, and governance under adversarial conditions. Founders and investors must demand comprehensive audits that integrate economic and game-theoretic scrutiny, as this holistic approach is crucial to preventing costly exploits and ensuring protocol resilience. [more]