TechRisk #137: Deskilling risk
Plus, attackers weaponizing red-teaming AI tools, Salesloft Drift widespread AI security incident on Salesforce, 20% chance of quantum breaking cryptocurrencies, and more!
Tech Risk Reading Picks
Deskilling risk: A new study in The Lancet Gastroenterology & Hepatology warns that prolonged reliance on AI tools in colonoscopy may degrade doctors’ diagnostic skills, a phenomenon called “deskilling.” Researchers in Poland found that after introducing Olympus’s CADe AI system, endoscopists’ adenoma detection rates (ADR) dropped significantly in non-AI-assisted procedures, suggesting reduced focus and responsibility when AI support was absent. While AI initially improves detection, ongoing use may lead doctors to over-rely on automation. This will weaken their independent judgment and attentiveness. The authors caution that further research is needed, but raise concerns that AI could unintentionally erode physician competence across medicine. [more]
Attackers weaponizing red-teaming AI tools: Threat actors are exploiting HexStrike AI, an open-source AI-driven offensive security tool originally designed to aid red teaming, bug bounty hunting, and CTF challenges, by repurposing it to weaponize newly disclosed vulnerabilities. The platform integrates with 150+ security tools and specialized AI agents for reconnaissance, vulnerability discovery, and exploit development, but reports from Check Point reveal it is being actively abused to exploit Citrix flaws, automate attack chains, and even resell vulnerable NetScaler instances on darknet forums. This misuse accelerates exploitation timelines, reduces human effort, and increases attack success rates, underscoring a paradigm shift where AI orchestration can rapidly scale real-world cyberattacks. Experts warn that such dual-use AI tools, including HexStrike AI and others like Velociraptor and PentestGPT, highlight both shrinking defense windows and new risks. [more]
Grok can amplify malicious links: Threat actors are exploiting a loophole in X’s AI assistant, Grok, to bypass the platform’s link-posting restrictions and amplify malicious ads. According to Guardio Labs researcher Nati Tal, scammers embed harmful URLs in the overlooked “From:” metadata field of video ads, then prompt Grok to reveal them in replies. This will effectively convert hidden links into trusted, clickable posts. As Grok is a system account, its responses boost the credibility, visibility, and reach of these links. Tal dubs this technique “Grokking,” noting it can generate millions of impressions. He suggests fixes like scanning all metadata fields and filtering Grok’s outputs, and has reported the issue to X. [more]
Working with AI: A new Arctic Wolf report shows that AI is rapidly shaping cybersecurity purchasing decisions, with 73% of organizations already adopting AI-driven tools and nearly all expecting it to influence buying this year. This accounts for an average of 39% of security tech purchases. While sectors like financial services lead adoption, most leaders want vendors to integrate AI into solutions to offset talent shortages. Key drivers include faster breach response, automation of tier 1 tasks, improved threat prediction, and reduced alert fatigue. Despite strong optimism (i.e. two-thirds expect positive impacts and 80% foresee better threat detection), security leaders stress that AI must complement, not replace, human expertise, with analysts shifting to higher-value tasks and organizations prioritizing staff upskilling. [more]
Free AI-powered brute-force testing tool: BruteForceAI is a free AI-powered penetration testing tool that automates realistic brute-force login testing for ethical security use. [more]
Salesloft Drift widespread AI security incident on Salesforce: Several major tech companies were hit by a data theft incident involving Salesloft Drift, an AI tool connected to Salesforce. Hackers stole customer information, including business contacts, support case data, and cloud access credentials. Cloudflare, Zscaler, and Palo Alto Networks confirmed exposure. Salesloft paused its Salesforce integration and took Drift offline. Investigations show hackers used stolen tokens to access data from 8–18 August. Cloudflare examined the compromised data and found 104 Cloudflare API tokens were accessed by the hackers, but it has not seen any suspicious activity related to these tokens. All of the tokens have since been rotated. Google reported similar token misuse and advised all Drift customers to treat connected credentials as compromised. The attack highlights risks from third-party integrations. [more]
Web3 Cryptospace Spotlight
20% chance of quantum breaking cryptocurrencies: Ethereum co-founder Vitalik Buterin has warned that there is a roughly 20% chance quantum computers could break modern cryptography by 2030, posing a serious threat to the security of cryptocurrencies. Unlike traditional computers, which process data in binary, quantum machines use qubits that can represent multiple states at once. It could allow them to crack encryption that would take today’s supercomputers thousands of years to solve. This possibility, first highlighted in the 1990s with Peter Shor’s quantum algorithm, fuels growing concern within the crypto community that the rise of quantum computing could undermine the integrity of digital assets and destabilize the broader crypto economy. [more]
$2.4M drained from DEX: DEX Bunni suffered a $2.4 million exploit after attackers manipulated flaws in its custom Liquidity Distribution Function. The attack drained USDC and USDT from Ethereum-based smart contracts, forcing the platform to pause all contracts and urge users to withdraw funds. Security experts revealed the attacker triggered faulty rebalancing logic by executing carefully sized trades, allowing repeated draining of assets. Bunni has offered a 10% bounty for the return of stolen funds. While built on Uniswap v4 and integrated with Euler Finance, the exploit did not affect Euler’s protocol. [more]