jacobhaimes

Hunting for AI Hackers: LLM Agent Honeypot

I co-authored the original arXiv paper here with Dmitrii Volkov as part of work with Palisade Research. The internet today is saturated with automated bots actively scanning for security flaws in websites, servers, and networks. According to multiple security reports, nearly half of all internet traffic is generated by bots, and a significant amount of these are malicious in intent. While much of these attacks are relatively simple, the rise of AI capabilities and agent frameworks has opened the door to more sophisticated and adaptive hacking agents based on Large Language Models (LLMs), which can dynamically adapt to different scenarios. Over the past months, we set up and deployed specialized "bait" servers to detect LLM-based hacking agents in the wild). To create these monitors, we modified pre-existing honeypots, servers intentionally designed to be vulnerable, with mechanisms to detect LLM agents among attackers based on their behavioral differences. Our current results indicate that LLM hacking agents exist but are in the very early stages of technology adoption for mass hacking. This post shares our methodology and findings about the current state of AI hacking agents in the real-world. The Project A honeypot is a decoy system or server purposely left vulnerable in order to attract attackers. Cybersecurity researchers commonly use honeypots to study the methods, tools, and behavior of real-world hackers. By monitoring everything that happens inside these environments, researchers learn how attackers discover, hack, and escalate on compromised systems. In our project, we deployed a network of honeypots that look like standard, weakly protected servers (e.g. with weak credentials) to attract cyber attacks. Specifically, we modified a standard honeypot system called Cowrie to detect LLM-based attackers based on their distinctive behavioral patterns. Additionally, we made our servers discoverable through traditional hacker-oriented search engines Shodan

35Feb 12, 2025

jacobhaimes

Message

106

Why Most Efforts Towards “Democratic AI” Fall Short

Setting the Stage Companies and venture capital firms can’t get enough of so called artificial intelligence (AI), with private investments in generative AI increasing by nine times from 2022 to 2023,[1] and tech giants Alphabet (Google), Amazon, Meta, and Microsoft projected to invest more than 1 trillion USD into AI...

Sep 29, 20252

Double Podcast Drop on AI Safety

TL;DR Two days ago, I released two podcast episode: * "One Big Bad Bill" on muckrAIkers ➡️ Igor Krawczuk and I discuss the substantial bill that Trump is attempting to push through the legislature, and I get a little animated... (IMO it's justified) Spotify, Apple Podcasts, YouTube, and landing page...

Jun 25, 20255

Hunting for AI Hackers: LLM Agent Honeypot

Feb 12, 202535

Understanding AI World Models w/ Chris Canal

On our latest episode of muckrAIkers we are joined by our first ever guest, Chris Canal! You can find the show on Spotify, Apple Podcasts, YouTube, or wherever else you listen. Chris is the co-founder of EquiStamp, a third-party evaluator providing objective and accurate evaluations of frontier LLMs and the...

Jan 27, 20254

Let’s Talk About Emergence

Emergence has found its way into machine learning vocabulary, but current use has resulted in a circular definition and has further confused an already complex domain. Clarote & AI4Media / Better Images of AI / Power/Profit / CC-BY 4.0 The field of machine learning has existed for many decades, but...

Jun 7, 20244

"Open Source AI" is a lie, but it doesn't have to be

NOTE: This post was updated to include two additional models which meet the criteria for being considered Open Source AI. As advanced machine learning systems become increasingly widespread, the question of how to make them safe is also gaining attention. Within this debate, the term “open source” is frequently brought...

Apr 30, 202419

Podcast interview series featuring Dr. Peter Park

Check out the Into AI Safety podcast on Spotify, Apple Podcasts, Amazon Music, YouTube Podcasts, and many other podcast listening platforms! As always, the best things come in 3s: dimensions, musketeers, pyramids, and... 3 installments of my interview with Dr. Peter Park, an AI Existential Safety Post-doctoral Fellow working with...

Mar 26, 20243

Load More (7/15)

LESSWRONG
LW

LESSWRONG
LW

jacobhaimes

jacobhaimes

jacobhaimes

Hunting for AI Hackers: LLM Agent Honeypot

"Open Source AI" is a lie, but it doesn't have to be

Documenting Journey Into AI Safety

Interview: Applications w/ Alice Rigg

jacobhaimes

Why Most Efforts Towards “Democratic AI” Fall Short

Double Podcast Drop on AI Safety

Hunting for AI Hackers: LLM Agent Honeypot

Understanding AI World Models w/ Chris Canal

Let’s Talk About Emergence

"Open Source AI" is a lie, but it doesn't have to be

Podcast interview series featuring Dr. Peter Park

Hunting for AI Hackers: LLM Agent Honeypot

"Open Source AI" is a lie, but it doesn't have to be

Documenting Journey Into AI Safety

Interview: Applications w/ Alice Rigg

Why Most Efforts Towards “Democratic AI” Fall Short

Double Podcast Drop on AI Safety

Hunting for AI Hackers: LLM Agent Honeypot

Understanding AI World Models w/ Chris Canal

Let’s Talk About Emergence

"Open Source AI" is a lie, but it doesn't have to be

Podcast interview series featuring Dr. Peter Park