Slopportunity knocks: how AI impacts security practices for startups

Latacora has spent lots of time talking with startups about AI and how security practices need to evolve to keep up. It comes up in conversations with current and prospective clients, with people we meet at conferences, and occasionally, with random baristas who don’t even know we work in security. From what we can gather, AI is kind of a big deal right now.

This is probably not news to you. If you’re reading Latacora’s blog, you’re probably (a) interested in security, (b) interested in startups, and (c) have heard plenty of hot takes about how AI is going to hack the planet, or whatever. So here are ours:

We think about risks coming from AI and LLMs in a few ways:

• AI in your products: These are the risks of embedding LLM-based agents into your product. This is highly product-specific, so we’ll save it for another post.
• AI in your organization: How pretty much everyone from engineering, to sales, to finance and operations is using AI tools day-to-day.
• AI against your organization: How threat actors are leveraging AI to attack organizations faster, better, and cheaper than they were previously able to.

We’ll start with the last category: how threat actors are using AI offensively against organizations, and what you can do to defend against it, then move on to managing AI inside your organization. In a follow-up post, we’ll discuss a common specific case of AI inside your organization: managing and sandboxing AI coding agents like Claude Code. For simplicity’s sake, we’ll use “LLM” and “AI” pretty interchangeably throughout, but when we say “AI,” we mean the current crop of tools that use large language models under the hood to produce human-like text output from a non-deterministic neural network.

How threat actors are using AI #

As with many new technologies, threat actors are AI early adopters. You’ve likely heard of attackers using AI in a variety of ways. These are some of the techniques we’re seeing during incident response engagements:

• Open-source vulnerabilities & rapid weaponization. AI models are getting extremely good at identifying vulnerabilities in source code, accelerating a longstanding supply chain attack vector: finding and exploiting flaws in open-source software and dependencies. The biggest shift is speed. We used to see days between proof-of-concept exploits appearing on GitHub and active exploitation in the wild; now we often see mere hours (see Zero Day Clock).
• Social engineering. The days of “you can tell if a page is phishing if it’s poorly designed and has typos” are long gone. Attackers can trivially create pixel-perfect clones of real login pages and other credential-harvesting targets. And they can be backed by real functionality: attackers are deploying phishing sites that can phish MFA credentials as well by collecting TOTP codes, triggering SMS/email and collecting one-time codes, or triggering push notifications. Social engineering has evolved beyond credential phishing and into novel techniques such as ClickFix to exploit the ability to rapidly generate websites and content which looks urgent, authoritative and trustworthy.
• Credential-stealing malware. While these more sophisticated phishing pages can often still work against MFA-protected accounts, attackers are also increasingly using credential-stealing malware to bypass authentication flows entirely. We frequently see this delivered through CI/CD compromises or ClickFix attacks, either through compromised dependencies or by tricking users into running malicious commands. Once installed, the malware scoops up credentials from developer endpoints and CI/CD pipelines and hands them over to the attackers. This isn’t a new pattern, but AI is accelerating it in two ways. First, the malware itself is evolving much more rapidly. Second, these credentials are getting used much faster. In the past, stolen credentials were often sold to other threat actors, creating a delay between compromise and exploitation. Now, we are seeing near-instant use of stolen credentials to pivot into additional systems and compromise even more accounts.
• Lateral movement inside environments: when responding to incidents, we’re seeing actors actively using AI to explore within a compromised environment, establish persistence, and move between systems. It’s becoming more difficult to contain breaches, because AI agents often move significantly faster through a system than human defenders can keep up with.

What can you do about it? #

This might sound pretty dire. And in many ways, it is: both the number of attacks we’re seeing and the sophistication of those attacks are on the rise. The good news? We already have most of the tools and practices needed to adapt to these attacks. The only difference is things that were once “nice to have” or “relevant for larger companies” are now becoming table-stakes security controls. We like to say that AI is turning “severity: info” into “P0”—that is, very minor defense-in-depth issues and hygiene issues are becoming critical weaknesses. What you might have adopted as your company grew to 50 or 100 or even 200 staff are now things you probably need to adopt even as an early stage startup with a handful of employees.

What are those things?

• Passkeys or security keys. The latest crop of credential phishing websites have no problem harvesting app-based/TOTP codes, SMS/email codes, or triggering push notifications. The only phishing-resistant options are passkeys and hardware-backed security keys. Passkeys are great since they don’t require procuring additional hardware; however, synced keychains introduce additional attack surface (there is an opportunity for them to be compromised during the syncing process). Hardware security keys (like Yubikey) offer the strongest protection. But either way, they provide a measurable upgrade over any other form of MFA, since they bind credentials to a specific website and can’t be phished. While we can argue the merits of exportable vs. nonexportable keys and whether using passkeys for passwordless sign-in is safe enough, any of these options that works for your organization is better than a traditional password + TOTP/SMS/push notification flow.
• Locking down installable software. One key way to reduce the risk of malware is to restrict what users can install on their endpoints. This isn’t going to help for most developer tools - you can’t typically lock down npm install in the same way as installing desktop software and even if you could you probably wouldn’t want to hamper your engineers in that way - but a great place to start is browser extensions and OAuth apps. Browser extensions are low-friction to install and are often granted extremely broad permissions to read user data on web pages, meaning that every coupon-clipping browser extension your users install is a potential vector if the publisher gets compromised or bought. Similarly, OAuth apps often grant broad access to company data from a single user click. Neither one goes through the same review and code-signing process required by, say, macOS desktop software controlled with Gatekeeper. Start by restricting browser extensions and OAuth apps to an allow-list.
• Fast, consistent patching process. Build the capability to deploy library patches to your application as quickly as possible. Ideally, this should be fully automated: you merge a Dependabot PR, it immediately tests and deploys to production. Having automated testing and zero-downtime deploys is a security feature: the faster you can get from “vulnerability patched” to “live in production,” the better your chances of patching before an AI-backed attacker exploits the vulnerability. Separately, have a fast, consistent patching process for endpoint software: attackers target weaknesses in the software you have running on endpoints, too. This is especially true of web browsers! AI agents are extremely adept at finding vulnerabilities in browsers, but browser makers are rapidly patching these vulnerabilities.
• Supply chain management. The easy win here is package cool-downs. The majority of incidents we’ve seen that originated with supply chain compromises would have been mitigated with a 2 or 3 day package cooldown: malware tends to get pulled down pretty quickly, so you just need to make sure you’re not installing packages right after they get published. Make sure this applies to anything you’re installing globally on developer endpoints or CI/CD environments, such as CLI tools you’re running with npx or pipx.
• Keep live credentials off of developer endpoints. The most common way malware grabs credentials off of developer endpoints is using something like trufflehog to find everything sensitive that’s on-disk. However, some increasingly sophisticated malware is also looking at running process memory or other non-disk sources. To start with, get credentials out of .env files and other on-disk locations and use something like the 1Password CLI to load them at run-time directly from a password manager. Even better, avoid having any credentials of value on developer endpoints: have completely separate dev credentials for authenticating to third-party systems for testing, and use short-lived/ephemeral SSO sessions to log into AWS or other cloud environments.
• Implement least-privilege access. It’s typical at small companies to just give everyone “admin” access to everything. That massively increases the options attackers have for lateral movement - if they can compromise one account or endpoint, they get access to pretty much everything. Restrict people’s access to systems they don’t need, and limit the number of admins in each system. You can use Privileged Access Management (PAM) tools or processes to limit standing access to admin accounts, and have folks elevate their permissions only as needed. For example, you can have a separate Google Workspace account for administrative access that you don’t use day-to-day, and sign out of explicitly when you’re done using it. That way, if your endpoint gets compromised, the only session cookies you have are for your day-to-day account.
• Endpoint Detection and Response (EDR), ideally with Managed Detection and Response (MDR). Malware on developer endpoints is a “when,” not an “if,” especially if you’re writing software (or having agents on your computer write software!). Supply chain vulnerabilities and social-engineering-driven malware installs happen; the question is what you’ll be able to do to identify and contain it. You should issue company-owned computers (because EDR software is much too invasive to ask anyone to install on their personal computer), manage them with Mobile Device Management (MDM) software, and push out some sort of EDR software that identifies when malware is running. MDR is likely to be on your to-do list shortly thereafter, since you need to actually tune and investigate EDR alerts, and it’s unlikely that a startup has the capacity to do that in-house 24/7.

Managing AI agents in your organization #

Chances are, most (if not all) of your employees are using AI in some form or another. This might be AI coding agents, desktop apps like Claude Cowork, or AI agents baked into the SaaS products you’re using. Having a bunch of AI agents running around on your devices and cloud-based systems carries a number of risks with it:

• AI agents sometimes do weird, unpredictable stuff. Just like infinite monkeys and infinite keyboards will eventually produce Shakespeare, infinite AI agents and infinite tool calls will eventually drop your prod database.
• Prompt injection will happen, and so will agent misbehavior. A lot of ink has been spilled on “prompt injection” - the idea of malicious content from a webpage or other untrusted source injecting malicious instructions into LLM context to cause agents to misbehave. This is real! You should definitely worry about it. You should not, however, think of it as the only way agents end up doing terrible things, or trust anyone or any system that says it has “prompt injection protection” or somesuch, because prompt injection is just a particular flavor of general agent misbehavior. Even if a tool could somehow ensure that no malicious content makes its way into the LLM context window, that still doesn’t help with hallucinations, bad judgements, and all the other things an agent can get wrong. You need to both reduce the risk of malicious instructions ending up in your agents, and minimize the damage an agent can do if it goes off the rails for whatever reason.
• Broad access and cross-system leaks: Similar to how people tend to accumulate access and permissions over time at an organization, agents tend to steadily accumulate more and more access to company systems and data. This presents both a straightforward risk of agents misusing data or access to a system, and it also compounds as you give an agent access to more systems. Consider an agent that uses data from your Google Drive to answer questions in Slack: unless you rigorously validate it only has access to documents which should be accessible to all members of the Slack channel it’s in, it can be tricked into answering a question in Slack using data from Google Drive that a Slack user in that channel shouldn’t have access to.

What can you do about it? #

So what can you do to reduce the risk of using AI agents in your organization? Well, I have some good news for you: because you studiously read the first part of the blog post, paused, implemented all of the suggestions I went through above, and only then kept reading up to this point, you’re already in good shape! Wait no, that’s probably not right. Well, the good news at least is that all the things you can do to harden your organization against attackers who are using AI are also the most effective things you can do to minimize the risk of AI tools within your organization. You’ll notice this list is similar to the above, but with considerations for how it applies to AI agents.

• Passkeys or security keys. Storing your credentials in some sort of hardware-backed module (a security key like a Yubikey, the Secure Enclave on Apple devices, or Trusted Platform Module on Windows) means that AI agents can’t get at them. This is one place where pure-software solutions like 1Password passkeys are notably weaker: a malicious AI agent can theoretically grab the secret key material from an unlocked 1Password vault, but it can’t get at passkeys stored on a physical security key, or in iCloud keychain (which uses the Secure Enclave to protect keys).
• Locking down installable software. There are two parts of this, in the context of AI agents. First, restricting what software can be installed, and what OAuth apps or browser extensions can be installed, helps protect against agents installing or telling users to install this software. Second, it also extends to software that extends AI agents: MCP servers, skills, and other software in the broader AI ecosystem that users might want to install or connect to expand their agents’ capabilities. Having a clear picture of what’s connected to your agents, and a vetting process for approving new software reduces the risk of a malicious or poorly implemented skill or MCP server making its way into your environment.
• Fast, consistent patching process. Make sure you’re able to quickly roll out patches to AI software that’s in use in your organization. The major agent harnesses (Claude Code, etc) have significant safeguards built-in, and security teams that are updating those safeguards based on observed attacks and misbehavior. While it’s important to keep all software patched, it’s especially important for agent harnesses, because these products and the attacks targeting them move quite rapidly.
• Supply chain management. Package cooldowns are especially important when AI agents might be the ones installing new dependencies. Agents also love to run stuff like npx and uvx, so make sure cooldown settings are applied system-wide and not just to project-scoped installs (via user-scoped config like ~/.npmrc, ideally managed via MDM).
• Keep live credentials off of developer all endpoints. Agent harnesses often give the agent read-only access to the entire filesystem (yes, even in “sandbox mode”). This is a particular issue for developer endpoints, which usually need a bunch of credentials for running development servers, but with agents in the mix it’s often the case for a broader swath of endpoints as more non-developers end up using agents to write code. At a minimum, keep credentials in a centralized credential manager like 1Password. Even better, avoid giving out long-lived, live credentials to anyone: use separate development credentials that don’t have access to real data, and use short-lived SSO sessions for access to real systems.
• Implement least-privilege access. This goes for agents too, not just people! Make sure that each agent has access to the minimal set of services it needs, and minimal permissions within that system. Segment out access for separate agents: try to avoid having a single agent with access to a sprawling array of systems where it might leak data from one place to another. Instead, give each agent a separate set of minimal permissions.
• Endpoint Detection and Response (EDR), ideally with Managed Detection and Response (MDR). Just like this helps monitor and respond to malware, it also helps monitor and respond to agent misbehavior. Make sure EDR is running in all the places your AI agents are, so you can have visibility into what they’re doing and identify malicious actions. In the same way that EDR lets you hunt for signatures of malware, it can also let you hunt for AI agents that have followed some particular pattern of misbehavior or fallen victim to a particular prompt injection payload by observing the behavior of those agents. If something’s going around that convinces Claude Code to exfiltrate your .env file to some attacker-controlled server, EDR provides visibility into which developer endpoints have connected to that server.