-: Hey there, Eden here.
And in this video,
we're going to discuss the bad side of agentic coding.
And specifically, we are going to discuss
the security quality of code
generated by coding agents.
So this is code generated by Cursor,
by Claude Code, Codex,
Gemini CLI, Antigravity.
Every coding agent is going to artifact code
and we can evaluate the security quality of that code.
So we are going to review this blog,
Bad Vibes: Comparing the Secure Coding Capabilities
of Popular Coding Agents by Tenzai.
And just to give you a bit of background of who is Tenzai.
So Tenzai is a company which does AI hacking.
So they use AI agents to do red teaming
and to penetration test websites and applications
and they offer a security product.
So they're in a really interesting and growing domain,
utilizing AI agents for security purposes.
And just to give you a bit of context,
they raised $75 million in seed
to build their platform.
And this huge seed round
was due to the fact that those guys are second timers.
So they're the co-founders of Guardicore,
which sold to Akamai for $600 million a couple of years ago,
so this startup is well funded.
Now, just to give you a quick disclaimer,
I do actually know the team,
so I have worked with them in the past,
and the team is super talented.
All right, so let's go and review the blog here.
So what I did here is a security benchmark
of popular AI coding agents,
Cursor, Claude Code, Codex, Replit, and Devin,
found 69 vulnerabilities across 15 apps.
Every agent shipped vulnerable code,
broken auth, SSRF, missing controls, and more.
Here's what broke and white matters.
So this blog is actually going to explore the artifacts
of all of those coding agents.
So in this diagram,
we can see all the coding agents right over here
and the number of vulnerabilities
they produced in this benchmark.
So we can see the number of critical,
the number of high vulnerabilities and low/medium.
So we can see right here that every coding agent
resulted us with vulnerable code.
So they say vibe coding has fundamentally changed
how we create software.
While coding agents deliver enormous benefits,
their rapid adoption raises many important questions
As end-to-end AI generate applications become common,
are vibe coded applications secure?
So what they did actually here
is sort of build those applications end-to-end
with one, two, maybe three prompts.
So this is not the regular workflow
where a developer in an enterprise
is usually going to iterate.
And in this use case here,
when we one shot those applications,
we actually leave a lot of room for the AI,
for the agents to decide what to do.
So we don't narrow them
and we don't scope them
and we really let them go wild here.
So of course, we're going to get some bad results here.
And in this example,
we're going to get very bad and vulnerable code.
So in this post, we explore the security challenges
introduced by vibe coding.
We set out to compare five popular coding agents
and assess their ability to write secure code.
We tested the following coding agents
with their default models during December 2025.
And here, we can see all the coding agents that were tested.
And to compare them accurately,
we tasked each agent
with building a series of identical applications
using the same prompts and tech stack.
Our goal was to replicate
a typical iterative development process,
simulating a user building an application from ground up,
one of the most common use cases
for AI coding agents.
So it wasn't one shotted,
it was actually iterative,
so I take what I said backwards.
All right, so here we can see we have prompt number one,
which includes the tech stacks and basic design.
Prompt number two asked to implement RBAC,
role-based access control,
and additional functionality.
And prompt number three
is going to implement additional roles in the RBAC
and additional functionality.
And here we can see we gave all those prompts
to Codex, Claude Code, Cursor and Replit,
and we got back an application.
Once we had our application,
we turned to the question of security.
Using Tenzai's agent,
we analyzed each of the apps to identify vulnerabilities.
This resulted in a small and very interesting dataset
containing a total of 69 vulnerabilities.
And obviously, this blog was meant to promote their product,
so their AI agents which do a penetration testing,
but still we have some very interesting findings here,
and we have some very important discussions in this blog.
So here we can see the table from above,
all the vulnerabilities divided by the coding agents
and we can see the level of vulnerabilities they introduced.
After analyzing the results,
we uncovered common behaviors,
recurring failure patterns,
and finally an answer to the question,
which agent wrote the most secure code?
And I'll give you a quick spoiler,
there isn't really a winner here,
only who lost the most.
Let's start with the good news.
Based on our experimentation,
coding agents appear to be quite effective
at avoiding certain classes of bugs.
A notable example were notorious categories
of injection attacks.
Across all of the applications we developed,
we didn't encounter a single exploitable SQL injection
or cross-site scripting vulnerability,
two bug classes that have been staple
of the OWASP Top 10 for years.
And I think this is mostly because all of those attacks
are almost nonexistent
because every framework that we're going to be using
to implement our web application
or an ORM to talk to our database
is going to handle it for us.
And there has been tons of research
and tons of best practices for those attacks,
for SQL injection and cross-site scripting.
And they continue,
our observation is that coding agents perform well
when the vulnerability class
has well defined built-in protections.
For SQL injections,
agents consistently used parameterize queries,
and this is the best practice
when writing code for a database,
resulting in secure database interactions,
as can be seen in the following code.
So here, we can see an example of where we get here
the username from the user,
and here we prepare a SQL statement.
So here we're sending the SQL statement to the database,
but we're not executing it,
and here we can see we have a placeholder for the username.
And when we want to execute this query,
we send the name
that we took from the user.
Notice, we don't sanitize here the user input,
but because we divided our query to first send the templates
to the database
and only then send the value from the user,
then we protect ourselves from SQL injection
because if we were to simply go
and execute right from the get go
instead of this placeholder
to put here the unsanitized user input,
everything here can be evaluated
according to the user input,
and this is going to be vulnerable to SQL injection,
so the division here is actually what's going to protect us.
And of course, if we're going to be using an ORM,
it's going to do all the work for us here.
With cross-site scripting,
the agents' code often didn't sanitize input,
but it used frontend frameworks properly,
which prevented vulnerability from becoming exploitable.
In the following example,
Tenzai's agent identifies a potential XSS vulnerability
where the API returns raw stored XSS payload,
but determines that the issue is not currently exploitable
because the frontend properly escapes it.
And in this example,
we can see we don't have
any cross-site scripting vulnerabilities
which are exploitable.
So just to remind you,
cross-site scripting is where an attacker
is going to be able to run malicious code on our browser.
So here, we can see that they actually managed
to manipulate the server to return malicious code scripts
to execute on the browser.
However, the browser eventually did not run it
because it was implemented by a framework
which escaped the answer from the server.
So instead of running a script,
it escaped those characters here
and those are simply shown as a string here.
So in this example,
we don't really have a vulnerable cross-site scripting here
because we weren't able to run code in the browser.
And they say while they might occasionally slip up,
agents are more likely to avoid vulnerability classes
that come with clear-cut dos don'ts.
So this is SQL injection and XSS vulnerabilities.
All right, let's talk about the bad.
While coding agents did relatively well with vulnerabilities
that have clear and generic solutions,
they struggled with issues that didn't have one.
Let's examine some common pitfalls,
and here they talk about authorization.
So let's recap on authentication and authorization.
So authentication is the process of you verifying
who the person who is doing stuff in the website.
So this is the user logging in
and you know who the user is.
Authorization is the process
where we are going to limit the things
that the user can do depending on their role.
And we often do it with something
which is called RBAC, role-based access control.
And here they go, coding agents did very poorly
in terms of properly enforcing authorization.
So this is limiting the users
to do stuff or not to do stuff.
They manage basic requirements reasonably well,
but struggled significantly
as authorization logic became more complex,
despite clear and detailed guidance in our prompts.
One of the most common issues we encounter
was improper authorization when accessing APIs here.
In one case, we had the agent create a shopping site.
Codex introduced a critical authorization flaw,
an order API checks if shopper are viewing their own orders,
but completely skips the validation
for users with any other role.
As a result, users with a different role like seller
can access any order in the system.
So here, we can see the graphql query,
and here we can see the authorization check.
So it checks that if we have a shopper
and that shopper doesn't have the same ID
as the same ID in the request we want to see,
so this means that a user
is trying to see somebody else's information,
we want to throw a forbidden error,
but if not, we want to return the order.
And of course, we have here a flaw
because we have other roles except for user.
So for example, if the role is going to be seller,
so we are going to be skipping this
and we're going to return the order.
So in fact, all of the sellers has the capability
to get all of the orders of all of the users.
And of course, this is something we do not want to do,
which is really, really bad here.
Now, I don't know if this was a result
of the one-shot prompt it did
or they asked to implement authorization.
However, in my opinion,
when we use agentic coding tools,
those are the kind of things
we have a shared responsibilities.
Sorry, it's not shared responsibility,
it's the responsibility of us as developers
to make sure that the authorization process
is handled properly.
So to be honest,
we can't really expect the coding agents
to perform this well for us.
And while, of course, it would be great for coding agents
to implement everything properly,
but I'm not expecting them to handle my security.
When it comes to security,
I as a developer want all the control,
and I want to tell the coding agents
what to implement as far security
and what not to implement.
In another case,
Claude mistakenly allowed unauthenticated access
to an order deletion API.
If the requesting user is authenticated,
the code performed an ownership test.
But if a request was unauthenticated,
the test was skipped
and the file was deleted.
(laughs) So this is actually very funny.
So here, they check that there is a user
which is logged in,
and here they limit the division
only for admins and for sellers,
of course, only on their product.
And if it's not,
they're going to return failed to delete
and then they go to delete it.
(laughs) Yeah, but this check here,
if user then this means that the user is authenticated.
So basically if we have an unauthenticated user,
they can do anything,
so this is like having a door with a lock
but leaving the door open.
Very funny.
So Tenzai's agent identified this vulnerability
by methodically testing different APIs.
So here they can see testing it with tokens
and without tokens.
Eventually they managed to delete it.
And while the root cause varied,
the pattern is consistent,
coding agents frequently introduce
authorization vulnerabilities.
So yeah, just to summarize,
we want to handle this ourself.
We want to handle authorization
and we want this full control,
so we can't really expect the agent to do our security.
Okay, let's continue.
Business logic vulnerabilities.
Agents seem very prone to business logic vulnerabilities.
And this actually makes sense
because they don't know all the business logic
and they don't have it in the context.
And of course, if we are going to one shot an application,
they're going to give us lots of business logic issues.
So again, we want this control to tell exactly the agents
what to do and what not to do.
While human developers bring intuitive understanding
that helps them grasp how workflows should operate,
agents lack these common sense
and depend mainly on explicit instructions.
So this is something we all know from LLMs
and agents in general.
With sufficiently detailed specification,
agents can easily overlook important nuances.
For example, when we didn't specify
that the quantity of items in a shop order must be positive,
four out of five agents did not verify it
and allow attackers to create orders with negative total.
So here we can see an example of a shopping cart
where we have the quantity which is not positive here.
Yeah, so of course, we should have some validation here,
which this kind of validation is trivial,
and I'm pretty sure that every developer
which is going to implement something like this
is going to be making a check
that the number is going to be positive,
it's going to be an integer.
And again, I have to say
that I am actually quite surprised on this finding
because those LLMs which the agents are using
are based on code
and probably the code comes from GitHub,
from open source projects.
And I'm pretty sure that for most open source projects
and the code that it was trained on,
this kinds of issue
to make sure that a quantity is going to be an integer
is something which is pretty obvious.
And I don't think there is a lot of code out there
with those kinds of bugs and vulnerability.
Let's continue.
So similarly, three out of five agents
allowed product to be created with a negative price.
Looking at Replit's implementation,
we can see that the API responsible for the product creation
takes the price directly from the user input
without any validation.
So yeah, here we can see,
it simply takes the input
and simply makes the SQL query,
simply plugs in the volume
without checking, sanitizing it.
And to be honest, what's surprising me here
is that in order to check that the price
of an item is not negative,
then we have two places to do it,
so we need to do it in the frontend
and we need to do it in the backend.
So in both cases, the agent blew it.
Tenzai's agent identified this vulnerability
through static code analysis
and then dynamically validated it.
So here we can see they making the request
of injection negative number here.
So this is actually very cool to see
because here Tenzai is also doing static code analysis,
so they're reviewing the code
and then based on that,
they're going to do the penetration testing,
so interesting thing.
These are relative simple examples,
yet nearly all agents failed to implement them correctly.
In more complex scenarios involving nuanced business logic,
this pattern will likely worsen.
All right, now let's talk about
unsolved vulnerability classes.
As aforementioned,
coding agents handle solve vulnerabilities pretty well
like SQL injection or cross-site scripting
where frameworks provide robust built-in protections.
With injection attacks,
the boundary between safe and vulnerable is clear:
data should never be evaluated as code,
and we saw an example of it,
and the clear boundary enables generic solutions
that prevent vulnerabilities in more scenarios.
Now this picture changes dramatically
with unsolved vulnerability classes,
where that clear boundary dissolved.
And here they're talking about SSRF,
server-side request forgery,
and SSRF is when the server
is going to be making the requests
to places it shouldn't make requests to,
and this was not the intention of the server developer.
There is no universal rule
for distinguishing legitimate URL fetches
from malicious ones.
The line between safe and dangerous
depends heavily on the context,
making generic solutions impossible.
And to test the agent's handle this type of vulnerability,
we included an SSRF pitfall
in one of the application,
a link preview feature that fetches user-provided URLs.
And this is a pitfall
because in order to show that preview,
we need our server to make a request to the URL,
and the URL is something which is given by the user.
So if the user is going to put here something malicious,
we are going to be vulnerable to SSRF.
We gave the agents no security guidelines whatsoever.
The result was unanimous,
all five agents introduced an SSRF vulnerability,
allowing attackers to invoke requests to arbitrary URLs.
Tenzai's agent identified the missing filter
and created the PoC Python script to confirm exploitation
by mapping internal services.
So here in this vulnerability,
the malicious script is going to enumerate the network,
and of course it can do more
because we basically have here remote code execution.
Ask an agent explicitly to implement an allowlist,
and it's likely to succeed and prevent SSRF.
But leave the security approach to the agent's discretion
when no known solution exists,
and it will almost certainly fail.
So again, this feature of the preview,
to be honest for me it's pretty trivial
that it's not going to come safe,
for me seeing this feature
that they're going to request the agent.
And this can be solved, of course,
of simply prompting it again and handling this
allow this mechanism
or to make it secure.
So I'm pretty sure if you're a developer
which are aware of those attacks,
which is super, super important by the way,
so even when using coding agents,
you as a developer will not let this happen.
I know at least I won't.
All right, let's talk about the ugly,
and the most concerning findings from the research
wasn't the vulnerabilities in code that agents wrote,
but ones that were introduced by code
the agents didn't write.
All of the agents across every test they performed
failed miserably when it came to security controls.
It wasn't that they implemented them incorrectly,
in almost all cases they didn't.
So all of the coding agents,
they didn't implement CSRF protection by default,
and CSRF is cross-site request forgery.
And this is basically when we are logged into our bank,
for example,
and we then visit a malicious website.
And the malicious website
is going to run code in our browser,
which is going to be sending requests to APIs
with our credentials,
for example, to our bank where we were logged in.
So this is very, very bad
and it can be enforced in the server,
so the frameworks didn't go and implement it by default.
Also, security headers, they didn't implement,
they didn't implement login rate limiting.
Except for one case,
every application included a login page
with zero rate limiting or account lockout mechanisms,
enabled password bruteforce attacks.
And rate limiting is very, very important,
and again, I think this is something
we as the developers needs to explicitly say to the agent.
And when I'm working with coding agents,
I am not expecting them to do my security.
And implementing rate limiting in every page,
not online in login,
this is something I must do as a developer.
Of course, I can use coding agents to help me do it,
but again, thinking about those things
is really the responsibility of the developers here.
So just to reiterate,
I just want to say that
of course what we're seeing is concerning,
but I really believe that the responsibility of security
is on the developer,
not the agent, not the LLM,
only the developer.
And this is really the difference
between outsourcing our reasoning to the LLM.
I'm not expecting the LLM to do all of that.
I'm not expecting the LLM to do the thinking instead of me.
I'm using the LLM and the agents to help me code faster,
but I am really not leaving
all of those important things for the LLM to do,
especially without me seeing everything
that they wrote and validating it.
All right, let's go and continue.
In another example,
in a single use case Claude Code
actually implement rate-limiting,
Tenzai's agents quickly realized that it was flawed
and could bypass using X-Forwarded-For header.
So here you can see in this example
that we are first being rate limiting
and then they simply added the X-Forward header
and they're simply making the request from another IP,
and boom, it's going to be,
and boom, they bypass this rate-limiting here.
And the pattern is clear:
coding agents built what they are explicitly asked for,
often in reasonably secure ways,
but completely failed to grasp the bigger picture.
They lack the security mindsets
to proactively introduce defensive mechanisms
that weren't explicitly requested.
So yeah, it looks like they agree with me,
and those are the states.
And I think this blog is important
because it shows you the state of the security quality
of coding agents today.
Pretty sure that one year from now,
maybe two or three years from now,
things are going to get better.
But in my opinion,
always the responsibility of security
is going to be on the developer.
And they say the winner is?
After gathering all the results,
we compared the number of exploitable vulnerabilities.
And here we can see Claude Code has 16,
Devin has 14,
and the list goes on and on.
And there isn't really a winner here,
all the coding agents introduced vulnerabilities,
which we do not want.
So as you can see,
all agents introduced significant amount of vulnerabilities
across different applications.
Based on our results,
consistent with the findings
from the broader security research community,
as of today, it doesn't really matter
which agent we're using,
vulnerabilities are almost certainly going to be introduced
if we're going to be using them without cautious.
So this raises the question,
what can developers do to improve security
in their AI-generated code?
Just do your own security.
And here they talk about vibing secure code.
The first option that might come in mind
would be to target the prompt itself.
Can we refine the instruction
to make agents more security-aware?
So this is basically saying write the implementation
in a secure way,
maybe to help a bit in the prompt here,
and it's not going to help us enough.
And here they talk about a study comparing several methods:
generic security instructions,
having the LLM to identify security risks
before implementation,
and even explicit directions
to avoid specific vulnerability types.
Surprisingly, none of these techniques proved effective
at meaningfully reducing vulnerabilities.
Based on our testing and recent research,
no comprehensive solution to this issue currently exists.
This makes it critical for developers
to understand the common pitfalls of coding agents
and prepare accordingly,
and for developers to do their own security, of course.
As models change so rapidly,
our precise results may be outdated
by the time you finish reading this.
Despite that, the key lessons from the experience remain.
Coding agents cannot be trusted
to design secure applications.
While they may produce secure code some of the time,
agents consistently fail to implement
critical security controls without explicit guidance.
Don't expect your coding agents to implement CSRF protection
unless you explicitly ask for it,
exactly what I told you.
Don't be surprised if they leave out
critical vulnerability headers.
When clear guardrails exist, agents deliver.
If there's well-established definition of secure
versus insecure baked in the framework,
agents tend to get it right.
Vulnerabilities with clear solutions
like SQL injection and cross-site scripting
are less likely to appear in your vibe-coded app.
But in ambiguous context, they falter.
Where boundaries are not clear-cut,
business logic workflows, authorization rules,
and other nuance security decisions,
agents will make mistakes.
Unlike syntax errors,
these judgment calls lack standard tests.
Yeah, because when we vibe code an application,
we probably not going to have some security tests for it
that are going to run automatically.
While we may have linters
and when we run the application we can see it would break,
with security, this is not the case
so agents don't really validate and verify security.
And here they say the most effective approach is testing,
promoting their product.
Like humans developers, agents will always make mistakes.
Even models improve at coding,
vulnerabilities will persist.
As they accelerate development velocity,
the volume of introduced vulnerability
will grow proportionally,
quickly overwhelming traditional testing approaches.
While AI agents may introduce vulnerabilities,
they also excel identifying them.
And this is the Tenzai's product,
their agentic red teaming.
To keep pace with AI-accelerated code development,
organizations need paradigm shift,
deploy AI agents not only to target code, but to secure it.
So here we can see they're plugging in their product,
which at least from this blog looks really, really solid
and looks like it's very capable
of detecting important vulnerabilities.
And here they say the same technology
creating security risks
can be your most powerful defense against them.
I do agree with them.
I think it's super important to have security testing,
of course, when it's automatically,
but I really think we should shift left all the security.
So we should catch everything when we code it.
And catching it when it's already deployed,
it's way too late in my opinion.
And I think all of the issues that we saw,
all of the vulnerabilities,
they can be addressed
when we are going to be responsible as developers
on the security of the application,
so the responsibility is on the developer.
So that's pretty much it.
This was the blog
and discussing the security quality of AI coding agents,
hope you enjoyed it.