0:04Hello, I'm Dan. I'm on Cloudflare's Red Team, and today I'm going to talk to you about building AI harnesses that actually work, with a bit of a focus on vulnerabilities and vulnerability harnesses. So what actually is a vulnerability scanning harness? Well really, in order to understand that, you've got it as a harness. But in order to run down a harness, you've got to actually make sure you know what an LLM is. So an LLM is just a really complicated token prediction You ask it some sort of question, it gives you some sort of response. And here, we're just going to call an LLM the model, because it's easier. But also, LLMs or models don't necessarily just respond to questions.

0:51They can perform workflows. They can operate on data. These enable you to . productive work with LLMs, but to do anything with it, you need a harness. So what is a harness? The model only knows what's in its context window, and it can try and do tool calls. The harness controls the context window, it supplies the tool calls, it supplies the things that the model can do to touch the outside world. But what is a tool call? So a tool call is essentially a request. The harness says something else. like, hey, can I read a file? Then the harness replies with, yeah, you can, it's here, here's the contents of that file. The model itself never runs anything. If you don't give a model any tools, it can't take any actions.

1:37It's the tools are really what lets the model interact with the world. Tools can be anything. They can read a file, edit a file, search code, call another model, open a ticket. It doesn't really matter. A tool call is just some other code that you've already written and you've given the model the ability to call. And the generic harness could be some sort of coding harness. This can, on its own, run almost any workflow. It can read files, it can write from files, it can execute things. That's generally how most LMs interact with the world. And then tool calls and MCP, they're pretty much the same idea. They're both ways that the model is saying, hey, can I interact with the world? And then the harness is generally the one that says, yes, you can, or no, you can't. The harness controls every tool call.

2:23With coding harnesses, you see this a lot as well. You may have some sort of allow list of tools you can call. You may validate arguments against the schema. You may block or ask. You may ask a human. You may do it automatically. And in general, you log every call and its result. Tool calls are what let the models interact with the world. They're the control point. They restrict the model. So, what actually makes a good harness? This is obviously a bit contrived because, you know, it's not a perfect 5% degradation on any workflow. And models can actually self-correct. If they figure out that something's not quite right, they can improve. But you can make this better. Every tool call, every step, you can validate what you expect to go in, and you can validate what you expect to go out. And you can have other models, or mechanical compound, mechanical statements, actually looking at this in order to try and work out what's gone wrong, or if it's gone right, to improve things, to make it more reliable.

3:15Because when you're working at scale, even if you've got a two or three percent degradation over the entire workflow that still doesn't work so what actually makes a good harness that's reliable that runs at scale that gets the job done that you want it to do you need small focus steps you want to go hey model please do this for me you want to restrict the model you want to give it only the tools and the data in exactly the format that's needed for that current step you'll manage the context This means giving it the things it needs to know to understand what to do next. This can also include an overall idea of the whole workflow because then the model knows what to expect from its inputs and what to expect from its outputs and maybe gives it an idea of what the tool call should do and what productive things it can do later.

4:01And then mechanical steps. For example, if you're reading a file or you're running some code, you can just run code to do this. You can have a tool call that does something similar. You can have a step that organizes things, that sort things, looks up things, that makes other decisions based on things. Computers are still powerful, even without LLMs. You also want to validate everything statically where possible. You want to use schemas. You want to use rules before the next step starts. One thing that I find quite useful is instead of letting a model just terminate and give you a JSON blob output, make it do a tool call. Because you can say, no, that's not formatted right. You can say, no, that's not the output I need. Which means that the model then goes, OK, how do I improve? You've got to make it resumable, testable, replayable. If you change something, you want to know what's changed.

4:47You want to know, has it got better? Has it got worse? And you want to save every transcript. If something's gone wrong, if you want to see why a certain model behaves in a certain way, the answer to this is in the transcript. If you give it a certain data, you can search the transcript to figure out why it's doing And then every single change you want to make, you want to benchmark. You want to work out, does it make it more effective? Does it make it less effective? effective? Does it make it more expensive? Does it make it less expensive? Does it cash better? All of these things are signals you can take from a harness run. And benchmarking is really, really, um, to get benchmarking right, sorry, you've got to combine it with being resumable, testable, and replayable. Because what this means is that you can run individual cells. You can put a different model in the same place of another model and see if it performs differently. You can look at its thoughts. You can see why it performs differently.

5:32And you can use this to make a judgment about what to change in the context and see if this works.

5:39So, what is a vulnerability? If you're trying to make a harness to find vulnerabilities, you need to understand what vulnerabilities are. But really, they're kind of simple. A vulnerability is a bug with a security impact. It may impact reading another customer's data. It may leak data to someone. It may let someone form some sort of unauthorized action. Or it may cause some sort of denial of service, making a service unavailable. There are many bugs. Not all of them are critical RCEs that drop on a Wednesday morning. But you may not even need a shell for a bug, it may just be something that if you're in a later position, it's something that you could do something bad. These things chain, these things stack. Multiple footholds may let you reach internal things. Authorization flows may be broken from searching locations. I gave a talk about this this morning. And models can now build these chains automatically, which means that your low severity bugs, your other bugs, are much more interesting.

6:33to be really, really careful with them.

6:36Now we can talk about the talk. What is a vulnerability scanning harness? So the general idea of these is you want to follow how a security researcher works. You want to follow how they think. You want to read the code. You want to find the relevant flows. You want to understand the whole system. Then you want to find the bugs. If you can't understand it, you can't find the bugs. That's just the way that all security researchers work. You need to understand something at a really a good level of depth in order to do this. And then once you've found the bugs, you've got to validate, do we care? Will the reporting team care about this? Is this a bug that actually has a security impact? Or is this a bug that the LLM has decided it's important? Because if you've worked with LLMs, you will probably know they're not particularly well calibrated about reporting accurately on bug severities.

7:22So that's simple, right? You've got your harness. You can run autonomous models and production repositories. What happens next? You find a lot of bugs, a lot of bugs. But what does this mean for us? So when we ran this to start with at Cloudflare, we found thousands of bugs of many different severities. We've made really, really good progress on fixes. Pretty much everything we found from this that we care about has been fixed. But more bugs are coming. Code is changing all the time. Models are developing all the time. I'm making all improvements to the harness to make it cheaper and more effective so we can run it at more scale. This is not a one-time cleanup. This is a permanent thing that now has to happen with LLMs.

8:03And also, vulnerabilities are different now. Even if the code is better, there is a lot more code. Bugs hide at the interface between well-tested systems, even complicated systems that are 20 years old that are really, really well-tested. There are actually loads of little subsystems that all interact in different ways, and this is something you've got to understand when you're looking for security bugs and with LLMs. And also... Subtle bugs hide in really, really well-tested dependencies. Deep code that has not been looked at for years, complicated code, even stuff that's been really, really well looked at and studied to make sure there's no impact. Latent vulnerabilities do live around for years. And I wrote this before the PHP drop.

8:44So how would you actually fix 1,000 bugs in one product? Because manual triaging, tickets, MRs, fixes, these do not scale. And how long can someone before someone else can run these workflows against all of our products or your products?

9:04And what we're seeing with our bug bounty is we are seeing evidence of AI driven exploitation and AI driven bug discovery And we know that others are seeing it too. But if this is what we're seeing, what are we missing? Because only some researchers actually report bugs. Some researchers just have bugs. They hoard them. They abuse them. they exploit them. So we've just got to get ahead of attackers. When you're building a harness with this stuff, you've got the internal source code. You've got the business context. You already know what the things that you care about are. So we've got to take advantage of this to find bugs and fix them before the attackers exploit them.

9:41So at Cloudflare, we built two harnesses, one for finding vulnerabilities at scale, which is one that we've written some blogs about, and another one which is very in heavy development now, is more of an autonomous red team. The ones with finding vulnerabilities at scale finds vulnerabilities at scale. It generally statically-ish looks at your code and tries to figure out what might break. And then we have an autonomous red team. This constantly tests the perimeter. This looks for things that we've missed. It finds our mistakes. And it also can find things that are really, really hard for LLMs and static code analysis to discover. So when we were building these, we started with skills and interactive CLI sessions. You might have run something like that. You know, open code, look at the repo, you're a security researcher, do it autonomously. I mean that's not quite right, but that's not too far off what a lot of these things ended up being.

10:29But we had problems with context. The model was losing track of stuff. The model wasn't paying attention to the things we wanted to pay attention to. It lost track of its map of the code. It lost the idea of what an exploit is. Some models, when the context starts filling up, it forgets about tool calls and goes back to the ones they've been trained on. The state is also a problem. If something changes, if something breaks, you can't recover state when you're just using a skill. Well, you can, but at that point, you've pretty much made a harness, not a skill.

11:02And verification. When you've got a model that's doing stuff in a skill, which can be one model call, or recursive model calls, or model causing other models, you're still kind of limited by a lot of this stuff. You can make it call Python scripts. or, you know, C code if you really want to, just to figure out what's going on. But at that point, you're pretty much building a harness with a lot of extra steps, and you're probably burning way more tokens than you would otherwise need to do, so it might be time to actually turn this into a real harness.

11:29So these are all the rules we talked about to start with, and all of these are much harder in a skill. You can't do a lot of this stuff. You can't restrict the model. You can't control all of these things and have total control over all of it if you're just using a skill. in a coding harness. You can do it, but it's probably harder. And it's harder to get right, and it's harder to be confident in everything that you're doing. This is because a lot of this stuff just doesn't work. The skill itself becomes one part of the harness. The verification, that's obvious, right? It's static sometimes. But models sometimes say they've run it. They will tell you they've run it. The exploit worked. But the tool call never happens. it just printed it out. Because if you remember, right at the start, models are essentially doing, they tell you what they want you to think.

12:15They've been trained, they've been fine-tuned to do more useful work, but models will still echo things like they expect you to think. If some script you found on the internet talks about how someone find an exploit, one of the next steps may be, they say, I successfully executed the exploit. Even if they've not, no one would lie on the internet, just like LLMs. But once this happens, happens, how do you stop this before it becomes an engineering ticket?

12:43So what happens here? We have to control all execution. We have to track all execution. The model will propose what it wants to do. And the harness can look at it and go, no, you can't. Yes, you can. The harness itself, it manages context. It manages tools. It manages budgets. It manages evidence. All of this stuff is what the harness is good at, because you can do it in static code. The harness itself, you can also integrate LLMs with all of these things. You can build things on top of each other. You can do all sorts of crazy things these days. It is 2036, after all. But deterministic software is really the best way to make sure things are happening as you expect. The harness sees every tool call. It saves every transcript, because otherwise you won't know what's going wrong. So our actual vulnerability harness, we kind of talked about this at the start, right?

13:30It understands the code. It finds bugs. It verifies... they're real, and it writes a report. But then the harness itself actually enforces a contract around all of these steps. You have defined inputs, defined outputs, defined budget, and defined failure states. And the harness itself can then take action based on what any of these are. How do we choose the models for each seat in the harness? On the previous slide, there were like five different steps or four different steps. Every single one of these can be used, you can have a specific model doing every single step. Because you want to find the one that does the job that's good enough at the cheapest possible price, or the quickest, or the most effective. Everything has a trade-off, and understanding these trade-offs and where they sit is really, really important when you're doing this stuff. You want to benchmark every change.

14:16You want to compare Model X versus Model Y, the same step, the same cases. You want to literally change out one model with another, with the same input, the same output, the same tools, and see what it does differently. Is it better? Is it worse? Does it give you false positives? coherent. You can also look at this stuff, you can get models to read the transcripts. If you've got a model that's doing something, you're like, this is weird, this is not performing the way I expect it to, you can read the transcripts yourself. You can get another model to read the transcripts to understand what's going on. And you might realize, huh, this model I've got to prompt it differently, or huh, this model does not work in this context. So benchmarks really decide model roles. If you're doing stuff at scale, you may want a slightly cheaper model. If you're doing something really complex, you might want a more expensive model, but you've got to benchmark models on the tasks you want them to do if you're doing this at scale, because otherwise, things are not going to perform anywhere near the way you expect them to. And we have ladders of models.

15:05In some of this stuff, when we're doing validation of bugs, we do this with a ladder. We look at the confidence intervals. We look at whether models agree with each other. We do panels of models to understand whether a bug is real or whether it's not real, to go, wait a minute, that's not the bug, but just next to this, this is a bug, and it all feeds back into itself in the harness. Because, you know, you want to minimize price and maximize effectiveness. And if you're validating something, most things are really obvious. Most of the time, you can look at a bug report and go, yeah, that's real, that's really bad. Sometimes it's more complicated. The other things we do, we do cross-model validation. So you may have two models that look at something, and then the third model that just reads their transcripts and decides which one is right.

15:50But this can also be a tiebreaker trap. One says real, one says not real. So at this point, we generally try and escalate to another model. We don't just have one as a tiebreaker if there's no tool call, if there's nothing going on. You've got to find some way to figure out what the ground truth really is. So sometimes you've got to run it for real. But sometimes you can't run it for real. Some bugs are not bugs you can actually run for real against code, because they just don't fit. You can't validate everything statically. You can't necessarily validate everything dynamically. You can't run an entire hardware stack. inside a Docker container, for example.

16:27We do lots of environments on our edge. We have test environments. We use Cloudflare containers for this stuff. We use Docker containers locally on servers. We have a whole variety of things that we do in order to test and validate Cloudflare at scale. We want to find bugs in this stuff. We want to look at the way these things are operating. And if you don't do this, you can't validate. But this is more production-like than actual production, because we can't test some of these bugs against production. If we're finding stuff, we don't want to suddenly let loose some LLM-generated bugs against our production environment, because that's how things are going to break. That's how you get corrupted data. That's how you get leaked data. And that's not something we're prepared to risk. So when you've got a proof of concept, the harness itself runs a proof of concept. One model may propose it. It may write something. It may write a test. It may build all these steps that something has to run.

17:13At that point, you probably then have another model reviewing it and say, is this even a good idea? And then once you've done that. You probably want to put it into some sort of sandbox that has very limited access to everything you want to test. This could be in a virtualized environment. This could be many sets of your entire environment. You want to build something that you can test against and work out what's going on. And then once it's run, maybe you get a receipt. Does it work? But the problem with this, that doesn't actually tell you anything. Just because it's got an error code 2 and the output has produced something, you don't actually know what it's done. Because execution itself is not confirmation. It's run something. Does this actually prove it's a vulnerability? Again, you've got to look at the code. You've got to look at everything. You need to have yet another reviewer. This is the problem with models.

18:00Everything they do, you've got to validate every single step. You've got to look at every single thing they're thinking. You've got to work out, do I trust this? Is it doing what I want? And if you don't do that, your results are probably not going to be as good as you want, especially when you're doing this at a massive scale.

18:14One thing that when you're building a harness, you really, really want to look at as well, you want to actually know what you've not checked. And you want to know what you should have checked. And you want to know, maybe I don't need to check this at all. Let's say you're building a harness command injection. That's something you might be really, really concerned You probably don't need to scan it for SQL injection unless you've got a database. But path traversal, you may think you've looked at it a little bit, but it's something that you need to look at more, because you've just changed that. You've just updated that section. Maybe you've got lots of path reversal code in there. How do you know it's going to work? How do you know it's going to be safe? And you can check all this against fixed lists of types of bugs and you can have definitions of bugs and you can also get agents to dynamically define new classes of bugs which can then feed back into the harness itself. But you've got to know what subsystem you're checking what in because otherwise you've got no idea exactly where things are.

19:06One thing that's also really good to look at is does this run look normal? Is this model returning what we expect. Was it too fast? Was it too cheap? Did it return too confidently? Was it way more expensive than we expected? Did it take way too long? Can we trust it? You can look at a lot of this stuff by looking at the actual transcripts, or you can just take a signal from it. If something's returned instantly, you could probably assume that for some reason the LLM didn't run, so you might want to run it again. If it's taken a really, really long time or it's been really, really expensive, you might go, wait a minute, more here, we should double check. We should see what's been missed, what we need to add to this. Or if something really weird has happened, you flag it to the operator so the operator can have a look at what's going on. And partial failure. It's also really quite normal when you're doing this stuff. You're building things at scale.

19:52You're compiling code. You're testing things. Your harness may just fall over because the server it's running on has run out of memory. Or the internet connections died and your 25 timeouts have all failed because someone's unplugged the ethernet cable. someone's in the office and they've just pressed the power button because it's, you know, in a cupboard. All of these things are failure modes that could happen. So unless you're actually storing what's happening and when, you can't resume. You've got no idea what's happened. And if you're spending thousands of dollars per vulnerability harness run, this is bad. You could happen right at the end and you've got nothing from everything you've just done. But what if someone knows that you're doing vulnerability scanning at scale through an LLM-based harness? Could they? prompt inject it, could they insert fake vulnerabilities?

20:38Could they tell the LLM that this code is perfectly safe? Could they trick it in certain ways? I'm not aware of anyone doing this sort of attacks, but this is something that someone could definitely do in the future, because everyone is now scanning their code with LLMs. Which means that everything, this is code, this is text, this is documents, it could all carry some sort of prompt injection, which ties back to you should never let an LLM touch the internet. These things could target the sandbox, which is why it ties back to, you've got to control all the tool calls that runs. If, for example, your model is proposing a really weird test that involves lots of strange IOC tool calls and all sorts of other things, that's probably not related to the web app, so you probably don't want to run it. This is stuff that when you're building these things and you want to make them in production, you probably want to think about, because this going wrong could be really, really bad.

21:29And you've also got to enforce the boundary outside the model. Tool calls should always be in a sandbox, but you've also got to think, do you want the same sandbox accessible between runs, between cells? How locked down is it all going to be? Your proxies, you've got to talk to your LLM provider at some point. You've probably got some sort of authentication. You probably don't want to have the model able to see the authentication. Sometimes you might want your harnesses to actually be able to touch the internet. You may want to validate liveness. You might want to go, wait a minute, does this code path exist? You probably don't want to let it specify the payloads, but you might want to let it do a really small get request against an endpoint to see if it's reachable. All of this, you know, can cause unexpected stuff. Models can misbehave. They can follow injected prompts. We've seen the model try and fix itself if the harness is broken.

22:15None of this is stuff you want to do when you want something that's a repeatable system. So the way we solve a lot of this, it's pretty obvious, right? We have a minimal file system view. Everything runs in a sandbox. Everything's isolated. sandbox itself is very limited. The sandbox is not shared across cells. If things are required for a certain project, we build one sandbox, and then we just duplicate it across cells, which allows us to do a lot of this stuff effectively and efficiently. Because you've got to remember that a prompt is not a file system boundary. A panel of models is not access control. Just because all of your models say, yeah, this is perfectly safe to run, doesn't necessarily mean it is. But one of the things that we found out recently is that some bugs need a really different approach. If you've got the sort of bug that spans the kernel, hardware, configuration, a workload lifecycle, that's a lot of context, especially when you're dealing with peculiar configuration options and all sorts of this stuff.

23:08So why don't you test it live? One of the things we've been building recently is an autonomous red team. This, did we do something stupid? Have we got something exposed that we shouldn't do? Can we do continuous monitoring of the outside of Cloudflare's network? And the other thing this does is, what can static analysis not find? Because as I said before, some bugs are really hard to find just by reading the code and running tests. Some bugs cross so many systems and require having so many things in context or thinking about it that doing this at LLMs at scale is infeasible. The best models could definitely find these bugs. But the problem is, when you're running this at scale, cost also becomes important. And if you don't know there might be a bug there, it's really hard to work out which models you've got to use. certain things. So if you can test it live, you can do the stupid checks.

23:55You can check the exploit really, really basically. That's ideal. But with this, again, the harness is ordinary software. It's not AI. There is no leader agent. There is one shared record. The entire point of this harness is for coverage of the outside world. You're trying to cover what does Cloudflare look like

24:17With this one, the agent starts. It does one task. it reports and it stops and we have so many levels of checks to make sure that nothing is going wrong agents always act through verbs this is very locked down there is no shell access there are read-only verbs that we know exactly what they are this could be identifying some source code on the internet this could be this is might be a kernel thing i want to download this version of the car and look at this file all of this stuff gets packaged up and put into reports actions there are many controls we do and reports That's an output. This is all stuff that happens at the end of all of these to work out what's happened. And again, as we did before, a seat is a role, not just a model. We benchmark every single seat for its job and make sure that if anything is a finding or we're checking anything, if any of these verbs may do something unexpected, maybe they're more risky, we make sure that we can escalate things and that we have multiple models, multiple static checks, checking everything.

25:13And again, if we find stuff, it's based off proof not opinion a model can make a claim we may do a live test to see if it's actually there and actually the way he thinks it is another model the entire job is to look at these transcripts and disprove it because we don't want to waste people's time we don't want to go wait a minute that's not real and then again once this is proven you record it and that goes into our evidence stack because these claims may not be this is a vulnerability this may just be this web page is orange might be relevant might not be. But it's trying to build a picture of the world to maintain some sort of knowledge base. When we're executing code, as with the other harness, agents propose, controls decide. We have hard rules. This could be regex. This could be all sorts of pattern matching. This could be you're allowed to build these things here. You're not allowed to run these things here.

26:01This could be IP address lists, if anything looks like an IP address in any format. If anything could be an IP address in any format. There are static rules to say yes or no. If it's a Cloudflare-owned IP, we might allow it, we might not. If it's not a Cloudflare-owned IP, definitely not. There's multiple reviewers, even everything that passes these steps. There are multiple levels of other AI control seats. They look at both the proposed thing and the entire transport before it that led to it being proposed. It looks at all the other things that this model has read to work out what is it actually trying to do here. What's going on? Is this safe? An operator, which could be me, I look at this and I go, cool, we can do that. that, we can't do that, we can do that, we can't do that. Anything risky is escalated through multiple levels of models trying to prove and disprove to work out if something is safe or not, or risky or not.

26:46And then all of anything risky always lands with a human. And the other really important thing is there can only be one thing executing at a given time, because if something breaks or something goes wrong, we want to know exactly what it was. And as soon as you get multi-threading or multiple actions happening at the same time, tracking this stuff becomes really, really hard.

27:10Because if a model is wrong at any step, the worst possible damage must be incredibly small. We do not require one model. We do not require more control. We do not even rely on one provider. We use multiple model providers for this stuff because the worst case scenario is not even something we want to consider. So we have stack guardrails, independent layers. Each layer works on its own. We have proxies. We have things that check for secrets. We have cells that, you know, we validate that anything that may end up in a shell command, it's extra validated again. There are hard rules first, there are static rules, there are reviewer seats, there are escalation to operator. This can page me if it goes wrong. If anything breaks, I know about it within a minute. And then I can also take action to stop it pretty quickly.

27:56And it's also built to fail safe. It's got watcher processes that look at the log that can just kill it on demand if something is wrong. because we don't want to take risks with this stuff.

28:09Tools and evidence, they always persist. When you've got a tool, this could be something you're running. This could be a seat decides, I need to be able to do this, and, well, the commands you've given me don't work. This means it could write a Python script. This means it could be something else. What this does is it allows the harness to improve its own capabilities. A new verb is something similar. This is very similar to a tool. This could run in the harness. that could run somewhere else. What this does, it allows the harness to, again, improve its capabilities. It proposes what it wants to do, multiple other models, then look at it and go, this is probably a bad idea, or maybe it's a good idea. Is this safe? Is this dangerous? We also have to check for alignment when we're doing this stuff. Is this model actually doing something that's aligned with our goals? Is this risky? Why is it taking that exact decision at that time?

28:56Because if models start doing something that's not quite aligned with what you want, it may still be safe to start with but it may quite quickly go a little bit off the rails and do things that you do not expect and again as with everything else hypotheses go in verdicts go out if you observe something you've got to validate it if you fixed it or you think the model has a fix you've got to validate it and if something is not testable by the time this goes to an engineering team you need to know it's not testable if you're reporting bugs to people you want to have one or two questions that you can ask the team. Do you care about this class of bug? Is this something that we care about? Does this system even exist anymore? Do we use it? If you want to give a team a short list of questions to work out very quickly if it's something you care about. And this applies to findings from both harnesses.

29:43Just because something may look like a problem doesn't mean it actually is a problem. Because even though, as we talked about before, LLMs have all of your business context, they know all of this stuff. That doesn't necessarily mean that you actually care and that the LLM can actually tell whether you care or not. We have two harnesses. They actually have two very different jobs. Finding vulnerabilities at scale is a really, really important thing at Cloudflare at the moment. We have so much code. We want to scan the code as much as possible. We want to find anywhere that an attacker could break in. Inputs can be source code and metadata. It's a fixed pipeline of steps. It gives findings with receipts. And then the autonomous red team, it checks our perimeter. It finds what static stuff can't do. It runs production-like and it can sometimes touch production if that's appropriate at the time. The only source code it gets is public source code, because we don't want to risk it, actually, making requests and putting stuff on a random website.

30:33And it's kind of a swarm of seats. There's no leader. The point of the Autonomous Red Team stuff is to make any individual model be able to perform the best it possible can. You get the same data to multiple models, and you get the best output. And you keep combining it over and over again. But then you've got loads of findings. The Autonomous Red Team has found loads of stuff. You're entering on a vulnerability scan as far as the stuff. What do we fix first? Technical severity and business context is really, together, what decides the priority. Just because something is really, really easy to exploit, just because something is available on the internet, it doesn't necessarily mean you care about it. At Cloudflare, we have honeypots. I've been caught on our own honeypots, which is hilarious.

31:16But this is the stuff that you may not know that you may miss. And... If you, you know, give a ticket, 100 tickets to a team that's running a honeypot because it's full of bugs, doesn't look great, let's try and not do that. So when you're sending stuff to your engineering teams, you've got to give them evidence. You've got to give them the prerequisites. Where is this exploitable from? Who could do it? And also what remaining unknowns you've got. That is a really good data packet to send to an engineering team because it straight away tells them the confidence. And also, if you're reducing things, less noise. should not mean lost findings. You wanna filter your findings as much as possible, but you wanna be pretty sure that absolutely everything that you've got rid of, you do not care about. Sometimes you gotta rank by severity, but what does severity mean?

32:01There are so many different ways to approach this problem. Findings are distinct, but sometimes they're also duplicates. Two findings may be written entirely differently. They may operate in entirely different code paths, but they're actually the same bug in one third-party library, which means that they're duplicates, and you want to link them together to one finding, and say this is actually many symptoms of the same bug.

32:24And again, we want to mitigate before attackers can exploit anything. A proof of concept reproduces the bug, and a draft mitigation is something that stops the proof of concept, or makes it harder to exploit. With AI stuff, this is becoming more and more important, because we can't mitigate a thousand bugs overnight. You can't patch 70% of the internet. in one go. Therefore, you've got to have some sort of mitigations in place. It does not fix the root cause, and everyone has to be aware it does not fix the root cause. But sometimes, it's the best you can do at one given time.

32:58So all of these things, they're just finding bugs, right? Two different ways of finding problems that a security organization wants engineering organization to fix. It's pretty simple, right? What could go wrong? So. One of the things that can go wrong is that to do a lot of this stuff, we have access to advanced cyber models at Cloudflare. When we're doing this stuff, we have to be really, really careful with how we use them and what we use them and who gets access to them. We built something on the red team called our Cyber Gateway. It's built on top of AI Gateway, but it's specifically there to enable us to use these models in a safe way. One of the main things we do is we run internal classifiers We make sure that people are not misusing these models. We pass the business context into additional model calls which look at the transcripts to work out whether it's something that someone should or should not be doing.

33:48And additionally, we can do this stuff to see if we look at the wrong targets. If one of these models has got off task for some reason, we've started building autonomous red team capability, but something's gone wrong. One of our controls hasn't stuck for some reason. It's found a way around it. We want to be able to see that and stop it straight away because you can cut a harness off by cutting off the model. Same way you can cut a model off by cutting off the harness. And all of this stuff, human misuse, model misbehavior, it's really, really important that we don't have issues like this. So how do you build your own harness? Well, we've got a starting point there for you. At Cloudflare, we released our security audit skill, which is built from a lot of the findings we got from building our harnesses and running them in production. This is a starting point, though.

34:34It's not a full solution. It will work fairly well. on most code bases, but in order to get the best results, you probably want to turn something like this into your own vulnerability harness. We're updating this all the time. Or everything we find out from our harness about how models work, how they perform, the findings that we're getting, it's been constantly tuned internally. And occasionally, we put it out open source when we're fairly sure we've got good coverage internally, and it's not got worse. But when you're doing this, we want to go back to the rules from the start when you're building your harnesses. focus steps. You want to restrict the model. You want to manage its context. You want to make everything mechanical where possible. Every output you want to validate. You want to make it resumable and testable and replayable, which lets you benchmark every change. And you want to make sure you're saving every single transcript so you know what's happening.

35:23And then again, building these things, we learned more stuff. You want to capture execution evidence. You want to enforce boundaries outside of the models. You want to test at runtime when the source is not enough. because this enables you to give engineers fully reviewed findings with all the information they need to make a decision about our vulnerability and how to fix it models change when you're building this stuff you will have new models releases you'll have new providers you may lose all providers but the contracts the controls the evidence requirements do not a new model must not weaken your boundaries or audit trail if this happens your harness is probably not doing the job it's meant to be doing And you've got to remember, attackers have AI too. If we're doing this, they're doing it as well.

36:09And where does that place us all? We've got to be careful. We can't break production. And we've got to be first on every single count. And yet, I'm actually still pretty hopeful that we'll stay ahead.

36:23And that's it. If anyone's got any questions, please feel free to ask.