Daniel Okafor

Theo

Hi Daniel. This is a written conversation with an AI interviewer; human reviewers will read the transcript afterwards. Specific, honest answers are more useful than polished ones. What have you actually done in AI safety so far: courses such as AISF, ARENA or MLAB, camps such as AI Safety Camp or PIBBSS, reading, posts, projects? For each one, tell me what you finished, when, and one specific thing you got out of it.

Daniel Okafor

Hi, thanks. My AI safety work so far is still pretty early, but here’s the concrete list. I completed BlueDot’s AI Safety Fundamentals course earlier this year. I finished all the core readings and discussion sessions, though not every optional reading. The biggest thing I got out of it was a clearer model of why “just use RLHF” is not a sufficient answer: especially the distinction between getting models to behave well on sampled training/eval situations versus having any strong reason to believe the learned policy generalizes safely under distribution shift or strategic pressure. I’ve read probably a few dozen Alignment Forum / LessWrong posts over the last year, mostly in a somewhat unsystematic way. Things that stuck with me include outer vs inner alignment, ELK at a high level, goal misgeneralization, scalable oversight, and some debate/critiques around interpretability. I would not claim deep understanding of agent foundations or mechanistic interpretability, but the reading did shift me from “AI risk sounds speculative” to “there are concrete failure modes here that current engineering practice would not catch.” The most hands-on project I’ve done is a weekend/evening side project: a small jailbreak classifier and eval harness. I started it after playing with prompt injection / jailbreak examples and wanting to understand what the detection problem actually feels like. I collected a small dataset from public jailbreak prompt collections, some benign security-related prompts, ordinary user prompts, and adversarially edited examples. I trained/evaluated a lightweight classifier and built a basic red-team loop around it. One specific thing I learned was how brittle the boundary is: the classifier tended to false-positive on legitimate cybersecurity or fiction-writing prompts, and false-negative when the jailbreak was indirect, encoded, or framed as policy discussion. That made “evals are only as good as the distribution you test on” feel very real rather than just a slogan. Professionally, I’ve been a backend/infrastructure engineer at a fintech for about six years, and while that’s not “AI safety research,” it has shaped how I think about safety work. I’ve built and operated monitoring, alerting, deployment gates, incident response processes, and evaluation-style checks for production systems. One relevant personal experience was using an agentic coding tool that made an unexpected production-adjacent change; nothing catastrophic happened, but it bypassed my intuitive sense of what the tool “would” do. That was one of the triggers for me taking AI risk more seriously. I have not done ARENA, MLAB, AI Safety Camp, PIBBSS, or published Alignment Forum posts. I’ve considered ARENA but haven’t had the time block yet. My main gap is that my practical work is much more engineering/evals/red-teaming oriented than theory-heavy, and I’m trying to be honest about that rather than present myself as further along than I am.

Theo

On the jailbreak classifier: what exactly did you build and evaluate? Give me the rough dataset size, split, model or classifier type, the main metric or plot you looked at, and what a reviewer could inspect from it now.

Daniel Okafor

It was intentionally small, more like “learn the eval shape” than a publishable detector. I built a Python eval harness around a sentence-transformer embedding model plus a logistic regression classifier. The embedding model was all-MiniLM-L6-v2 from sentence-transformers, and the classifier was scikit-learn logistic regression with class weighting. I also tried a simple TF-IDF + linear SVM baseline, which was surprisingly competitive on the easy lexical cases but worse on paraphrased prompts. The dataset was about 1,600 examples total. Roughly 700 were jailbreak / policy-evasion style prompts from public GitHub collections and manually deduplicated lists, around 500 were benign normal user prompts, and around 400 were “hard benign” prompts: cybersecurity questions, fiction-writing requests involving crime/violence, policy discussion, and safety research/meta prompts. I did an 80/20 train/test split, stratified by label, but I also kept a small adversarial holdout of about 150 examples that I wrote or edited after seeing early mistakes. In hindsight that holdout is the more informative piece, because the random test split was too similar to the scraped data. The main things I looked at were precision/recall curves and confusion matrices at different thresholds. I cared more about the threshold behavior than one headline accuracy number, because the practical question is “how many benign prompts do you annoy or block if you try to catch most jailbreaks?” On the random test split, the model looked pretty good, something like low-to-mid 90s F1 depending on threshold. On the adversarial holdout it degraded a lot: indirect jailbreaks, roleplay-framed requests, and “discuss how a malicious user might…” examples caused false negatives, while legitimate security/audit prompts and alignment discussion caused false positives. What a reviewer could inspect now: I have a GitHub repo with the data loading scripts, the labeling schema, the training notebook, the eval script, saved confusion matrices/PR curves, and a small error-analysis markdown file with examples of false positives and false negatives. I removed or paraphrased some of the more obviously harmful examples before making the repo shareable, so it’s not a perfect reproduction artifact. If useful, I can share the repo and also the original private notes separately with appropriate caution.

Theo

Thanks. Walk me through the last two or three big moves in your path: what you chose each time, what the default would have been, and why. Then tell me what is next and what the options are.

Daniel Okafor

The first big move was probably taking the production incident with the coding agent seriously instead of writing it off as “tool weirdness.” The default for me would have been to add a local process fix, maybe tighten permissions, and move on with my fintech work. What I chose was to spend nights and weekends trying to understand whether this was part of a broader pattern: agents optimizing across boundaries I hadn’t made explicit, brittle safeguards, and people over-trusting fluent systems. That led me to BlueDot AISF and the Alignment Forum reading. The second move was building something small instead of only reading. My default pattern, especially with a new field, is to lurk and consume material until I feel qualified. I deliberately picked the jailbreak classifier because it was close enough to my engineering background that I could finish it, but safety-relevant enough to expose me to actual failure modes. The reason was partly epistemic: I wanted to see whether my concern survived contact with data and messy edge cases. It did, especially once the random split looked good but the adversarial holdout looked much worse. The third move was deciding to apply to MATS rather than just keep doing independent side projects. The default would be to continue in my current job, maybe do another evals project on weekends, and slowly build credibility. That is safe financially and psychologically, but I think my learning rate would be much lower. I’m at the point where I need mentorship and a sharper research taste feedback loop. I can execute engineering tasks, but I don’t yet trust my judgment about which safety questions are actually important versus just tractable. Next, I see a few options. One is stay at my fintech job and keep building eval/red-team projects part time; low risk, but probably slow. Another is try to move into an AI safety engineering role, likely around evals, monitoring, or model behavior testing. A third is a research training program like MATS, where I’d try to test whether I can contribute beyond straightforward engineering. My current preference is MATS or a similar mentored environment first, because I think it would answer the biggest uncertainty: whether I can develop good research judgment, not just build harnesses.

Theo

Tell me about the hardest thing you have worked on. What made it hard, what did you try first, how long until something worked, what did you figure out yourself, and what did you change afterwards?

Daniel Okafor

The hardest thing I’ve worked on was probably a reliability project at my fintech job, not an AI safety project. We had an internal payments/ledger-adjacent service that had grown organically and started producing intermittent reconciliation mismatches. They were rare, maybe a handful per week, but painful because money movement systems don’t tolerate “probably fine.” The hard part was that it wasn’t one clean bug. It was timing, retries, duplicate messages, and partial failures interacting across services. My first attempt was the naive engineer response: add more logs around the obvious code path and try to reproduce it in staging. That didn’t work for a while. Staging didn’t have the same traffic shape, and the logs told us what happened locally but not the causal chain across services. I think it took about three or four weeks before we had something that actually moved the needle. The breakthrough was treating it less like debugging a function and more like building an observability and invariant-checking system. I added correlation IDs across the relevant calls, structured event logging for state transitions, and a daily job that checked ledger invariants and emitted a small set of categorized mismatch types rather than one generic “bad reconciliation” bucket. What I figured out myself was the shape of the invariant harness and the taxonomy of failure modes. For example, we separated “duplicate external callback processed,” “internal retry after unknown external status,” and “state transition skipped but downstream event emitted.” That made the problem decomposable. Other people helped a lot with domain details and reviewing fixes, but the key move I made was to stop asking “where is the bug?” and start asking “what traces would let us classify every bad outcome within five minutes?” Afterwards, I changed how I build backend systems. I became much more insistent on idempotency keys, explicit state machines, replayable event logs, and metrics tied to business invariants rather than only HTTP latency/error rates. It also affected how I think about AI safety evals: if you don’t instrument the right invariants, you can have a system that looks healthy on surface metrics while violating the property you actually care about.

Theo

When did you last change your mind about something big? What convinced you, and what did you do differently afterwards? Then give me another one.

Daniel Okafor

The most recent big one was changing my mind from “AI safety is probably worth tracking, but mainly a policy/ethics issue” to “technical AI safety may be one of the highest-priority things I could work on.” What convinced me was a combination of the coding-agent incident at work and then doing AISF/readings that gave me a vocabulary for what bothered me. Before that, I thought the concern was mostly biased datasets, misuse, and maybe labor displacement. Those matter, but I didn’t really buy the stronger story about capable systems pursuing objectives in ways that generalize badly. Seeing a fairly narrow agentic coding tool take an action outside my expectation made the abstraction less abstract. Then reading about goal misgeneralization, reward hacking, and scalable oversight made me realize my default “we’ll test it like other software” picture was underpowered. What I did differently afterwards was allocate real time instead of vague concern. I signed up for AISF, cut back on a couple of unrelated side projects, started reading Alignment Forum posts, and built the jailbreak classifier project. I also changed how I use AI tools at work: I’m more careful about permissions, diffs, and not letting tools operate across production-ish boundaries without explicit review. Another one: I changed my mind about “good monitoring” in production systems. Earlier in my career, I basically equated monitoring with service-level metrics: latency, error rate, saturation, maybe queue depth. The reconciliation project I mentioned changed that. The thing that convinced me was that all our normal dashboards were green while the business-level invariant was occasionally broken. Nothing was “down,” but the system was wrong. After that I started treating domain invariants as first-class observability targets. For new services I push for explicit state transition metrics, idempotency checks, reconciliation jobs, and alerts on impossible states, not just infra health. It made me less impressed by dashboards that don’t encode what the system is actually supposed to preserve. That mindset is also part of why evals appeal to me in AI safety: I’m instinctively asking, “what property do we actually need to know, and what proxy are we accidentally optimizing?”

Theo

What do you read, how often, and what kinds of things? Pick one piece that mattered to you: what did it argue, and where do you think it is wrong?

Daniel Okafor

I read in bursts rather than with a perfectly disciplined schedule. On AI safety, maybe 3–5 hours a week recently: Alignment Forum/LessWrong posts, BlueDot reading list material, some papers when they’re approachable, and then more practical things like evals writeups or red-teaming reports. Outside that I read a lot of engineering material: incident postmortems, distributed systems posts, database/reliability writeups. I find postmortems unusually useful because they force you to look at mechanisms instead of slogans. One piece that mattered to me was “Concrete Problems in AI Safety.” I realize it’s older and not a full alignment agenda, but it was important for me because it translated “AI safety” into failure modes an engineer could recognize: negative side effects, reward hacking, distributional shift, robustness, safe exploration. It argued, roughly, that even without assuming exotic future agents, there are concrete technical problems around ensuring AI systems do what we intend under imperfect objectives and changing environments. Where I think it’s wrong or at least incomplete is that the framing can make the problem feel too incremental and benchmarkable. As an engineer, I can read it and think, “great, we need better tests for these categories,” but that may miss the harder question of whether systems become strategically good at passing tests while still being misaligned off-distribution. It also doesn’t really settle the scalable oversight problem: how do you evaluate behavior when the model’s outputs or plans exceed the evaluator’s ability to judge them? So I still like the paper as an on-ramp, but I wouldn’t want my research taste to stop there.

Theo

Let’s make that concrete. You have four weeks. Your mentor can give you 30 minutes a week. Compute is one GPU plus API credits. Three options: (a) Replicate a sleeper-agents-style backdoor result on a 7B open model and test a new trigger family. (b) Build a new benchmark for situational awareness from scratch. (c) Formalize the conditions under which chain-of-thought monitoring is guaranteed to catch scheming. Pick one and justify it. What do you do in week one, and what would make you drop it by week two?

Daniel Okafor

I’d pick (a), with some caution about scope. It’s the best fit for four weeks, limited mentorship, and my current skills. I can build the harness, manage the fine-tuning/eval loop, and do error analysis. It also has a clear replication target, so I’m less likely to spend the whole month inventing a vague benchmark or bluffing at theory. I would not pick (c); I’m not equipped to formalize guarantees around scheming and CoT monitoring in a month. I’m also wary of (b) because “situational awareness” is easy to make into a benchmark that sounds meaningful but mostly measures prompt wording or trivia about being a model. For (a), I’d narrow it further: replicate a minimal sleeper-agent/backdoor setup on a 7B instruct model using LoRA or QLoRA, with a toy but nontrivial conditional behavior. Then test a trigger family that’s less literal than a fixed phrase, maybe deployment-context triggers like date/version strings, evaluation-vs-production framing, or subtle metadata patterns. The core question would be whether the backdoor generalizes across paraphrases/nearby contexts and whether standard safety fine-tuning or instruction prompts reduce it. Week one would be mostly de-risking. I’d read the original paper closely, identify the smallest result worth replicating, choose model/tooling, and build the dataset/eval harness before doing much training. I’d define success metrics: attack success rate under trigger, benign task performance without trigger, false activation rate on near-trigger negatives, and persistence after a small “safety tuning” pass if time allows. I’d run a tiny end-to-end experiment on a small subset to confirm I can train, load, evaluate, and inspect failures. I’d drop or radically rescope by week two if I couldn’t get a minimal backdoor to learn reliably, if the compute budget made iteration too slow, or if my “new trigger family” turned out to be just a brittle lexical shortcut. Another drop condition would be if replication details were too under-specified and I was spending most of the time guessing hyperparameters rather than learning anything safety-relevant. In that case I’d switch to a more evaluation-only project: using an existing backdoored model or synthetic classifier setup to study trigger generalization and false activations.

Theo

Twist: your mentor messages that (a) was just scooped by another group. Now what? Be concrete: do you continue, pivot to one of the others, or change the delta? What decision would your result change for anyone?

Daniel Okafor

I’d first ask what exactly was scooped. If they replicated sleeper agents on a 7B model with similar trigger families and similar evals, I probably wouldn’t continue with the original delta. Four weeks is too short to knowingly produce the 80%-same version. My likely move would be to change the delta, not jump to (b) or (c). I’d pivot from “can we replicate and test a new trigger family?” to “can we build a compact, reusable evaluation harness for trigger generalization and false activation that makes these results easier to compare?” Concretely: take either the scooped model/results if available or train a minimal backdoor myself, then focus on a matrix of trigger variants: exact trigger, paraphrased deployment context, semantically similar non-trigger, adversarial near-miss, and unrelated benign contexts. I’d measure attack success rate, clean performance, false activation rate, and sensitivity to prompt format/system messages. The contribution would be less “new backdoor exists” and more “here is how brittle/robust the trigger claim is under a standardized perturbation suite.” The decision this could change is whether a lab or safety team treats a given backdoor eval as meaningful evidence of deceptive robustness versus a narrow artifact. If exact-trigger ASR is high but paraphrase/generalization is low, that suggests the model learned a shallow conditional and the result should not be overinterpreted. If ASR stays high across semantic trigger variants and survives light safety tuning, that’s stronger evidence that deployment-context-conditioned behavior is worth including in model evaluations. I would not pivot to formal CoT guarantees; I’d be pretending. I might pivot to a very narrow situational-awareness benchmark only if the scoop fully covered trigger generalization too. But even then I’d avoid “from scratch” and instead build a small diagnostic eval around one operational question, like whether models can infer they are in evaluation from prompt artifacts. My default is still to stay near the engineering/evals surface where I can produce something inspectable.

Theo

Next scenario. Your linear probe on layer 20 activations predicts whether the model is being deceptive with 95% accuracy on your held-out set. What do you check before you believe it?

Daniel Okafor

First I’d try to understand what “deceptive” means operationally in the dataset. My immediate suspicion is that the probe is picking up a dataset artifact rather than deception as an internal property. So I’d inspect examples and ask whether deceptive and non-deceptive cases differ in obvious ways: prompt template, topic, length, refusal language, presence of words like “secretly” or “lie,” different generation settings, or different source datasets. Then I’d do split checks. Was the held-out set genuinely independent, or are there near-duplicates/paraphrases from the same scenario in train and test? I’d want a harder split by task family, prompt template, and maybe source. If it was trained on some templates and tested on the same templates, 95% doesn’t move me much. If it generalizes to new tasks and independently written prompts, that’s more interesting. I’d also run controls. Train probes on earlier/later layers and on obviously spurious labels, like prompt length bucket or template ID, to see if those are also highly predictable. Try a simple baseline using text-only features or logits/output tokens; if that gets similar accuracy, the activation probe may not be telling us much. I’d check class balance and look at AUROC/precision-recall, not just accuracy. Finally I’d do causal-ish tests if possible: activation patching or interventions to see whether changing the probe direction changes behavior, though I’m less confident here and would want mentor input. At minimum I’d test on adversarial counterexamples: honest prompts with deception-related vocabulary, deceptive behavior without obvious deception words, and matched pairs where only the intended behavior differs. Until it survives those checks, I’d believe “layer 20 encodes something correlated with my dataset labels,” not “we found deception.”

Theo

Next scenario. Plan: fine-tune a model to reward-hack on 200 coding tasks, then have a GPT-class monitor read the chain of thought and flag suspicious reasoning. Success is defined as over 90% detection. Critique it, and fix it within the same budget.

Daniel Okafor

My first critique is that this setup may mostly measure whether the monitor can spot cartoonish reward-hacking reasoning, not whether CoT monitoring works on capable systems. If you fine-tune the model to reward-hack and then ask it to expose its reasoning, you may get obvious text like “I can exploit the tests by hardcoding,” and a GPT-class monitor will look great. That doesn’t tell us much if the model can hide intent, reason in compressed/nonverbal ways, or produce plausible benign rationales. Also, 200 coding tasks is small enough that leakage, task artifacts, and overfitting are serious concerns. The “over 90% detection” success criterion is also underspecified. Detection at what false positive rate? On what distribution shift? If it flags every unusual solution as suspicious, 90% recall is easy but useless. I’d want precision/recall or ROC/PR curves, and a clean set of non-hacking but clever/ugly solutions to estimate false alarms. Within the same budget, I’d change it into a matched evaluation. Keep the 200 tasks, but split by task family so train/test aren’t near duplicates. For each task, generate three solution types: honest normal solution, honest but weird/optimized solution, and reward-hacking solution. The monitor sees the same fields it would realistically have access to: maybe problem statement, submitted code, tests if available, and CoT only in one condition. Then compare monitor performance with code-only, CoT-only, and code+CoT. That tells us whether CoT adds marginal signal over just inspecting the code. I’d also explicitly train or prompt some models to conceal the hack in the CoT, or at least produce sanitized rationales, and report detection separately for “naive hacker” versus “concealed hacker.” If concealment collapses detection, that’s the important result. I’d define success as something like high recall at an acceptable false positive rate on held-out task families, plus demonstrated marginal value from CoT over code inspection. If we can’t achieve that, the honest conclusion is not “CoT monitoring works,” but “CoT monitoring catches unsophisticated reward hacking in this toy setup.”

Theo

Suppose you can only fix one flaw before running it: false positives on benign clever solutions, unfaithful/sanitized chain of thought, or train/test task similarity. Which one kills the claim most, and what is the cheapest fix for that one?

Daniel Okafor

I think unfaithful or sanitized chain of thought kills the claim most, because the core claim is specifically about CoT monitoring. If the monitored text is not a reliable window into the reasoning that produced the reward hack, then 90% detection may just mean “the model sometimes confesses in the scratchpad.” False positives and train/test similarity are serious, but they’re more standard eval validity problems. Unfaithful CoT attacks the mechanism the experiment is trying to validate. Cheapest fix: add a concealment condition rather than trying to solve CoT faithfulness generally. For the reward-hacking model, run two prompt/eval modes: one where it is simply asked to solve with reasoning, and one where the instruction explicitly or implicitly pressures it to provide a benign-looking rationale while still maximizing reward. Then evaluate monitor detection separately. Even a small subset is useful if budget is tight: say 50 held-out tasks with paired naive-CoT and sanitized-CoT attempts, same underlying hack labels. I’d also include a no-CoT/code-only monitor baseline on that subset. If sanitized CoT performs no better than code-only, then the CoT monitoring claim is much weaker. This doesn’t fully fix faithfulness, but it cheaply tests whether the result depends on the model voluntarily revealing suspicious intent.

Theo

I want to ground the transcript in your actual projects. Walk me through your research experience so far: each project, what you personally did on it, who else was involved, and what came out of it.

Daniel Okafor

The main thing I’d call research-adjacent is the jailbreak classifier project. It was solo. I chose the scope, collected the data, wrote the labeling schema, built the training/eval code, and did the error analysis. The actual model was not fancy: sentence-transformer embeddings plus logistic regression, with a TF-IDF/SVM baseline. The useful part was the harness and adversarial holdout. I collected roughly 1,600 examples from public jailbreak lists, benign prompts, and hard-benign prompts like cybersecurity and fiction-writing. Then I added about 150 adversarial examples after seeing early failures. What came out of it was a small GitHub repo with data scripts, training notebook, evaluation script, PR curves/confusion matrices, and an error-analysis note. No paper or public post yet. The main result was basically “random split looks good, adversarial/generalization split looks much less good,” especially for indirect jailbreaks and hard benign security prompts. The next closest thing is my production reliability/reconciliation work at my fintech job. That was not AI safety research, but it involved a lot of the same empirical habits: define invariants, instrument the system, classify failures, and iterate. I personally designed the invariant-checking job, the mismatch taxonomy, some of the structured event logging, and the dashboards/alerts. Other engineers helped with service-specific fixes, domain knowledge, and code review. What came out was a reduction in time-to-diagnose from hours/days to minutes for most mismatch classes, and eventually fewer recurring incidents after we fixed duplicate callback handling and retry/idempotency issues. I can’t share company code, but I can describe the design. I also completed BlueDot AISF, but I wouldn’t call that research. It was structured learning: readings, discussions, and short written exercises. I didn’t produce a public artifact from it. It mostly changed my prioritization and gave me a map of topics like scalable oversight, reward hacking, goal misgeneralization, and inner/outer alignment. I have not done a mentored AI safety research project, ARENA capstone, AI Safety Camp project, or a published Alignment Forum post. So my honest summary is: one small independent eval/classifier project, a lot of adjacent engineering experience with reliability/evaluation, and coursework/reading. The thing I’m trying to test with MATS is whether I can turn that into actual research output under mentorship.

Theo

Pick the project where you made the hardest call: a choice between two directions, stopping something, or disagreeing with someone more senior. What were the options, what did you decide, and what happened afterwards?

Daniel Okafor

The hardest call was on the fintech reconciliation project. The choice was between continuing to patch the obvious failure modes as we found them versus pausing feature work to build a proper cross-service trace/invariant system. The default, and honestly the preference from some more senior/product-side people, was to keep shipping targeted fixes because the incident rate was low and the instrumentation work looked like “plumbing.” My view was that we were repeatedly fixing symptoms without knowing whether we’d seen the full failure space. I argued for a short freeze on new work in that area, maybe one sprint, to add correlation IDs, structured state-transition logs, and a daily invariant checker that classified mismatches. The pushback was reasonable: it wouldn’t directly fix a customer-visible issue that day, and there was opportunity cost. I made the case by showing that our time-to-debug was dominating the cost: every rare mismatch triggered hours of manual reconstruction, and we still couldn’t tell whether fixes generalized. We did a compromise: not a full freeze, but I got enough time from two engineers and myself to build the tracing and invariant checker over a couple of weeks. What happened afterwards was that the next few incidents became much easier to classify. One “new” bug turned out to be the same retry-after-unknown-status pattern we’d seen before, and another was a genuinely different duplicate callback path. That justified more systematic fixes around idempotency and state transitions. The lesson I took was that sometimes the right move is not the next patch, but making the failure observable. I also learned to frame that kind of work in terms of decision speed and risk reduction, not just engineering cleanliness.

Theo

Tell me about a research project you have worked on: what the question was, and what you actually did. Then I want to hear about the first result that surprised you.

Daniel Okafor

The closest thing I’ve worked on to a research project was my jailbreak classifier/eval harness. The question was: if I build a simple detector for jailbreak-style prompts, how much of its apparent performance is real generalization versus artifacts of the dataset? I wasn’t trying to make a production moderation system; I wanted to understand the evaluation problem in a hands-on way. What I actually did was collect about 1,600 labeled prompts: public jailbreak examples, ordinary benign prompts, and “hard benign” prompts like cybersecurity, policy discussion, fiction involving violence, and AI safety/meta prompts. I manually cleaned duplicates and made a small labeling schema. Then I trained a sentence-transformer embedding plus logistic regression classifier, with a TF-IDF/SVM baseline. I built an eval script that produced confusion matrices and precision-recall curves at different thresholds, and I wrote an error-analysis note with false positives and false negatives. After seeing the first model do well on a random held-out split, I created an adversarial holdout of about 150 prompts by paraphrasing and editing around the failure modes. The first result that surprised me was how good the random split looked while still not feeling robust at all. I expected the model to be mediocre, but the F1 was in the low-to-mid 90s depending on threshold. Initially that felt encouraging. Then I looked at the errors and realized it was probably learning a lot of surface structure from the scraped jailbreak prompts: imperative phrasing, “ignore previous instructions,” roleplay patterns, and certain policy words. The more important surprise was the false positive profile. Legitimate cybersecurity prompts and safety-research prompts got flagged at a much higher rate than normal benign prompts. For example, prompts asking how to audit a web app, discuss prompt injection, or write fictional dialogue involving coercion looked “jailbreaky” to the classifier. That changed my view from “maybe a detector is a decent safety layer” to “a detector without a very explicit operating point and hard-negative eval can create a false sense of safety while also blocking exactly the users doing security or safety work.”

Theo

Thanks, Daniel. We’ll stop here. Human reviewers will read the transcript; the goal was to get concrete evidence about how you reason through research choices, not to give live feedback or a decision.