Rogue Agent Data

A couple of data sources on rogue agents have been making the rounds, which I find fascinating. So over the weekend I created a centralized backup of the data:

https://github.com/alexander-hanel/rogue-agents-data

I also created the following page to explore the data and reports:

https://alexander-hanel.github.io/rogue-agents-data/

And, yes, the game is playable. The site was created using two DGX Sparks running DeepSeek V.4.1, having multiple agents review all the logs, summarize the results and then having OpenAI’s Daybreak Blue (cyber capable model) validate the findings. I have been manually reviewing and clarifying the findings to better understand how rouge agents act.

StressingLLMs Token Results

Hello, here are the results of the token evaluation side of my StressingLLMs project; it is titled The cost of unfamiliar text. The results consisted of evaluating how large language models handle different type of character encodings. My early assumption for token inflation was that repeated characters would stress the tokenizer but this was incorrect. What stresses the tokenizer is rare Unicode characters. Here is the TL;DR

Rare high-plane Unicode can multiply an LLM request's token count and cost, creating a context and billing amplification risk for systems that process untrusted text.

 A fair amount of the text from the link was generated by AI but also edited by myself. The page is more of a way for me to explore the data and learn the underlying concepts. I hope it does the same for others. Cheers. 

StressingLLMs OpenCode Benchmark Results

Here is a link to the StressingLLMs OpenCode benchmark results. The process consisted of replicating my StressingLLMs tests using models available through OpenCode. I evaluated 11 models, most of which were free: Muse Spark 1.3, Muse Spark 1.2, Big Pickle, Nemotron 3 Ultra, Ling 3.0 Flash, MiMo v2.5, and Nemotron 3.5 Lightning. I also tested four paid models: GPT 6 Astra, Claude Fable 5.1, GLM, and Kimi K3. Altogether, I spent about $100. I don’t understand how people afford to benchmark models regularly. Claude and GPT 6 could chew through $20 in less than an hour.

The results are fun to explore. Most of the free models performed on par with the local models. The free model that surprised me most was Muse Spark 1.3. It was able to write a decryptor for an executable with 57 rounds, which is basically a cryptographic hashing workload, but failed at 58 rounds. GLM 5.2 made it to 67 rounds before failing. GPT 6 Astra and Claude Fable 5.1 both completed 68 rounds. Shortly after reaching those numbers, I stopped testing because my curiosity wasn’t worth the cost.

Based on the runs I completed, I estimated the cost, token usage, and time required for a single run. These figures are available in the “Estimated Single-Attempt Resources” table. According to the estimate, a single 10,000-round run using Claude Fable 5.1 would cost $239.14, take 4.7 hours, and use 118.9 million tokens. There’s no way to know whether this estimate is accurate without actually running the test, but it’s still fun to explore.

That highlights an important point about the page: I created it as a way to explore the data, and its results should not be treated as absolute.

One section I recommend exploring is the “Roadmap” table. Clicking the checkboxes displays the corresponding results and submission. It’s fascinating to see how different models approached the same problem.

Stressing LLMs Local Model Results

For anyone interested in seeing a final version of the local model results for Stressing LLMs, I created a results page hosted on GitHub pages. 

https://alexander-hanel.github.io/StressingLLMs/

Cheers. 

Alex

Exploring APTNotes with LLMs

I’d like to share a few projects I’ve been working on.

The first project is APTNotes-md, which is APTNotes converted to Markdown format. If you’ve never heard of APTNotes, it’s a GitHub repository of APT-related threat intelligence reports in PDF format, organized by year. I created APTNotes-md for anyone who wants to explore the data using large language models. APTNotes-md contains additional reports that I added because the original project was last updated in 2024.

The second project is feed-report-monitor, a pipeline for converting RSS feeds into PDF and Markdown files. It relies on RSS, Python, Marker, PyTorch, Ollama, and local models. This pipeline was also used to convert the APTNotes PDFs into Markdown. Converting 869 PDFs took about five days on a DGX Spark. The process could have been significantly faster, but I chose to use a local gemma:26b model to verify that the Markdown conversion was formatted correctly.

The third project is report-eval. This project has been both the most fascinating and the most frustrating, so I guess it balances itself out.The project started as an attempt to build a SQLite database populated with metadata and attributes extracted from the reports in APTNotes-md. The goal was to allow an agent to answer questions such as, “Summarize all reports related to Operation Aurora,” returning both the results and their supporting sources using xSQL. Populating the database sounded simple: create a set of questions, have an agent build an evaluation harness, then iterate over every report and ask every question. It turned out to be anything but simple.

The report-eval project was restarted two or three times. To give you an idea of how bad it got, one Saturday night I became so frustrated that I switched to GPT-5.6 Sol Ultra, typed “fix this,” and went to bed.

Pro tip: don’t do this.

The next morning my weekly usage limit was at 0%, with six days remaining until it reset. I managed to burn through an entire week’s worth of tokens while I was asleep. I ended up using one of my four resets just to get the harness into a working state. In retrospect, I should have reviewed the plan in more details before repeatedly saying yes. After realizing it would take another four days to process all of the reports locally using gemma:26, I decided to use OpenAI to perform the report classification instead. I quickly burned through the rest of my resets over the next day or two and temporarily signed up for the Pro plan to finish processing the reports.

If you’re interested in learning from my mistakes, check out the after-action report. If you’d like to see some of the reasoning failures and pitfalls I encountered along the way, take a look at fallacies.md. A couple of the points could be a full blogpost. The results of the current eval can be found here. 

Below are some results of the eval using Codex. 

Note: Start of AI Generated content

Main findings

gemma4:26b completed the corpus and produced relatively few final errors:

• 22 timeouts
• 17 unsupported status values
• 2 malformed evidence errors

gemma4:31b-it-qat was stopped after reaching approximately 4.48% of the corpus.

• All 416 failures were caused by empty answers.
• The model commonly returned status: "not_stated" with answer: "".
• This appears to be a response-format contract problem rather than necessarily an analytical failure.

hermes3:8b was stopped after approximately 15.57% of the corpus.

• 913 failures were empty answers.
• 928 used unsupported status labels such as stated, confirmed, or attributed.
• Only six failures came from timeouts or malformed evidence.
• About 99.7% of its recorded failures were therefore output-schema or validation failures.

openai-terra-high has no recorded mechanical errors, but this does not prove every classification is semantically correct. 

• The files contain no explicit correctness score or gold-label field.

Data-quality issue

• gemma4:26b contained 901 duplicated report/question pairs, adding 918 surplus rows. These appear to be retries appended to existing files.
• Using the raw rows gives gemma4:26b 423 errors, but many older failures were later corrected. Selecting the newest generated_at result reduces its current failure count to 41. Results should therefore be deduplicated before analysis.

Most problematic question categories

Within the partial Hermes run, the highest observed failure rates included:

• attacker_ai_use: 129/129 failed
• staging: 113/129 failed
• credential_access: 111/129 failed
• attribution_reported_confidence: 105/130 failed
• attribution_claim_source: 104/130 failed
• privilege_escalation: 98/129 failed
• exfiltration: 97/129 failed

These concentrations mostly reflect the same response-contract problems, especially blank not_stated answers and unsupported status terminology.

Overall conclusion

The dominant failures were not clearly incorrect APT classifications. They were failures to follow the required JSON answer contract. gemma4:26b successfully completed the evaluation after retries, while the two stopped models repeatedly produced responses rejected by the validator. Semantic correctness should be evaluated separately by comparing normalized labels against a designated reference set.

Note: End of AI Generated Content

Stressing LLMs - Local Model Complexity Attacks Progress

Hello, this is an ongoing personal learning series on Large Language Models (LLMs) and automated reverse engineering. In a previous blog post, I described a type of complexity attack against LLMs. I am using "attack" in the practical reverse-engineering sense: intentionally increasing the amount of interdependent code and state the model has to reason about. My hypothesis is that increasing the number of interdependent round functions increases the chance that a model will make a small but fatal translation error, even when it identifies the correct high-level algorithm. Here is an excerpt from that blog post that describes the methodology I'm using.

The complexity attack increases computational complexity by generating binaries with a large number of interdependent functions. Instead of hiding the logic, the goal is to make the amount of state and code too large for practical static reasoning. The executable contains a toy XOR cipher with a keystream derived from a set of N functions, where N is the number of generated rounds. A Python script generates C source code with an embedded encrypted string and decryption loop. GCC is then used to compile the C source into an executable. At runtime, the decrypted string is printed to the console. To make this concrete, we can walk through generating the code and compiling it.

What is useful about this approach is that I can measure the complexity of the generated binaries and then test them against an LLM. At first glance, this sounds easy, but there are a lot of nuances to running models locally. For example, one model started to fail after I updated Ollama. In this blog post, I'm going to describe the results, themes of failures and lessons learned. Before diving into those, I think it's worth mentioning the hardware and tooling.

I'm running all of the local models on a DGX Spark and using OpenAI's Codex as an agent. On the Spark, I use Python and Bash for all of the scripting and tool execution, clearbluejar's pyghidra-mcp within Docker to interact with Ghidra, and Ollama to interact with models. I'm using the following models for evaluation:

• gemma4:26b
• qwen3.6:35b
• qwen3-coder:30b
• mistral-small3.2:24b
• command-r:35b
• cogito:32b
• hermes3:8b

I attempted to download and use a number of other models, but they did not work because they were not accessible via MCP, timed out, or had other similar issues. Before I dig too deep into the evaluation process and data, I wanted to share the results of the most recent evaluation. In these runs, gemma4:31b is clearly the best local baseline. gemma4:26b was historically successful, but it became unstable after updating Ollama. qwen3.6:35b was able to solve fixtures (e.g. binary test case) 0001 and 0002 after updating the prompt.   Below is a table of the results from 102 runs. Please note that the results improved over time with updates to the prompt, so this should be read as a practical lab notebook rather than a perfectly controlled benchmark.

model	rows	verified	completed false	timeouts	other inconclusive	successful fixtures
gemma4:31b	15	11	2	2	0	0001, 0002, 0003, 0004 across later runs
gemma4:26b	23	10	8	3	2	mostly 0001, plus 0002 and 0003
qwen3.6:35b	11	2	6	1	2	0001, 0002
qwen3-coder:30b	9	0	8	0	1	none
qwen3.5:35b	4	0	2	1	1	none
cogito:32b	9	0	8	0	1	none
command-r:35b	9	0	8	0	1	none
mistral-small3.2:24b	9	0	8	0	1	none
hermes3:8b	9	0	8	0	1	none
lfm2:24b	4	0	4	0	0	none

The following table is from the last run in which gemma4:31b was able to successfully write a decryptor for binaries with 1, 2, and 3 rounds of XOR functions.

model	fixture	rounds	final grade	completed	decryptor correct	py blocks	status	elapsed sec	MCP calls	error
gemma4:31b	/fx_r0001_sl0016_sp0000_dwarf.exe-2f588b	0001	true	True	True	1	completed	958.73	8
gemma4:31b	/fx_r0002_sl0016_sp0000_dwarf.exe-861fc4	0002	true	True	True	1	completed	1101.385	11
gemma4:31b	/fx_r0003_sl0016_sp0000_dwarf.exe-135f91	0003	true	True	True	1	completed	918.604	9
gemma4:31b	/fx_r0004_sl0016_sp0000_dwarf.exe-9f4750	0004	inconclusive_timeout	False	False	0	ollama_timeout	2287.903	10	TimeoutError: timed out
qwen3.6:35b	/fx_r0001_sl0016_sp0000_dwarf.exe-2f588b	0001	true	True	True	2	completed	414.498	23
qwen3.6:35b	/fx_r0002_sl0016_sp0000_dwarf.exe-861fc4	0002	true	True	True	4	completed	281.356	13
qwen3.6:35b	/fx_r0003_sl0016_sp0000_dwarf.exe-135f91	0003	false	True	False	1	completed	304.465	29

The analysis of the 4-round binary timed out after 1800 seconds (30 minutes). Previous runs were able to decrypt 4-round binaries, but not 5-round binaries. There is evidence in this setup that the models start to degrade as the binary becomes more complex, but the results are still inconclusive.

Failures

Disclaimer: The failure report was generated using AI. I personally find the failures fascinating.

The failures cluster around a small set of reverse-engineering translation hazards:

• uint32_t + uint32_t wrapping before a later uint64_t cast
• C unsigned literal widths, especially constants ending in u
• C cast timing, such as (uint64_t)(expr) where expr was already evaluated at 32 bits
• CONCAT44(a,b) high/low argument interpretation
• C operator precedence involving +, ^, |, <<, and >>
• 64-bit rotate idioms such as (x << 21) | (x >> 43)
• final derive_state accumulator behavior and final xorshift32(...)
• preserving byte order from mixed character/hex array initializers
• implementing every generated round function in order

The most interesting lesson so far is that the models often do the hard-looking part first. They find the right functions, recover the encrypted bytes, and understand the XOR loop. The thing that breaks them is often much smaller: one C integer-width rule, one cast at the wrong time, or one generated round translated almost-but-not-quite correctly.

1. The Hardest Failures Are Now Translation Errors, Not Tool Access

The strongest models usually find the right region of the binary: main, derive_state, xorshift32, encrypted bytes, and the generated round functions.

The most important remaining failure class is translating C/Ghidra semantics into Python exactly. This shows up as scripts that look plausible, run successfully, but print non-plaintext bytes.

Note: 20260611-** is the id of the run 

Confirmed examples:

• gemma4:31b, 20260611-aa, fixture 0004

  • The model recovered the right functions and wrote a full decryptor.
  • Failure was in R3: it translated (uint64_t)(0xc8e57b40u + m) as a wide Python addition instead of u64(u32(0xc8e57b40 + m)).
  • One-line fix made the script print Hello, World.

• gemma4:31b, 20260611-cc, fixture 0005

  • Same root cause in R3.
  • Correct C: p->b ^= (uint64_t)(0xc8e57b40u + m);
  • Model Python missed the 32-bit wrap before widening.
  • One-line fix made the script print Hello, World.

This is best classified as incorrect-reasoning with a bad-type-recovery or cast-timing secondary cause.

2. Prompt Updates Improved gemma4:31b, but Did Not Fully Solve Cast Timing

gemma4:31b improved materially after prompt changes. In 20260611-cc, it solved fixtures 0001 through 0004 and failed on 0005. Earlier, in 20260611-aa, it solved 0001 through 0003 and failed 0004.

The remaining 0005 failure shows that telling the model to use ctypes is not enough. The model imported or mentioned fixed-width behavior but still used raw Python arithmetic for a critical intermediate expression.

The prompt now needs to force helper usage, not just mention ctypes:

def u32(x): return ctypes.c_uint32(x).value
def u64(x): return ctypes.c_uint64(x).value

Critical translation rule:

(uint64_t)(a_uint32 + b_uint32) -> u64(u32(a + b))

This is the kind of bug that makes the benchmark useful to me. The model is not completely lost, but it is also not correct. That middle zone is where a lot of reverse-engineering automation gets interesting.

3. gemma4:31b Is the Current Best Local Baseline

gemma4:31b has the strongest completed results:

• 20260611-aa: solved 0001, 0002, 0003; failed 0004.
• 20260611-cc: solved 0001, 0002, 0003, 0004; failed 0005.
• 20260611-dd: solved 0001, 0002, 0003; timed out on 0004.

It is slow, but its failures are now narrow and mechanically diagnosable.

4. gemma4:26b Is Historically Useful but Unstable

gemma4:26b has repeated successes, mostly on fixture 0001, and solved 0001 through 0003 in 20260611-bb.

However, it also regressed repeatedly:

• wrong decryptor output on fixture 0001 in some runs
• timeouts on early fixtures
• empty or pseudo-tool response on fixture 0002
• inconsistent ability to proceed past fixture 0001

It remains useful as a historical baseline, but it is not as reliable as gemma4:31b.

5. qwen3.6:35b Became Interesting in Later Runs

Earlier qwen3.6:35b runs mostly produced invalid or non-printing Python, timed out, or failed to converge.

In 20260611-dd, it solved fixtures 0001 and 0002, then failed fixture 0003 with invalid/malformed Python. That suggests it is not merely MCP-compatible; it can solve the simpler generated fixtures under some settings. It still degrades as round complexity increases.

6. Many Models Are Tool-Compatible but Do Not Converge

Several models can call MCP tools but fail to produce a usable decryptor:

• command-r:35b
• lfm2:24b
• some qwen3.6:35b and qwen3.5:35b runs

These failures usually are not MCP server failures. They are either:

• tool use without a final algorithm
• excessive tool loops
• target drift
• loss of the objective after accumulating tool output

lfm2:24b is the clearest example: it used many MCP calls in some runs but did not produce a Python decryptor.

7. Smaller Models Often Stop Too Early

The most common pattern for mistral-small3.2:24b and hermes3:8b is minimal MCP usage followed by no decryptor.

These are mostly incomplete-xrefs failures:

• did not inspect enough of main
• did not inspect derive_state
• did not inspect all generated round functions
• did not recover the encrypted bytes and loop bounds

8. Invalid Python and Fence Extraction Remain Separate Problems

Some failures are model output quality problems:

• invalid Python syntax
• Markdown/prose inside extracted code
• several fenced blocks, none of which are a clean decryptor
• claimed plaintext inconsistent with script output

There is also a harness extraction issue observed in 20260611-cc for gemma4:31b fixture 0005: the saved block_0.py was not the explicit Python block. It captured Markdown around the recovered-output prose because earlier c fences confused the plain-fence extractor. The actual Python block in answer.md ran, but printed wrong bytes until the R3 cast-timing bug was fixed.

This means two separate checks are needed:

1. Did the model write a correct Python decryptor?
2. Did the extraction harness capture the intended Python block?

Back to non-AI-ish text.

Limitations

There are a few caveats worth calling out before the conclusion. The prompt changed during the study, and some of the later improvements came from manually comparing generated Python against the original C and feeding that back into the prompt. Ollama updates may also have changed model behavior, especially for gemma4:26b. There was also at least one harness extraction issue where the saved Python block was not the intended final decryptor. Finally, timeouts are inconclusive. They show that the run did not finish inside the configured timeout, not that the model could never solve the fixture.

That means the results are best read as evidence from one local setup, not a universal ranking of models or a final statement about LLM reverse-engineering ability.

Conclusion

More rounds do not make solving impossible, but they substantially increase the chance of failure once a model must preserve a longer state mutation chain.

The dominant complexity effect is not discovering the high-level XOR scheme. Models often find:

• main
• encrypted bytes
• seed
• derive_state
• xorshift32
• generated round functions

The failure emerges when translating every round exactly:

• more generated functions to implement
• more mutable state updates to preserve in order
• more C integer width boundaries
• more unsigned literal/cast timing traps
• more rotate and precedence opportunities for one-bit state errors

For this fixture generator, prompt, harness, and local model setup, the practical threshold appears to be:

rounds 1-3: solvable by gemma4:31b with good reliability
round 4: boundary where failures/timeouts start
round 5: current failure point for gemma4:31b

That conclusion should be treated as provisional because there are few high-round samples.

Next Steps

• Automated prompt improvement using failure analysis

  • This process was done manually by comparing the generated Python code against the original C code, then having the harness recommend upgrades to the prompt.
  • This substantially improved the results. qwen3.6 started working after the first iteration of this approach.

• Compare against frontier models.

• Keep learning and keep reading.

Any recommendations? I would love to hear feedback, hints or thoughts from others. Feel free to send me an email to Alexander dot Hanel at gmail dot com or leave a comment. Cheers.

Frost64

At my day job, I run a RE mentorship. It’s a simple process for me. I have a link to the task, and the user emails me the answers to the questions in the task. If their answers are correct, I send them the next task. If the answers are incorrect, I let them know their mistakes and send them references to help. Sometimes people get tripped up with learning Assembly, or they think it’s boring (which it kind of is). One tool I have always wanted was a way to gamify the learning of Assembly.  So, my vibe-coding weekend project was a SoftIce-like interface that teaches the basics of Assembly. It uses the Godot gaming engine, Unicorn-Engine for emulation, and iced. I hit my quota for the next couple of hours I figured I’d post a screenshot:

Overall I’m very happy with it. It’s a complete emulator, has correct addresses, bytes, validation and reminds me of SoftIce. I still need to audit the text in the lesson notes because they are duplicated in the command window text. Once it’s complete I’ll post the code to GitHub or maybe post the game on Steam.  

I’m still working on my Stressing LLMs project. Some local models that I tested failed miserably on writing basic python decryptors. I have been digging into proper prompts, skills.md and settings for Ghidra. That project should be out in a couple of weeks.