Exploring APTNotes with LLMs

I’d like to share a few projects I’ve been working on.

The first project is APTNotes-md, which is APTNotes converted to Markdown format. If you’ve never heard of APTNotes, it’s a GitHub repository of APT-related threat intelligence reports in PDF format, organized by year. I created APTNotes-md for anyone who wants to explore the data using large language models. APTNotes-md contains additional reports that I added because the original project was last updated in 2024.

The second project is feed-report-monitor, a pipeline for converting RSS feeds into PDF and Markdown files. It relies on RSS, Python, Marker, PyTorch, Ollama, and local models. This pipeline was also used to convert the APTNotes PDFs into Markdown. Converting 869 PDFs took about five days on a DGX Spark. The process could have been significantly faster, but I chose to use a local gemma:26b model to verify that the Markdown conversion was formatted correctly.

The third project is report-eval. This project has been both the most fascinating and the most frustrating, so I guess it balances itself out.The project started as an attempt to build a SQLite database populated with metadata and attributes extracted from the reports in APTNotes-md. The goal was to allow an agent to answer questions such as, “Summarize all reports related to Operation Aurora,” returning both the results and their supporting sources using xSQL. Populating the database sounded simple: create a set of questions, have an agent build an evaluation harness, then iterate over every report and ask every question. It turned out to be anything but simple.

The report-eval project was restarted two or three times. To give you an idea of how bad it got, one Saturday night I became so frustrated that I switched to GPT-5.6 Sol Ultra, typed “fix this,” and went to bed.

Pro tip: don’t do this.

The next morning my weekly usage limit was at 0%, with six days remaining until it reset. I managed to burn through an entire week’s worth of tokens while I was asleep. I ended up using one of my four resets just to get the harness into a working state. In retrospect, I should have reviewed the plan in more details before repeatedly saying yes. After realizing it would take another four days to process all of the reports locally using gemma:26, I decided to use OpenAI to perform the report classification instead. I quickly burned through the rest of my resets over the next day or two and temporarily signed up for the Pro plan to finish processing the reports.

If you’re interested in learning from my mistakes, check out the after-action report. If you’d like to see some of the reasoning failures and pitfalls I encountered along the way, take a look at fallacies.md. A couple of the points could be a full blogpost.

Below are some results of the eval using Codex. 

Note: Start of AI Generated content

Main findings

gemma4:26b completed the corpus and produced relatively few final errors:

• 22 timeouts
• 17 unsupported status values
• 2 malformed evidence errors

gemma4:31b-it-qat was stopped after reaching approximately 4.48% of the corpus.

• All 416 failures were caused by empty answers.
• The model commonly returned status: "not_stated" with answer: "".
• This appears to be a response-format contract problem rather than necessarily an analytical failure.

hermes3:8b was stopped after approximately 15.57% of the corpus.

• 913 failures were empty answers.
• 928 used unsupported status labels such as stated, confirmed, or attributed.
• Only six failures came from timeouts or malformed evidence.
• About 99.7% of its recorded failures were therefore output-schema or validation failures.

openai-terra-high has no recorded mechanical errors, but this does not prove every classification is semantically correct. 

• The files contain no explicit correctness score or gold-label field.

Data-quality issue

• gemma4:26b contained 901 duplicated report/question pairs, adding 918 surplus rows. These appear to be retries appended to existing files.
• Using the raw rows gives gemma4:26b 423 errors, but many older failures were later corrected. Selecting the newest generated_at result reduces its current failure count to 41. Results should therefore be deduplicated before analysis.

Most problematic question categories

Within the partial Hermes run, the highest observed failure rates included:

• attacker_ai_use: 129/129 failed
• staging: 113/129 failed
• credential_access: 111/129 failed
• attribution_reported_confidence: 105/130 failed
• attribution_claim_source: 104/130 failed
• privilege_escalation: 98/129 failed
• exfiltration: 97/129 failed

These concentrations mostly reflect the same response-contract problems, especially blank not_stated answers and unsupported status terminology.

Overall conclusion

The dominant failures were not clearly incorrect APT classifications. They were failures to follow the required JSON answer contract. gemma4:26b successfully completed the evaluation after retries, while the two stopped models repeatedly produced responses rejected by the validator. Semantic correctness should be evaluated separately by comparing normalized labels against a designated reference set.

Note: End of AI Generated Content