So we had access to some genAI tools at work, and what started as a joke on July 3 (hey let’s build this in a week!), turned into a project I didn’t intend to do and lessons I didn’t expect I would learn.
- 65 days from July 6 to Sept 8
- 341 commits (across 25 days)
- 18,900 lines of Rust (which I don’t write in btw) + * 3,150 lines of tests (50 test functions), 35 source files
- 10,100 lines of python across 42 scripts as part of validation tooling
- 32,700 lines of markdown
- Published v0.1.2 at USNISTGOV https://github.com/usnistgov/sageRecon
The big lesson I want to talk about here isn’t the obvious one (recon is awesome and using mzsniffer, sage and metamorpheus was essential; or how much I learned about how each step works or assumptions needed).
So I was mostly using API access to Anthropic models using Claude Code in VS Code, and using git to manage everything (and of course the agent toolkit… which btw, I have been updating still based on how it is working in my hands across projects, since if you take a gander at the drift of NOTES.md and even the PLAN.md on sageRecon, you can see why I switched to stricter directions for using FINDINGS.md), but the models we had couldn’t use web search.
This ended up being a feature not a bug, and has made me re-think about context management a lot more.
Since I couldn’t use websearch, it means I had to build out a knowledgebase. I did this by vendoring in information (is this a real term or an AI made up term, I don’t know, but go with it). So I cloned in repos I wanted to inspect (Crystal-C, deltamass, IQMMA, MetaMorpheus, mzsniffer, PTM-Shepherd, pymzML, and of course sage). I ended up building in mzsniffer functionality, and the curated list of mods MetaMorpheus uses was a life saver for me to determine stats (that’s another dev story), and I saw how PTM-Shepherd worked and it was very clever. BUT, the real fun came in that I also vendored in notes. Some of these were discussions where I would “teach” the agent things (like it couldn’t predict fewer IDs if fasta got huge, so I explained it), but a lot of times I would dump a paper (or a timely blog post by Phil Wilmarth on Deamidation, which turns out was really hard) into Perplexity and ask for a summary. BUT it got really handy in later steps if there was an issue the agent was hitting, and I would say “what is a prompt and reference so you can get exactly what you need from it”.
https://github.com/usnistgov/sageRecon/tree/main/_dev/reference-notes
I didn’t realize how much got put in here over time, and I should likely go back and double check some of it for factual errors and general correctness, but having this locally was pretty cool. On one hand it feels like the GROUNDING.md concept where we define what is truth, not let the LLM go out and decide, but also the idea of having an LLM available knowledgebase (see karpathy I am sure).
BUT, then the idea came up again as I was fooling around with some cooking recipes on my own. I realized that though the LLM might have been trained on all information, it doesn’t really “know” the information. Like it doesn’t know there are general techniques and themes in cooking (ex. make a stock with a whole chicken, remove and shred chicken, strain stock, THEN you can make like 5 different southern dishes). Well same thing for MS proteomics and stats. The LLM knows all, but that means it knows ALL, which means it doesn’t know what to know, not really. So maybe something after GROUNDING.md is some verifiable proteomics knowledgebase to include in context? Not the worst idea.