Updates
Updates from my lab, July 13, 2026. Including a new LMLM model, recent insights contrasting state vs. prediction in Transformers, and some more.
Posted on: July 13, 2026
This is a reduced version of an update I recently sent via a new mailing list. I hope to send updates irregularly, maybe every few months. In this (practice and structure), I copy Colin Raffel.
What we’re working on
I am fairly excited about our next LMLM (limited memory language model). LMLM class models externalize knowledge into an induced KB, rather than internalizing it into parameters. Our first iteration showed interesting properties (factuality, unlearning, attribution, better scaling, etc – all come out naturally), but it was brittle and had serious scaling and expressivity issues. Our new version, Co-LMLM, solves most issues, replacing the relational retrieval with learned continuous retrieval. The results are interesting, e.g., 360M model gives better PPL than an equal-sized model trained on 40x data and better SimpleQA-verified performance on par with gpt4-o-mini and higher than Claude Sonnet 4.5. Plus, all the LMLM knowledge benefits, but better. LMLM is a large cross-lab project with Jennifer and Kilian. We are now shopping around for compute to scale (openly).
In a second paper, we formed and studied the state-prediction separation hypothesis (or SPS). This started as a compression project. We have done some work on compression (in RAG, and reasoning), but it left me quite unsatisfied. Compression should be adaptive to the amount of information flowing through. Our method was too adaptive though, and constantly found the inverse: using more flops, but differently gave lower pre-training loss. We realized that the model was happiest when separated into two streams: prediction and state preparation, and we designed a very simple SPS transformer, which brought about pretty interesting results in data efficiency, overall loss, and higher impact from future losses.
A third highlight is about the LM head and how it bottlenecks gradients. Bottleneck issues were studied with the forward pass, but our focus was training. It’s interesting to see how much of the gradient signal is lost. The crux of the paper is the theoretical analysis, but there’s a nice set of empirical experiments mirroring the theory remarkably well. We tried to think of ways to solve this issue, but so far we are stuck.
What I’m reading
MemSinks is an interesting line of work from Aditi Raghunathan’s lab. The first paper is almost a year old. The more recent one is more mature and is focused on unlearning. The idea is simple: designate a subset of your parameters as “sinks”, and activate only a small subset of these for each training example. To unlearn an example, just turn off the sinks activated for it. I generally like the idea of baking the properties we want from LLMs into training from scratch, in contrast to post-and-pray. I also find the exploration ideas in Dylan Foster’s recent work quite interesting. Simple, but effective idea: sample plenty from your model, but only compute rewards to a subset of rollouts that will give you good coverage of the space under some representation. So, you get sample efficiency in terms of reward computations.
What I’m thinking about
I am technically frustrated about how scale keeps marching on. The recent GLM is simply a beast, and the toll on memory just keeps growing. As a computer scientist, it feels almost offensive (although I appreciate the engineering efforts). Beyond energy and memory costs, I sense it drives us further away from autonomous compute (doing our own compute, and owning it), and that’s particularly painful given the data that flows into these models. One of the motivations for our LMLM and compression work was to address this. I somehow thought that blind scaling was just a phase. So far, I am proven wrong.
I am coming to see the publication system as hopelessly broken. COLM is doing fine. Submissions are 3x this year, with some undesired behaviors (generated papers, and the incentives for startups to try to get one accepted). But, overall, the process is running well (Greg Durrett is doing a remarkable job as an SPC, and the rest of the team is fantastic). There are a number of reasons for this. COLM is smaller, but it’s not only about size. My current thinking is that the academic world has put together a system under a certain set of values, but shows little willingness for an honest discussion now that the system has been gamed into meaningless dust. The game is largely about numbers, and the +1 you get for each paper is only meaningful as such.
Looking beyond my own corner of the woods, the societal costs of AI come to mind increasingly often. Labor is one. Not particularly original. I increasingly wonder about the framing the industry uses. “We will take all your jobs” is not only a warning, but it is also a guiding principle in how systems are designed and trained. Then there are the remedies, different versions of UBI, all absurd. Not because they are not possible – that aside. But because they fix so little. Could society have chosen different framing? What would our technical landscape look like in that case? Post-training data looks the way it does for a reason.
Two other societal problems that bother me are the data ecosystem (which I wrote a bit in 2023, and a recent piece about Wikipedia touches on) and the power of the state. I need to think more of the latter, and maybe write something. I did discuss it to some degree in a recent conversation with Helen Nissenbaum (I also need to figure out how to make firesides less stilted).