\tHave you ever heard of The 78 Project? It’s a 2014 documentary film and recording project where a couple of filmmakers, inspired by the work of acclaimed folklorist Alan Lomax and his early 20th century recordings of folk music, got their hands on an antique audio recording apparatus similar to what Lomax had used in his recordings. With this recorder, they traveled the United States and had musicians perform traditional folk, blues, gospel, and Cajun music, then produced a documentary film and album documenting the process.
\n\tThis project really fired up my imagination, although not necessarily in the way it was intended. While I was completely on board with their project of recording modern musicians with this obsolete technology, I was also very frustrated at the possibilities that they had left on the table. Why, I wondered, if you were to get your hands on this historical recording technology, would you ask modern musicians to perform only music similar to the recordings we already have from when that piece of equipment was modern and new? That is literally the only type of music we have in this format. So why make more of it?
\n\tI conceived of a sequel project which I named in my head The 79 Project (78 + 1 = 79, also the year of my birth) and the concept would be in some ways the same as the 78 project, but in other ways the complete opposite. I would travel the country, recording musicians making modern music with these ancient tools. That seemed so much more compelling to me. It would be such a good way to make visceral the fundamental reality of one of the most important limitations on human creativity: you can't create something if you cannot conceive of it. The 79 project could have hip hop, R&B, beatboxing, and more. It could feature all the new types of music that we know today, and which would have been completely technologically possible in the 1930s, but which were completely impossible at that time, because human musicians and human culture had not yet put in the work of expanding our horizons to encompass these modern genres and techniques.
\n\tThat stark contrast between what was technically possible then and what was culturally possible would be a beautifully succinct illustration of the invisible and really quite hard to fathom limitations imposed on our thought by the possibility space we are able to envision and explore, a sort of cultural Overton window of the imagination. I have a feeling that the machine learning models we already have can do a great many things we haven’t yet thought to ask of them, and that one compelling demonstration can get a great many people thinking about what else they might be able to do. It was only years later when I was introduced to the concept of technological overhangs, meaning potential capabilities or resources that have not yet been exploited or developed. I first came across this concept in the context of large language models and the way that they tapped into the incredibly valuable information overhang created by millions and billions of human hours worth of thought recorded and accessible in the form of the internet. Much of that thought is incoherent or unreasonable, but not all of it. It contains a fair cross-section of all human thought, spanning the entire spectrum from worthless to priceless. And the availability of this linguistically encoded intellectual resource is why, for instance, language models have come so far so fast while machine learning models built from different primitives such as Yann LeCun's JEPA (Joint Embedding Predictive Architecture, a type of world model predicated on predicting real-world physical effects and consequences instead of predicting language in the form of tokens) are still struggling to make it out of the lab with a maturity level as a technology that I would characterize as similar to that of LLMs of the GPT-2 era (an early OpenAI LLM that predates ChatGPT and is extremely primitive by today's standards), even though I would estimate that significantly more intellectual talent and compute have been poured into world models like JEPA than was required to create the original GPT-2 itself.
\n\tI was reminded of this again recently when a new machine learning architecture (called a System One model by their creator, TypeSafe AI) and their new model, JEV, hit the internet and caused quite a sensation. Some have claimed that JEV is not actually anything new and whether or not it is new is both beside the point and entirely the point, in my eyes. What really had me excited about JEV was not, or at least not only, its remarkable capabilities in terms of latency, throughput, and efficiency, but the ways in which JEV was built. We don't know exactly how it was built because it's a closed source architecture, but the sheer number and variety of fast followers released by seemingly every machine learning practitioner in the world within days and weeks of JEV's release, and with similar capabilities, suggests that we do have a pretty good idea of how it was built even if we don't know the exact details.
\n\tJEV and its many cousins and siblings, called "System One models" by JEV's creator, or also "decision models" by many others, are made by taking the pre-trained backbone of a typical LLM (Alibaba's Qwen series of LLMs is very popular for this use case) and replacing the decoder or output components of the model with a task-oriented output head, configured such that the product of the model's inference is no longer words in the form of tokens, but specific selections from a finite menu of options and an attached confidence score. Thus you can give the model a question + a state (an input describing a situation or condition to which the question is applied) and JEV will select any of up to 255 possible outputs and also give you a confidence score about how certain it is of its answer. The trade-off in return for restricting the model's possible outputs from the entire world of language to at most 255 specific options is that these questions can be answered in parallel at great speed and for a cost so cheap that it's nearly free. There's a bit more to it than that, but that's the gist.
\nSome might argue, and reasonably, that reusing pre-trained representations with task-specific output heads is nothing new. For instance, BERT could already be fine-tuned with a task-specific output layer, but I would argue that when it comes to capturing the imagination and harnessing the efforts of machine learning researchers, it is one thing to know that someone did a thing once in the dim dark days before the chatbot boom, but it is entirely another thing to have a historical approach validated so clearly and effectively as is the case with JEV. Surely there was no shortage of machine learning engineers familiar with BERT and other similar early experiments, but in terms of awareness shaping efforts and decisions in the work of researchers pushing machine learning forward, an entry in the textbook is one thing, but an exciting new model with unexpected new capabilities is far more salient and far more influential.
\n\tApart from revealing a new category of utility and functionality made possible by this reinterpretation of how a machine learning model should be configured and interacted with, this really opened my eyes to the capabilities overhang represented by these pretrained model structures, which are not designed or built so much as grown in a process known as pre-training, where they are fed a steady diet of almost every piece of information we can get our hands on. And I don't think I'm the only one whose eyes were opened in that way. In fact, I'm quite certain that one of the effects of JEV on the machine learning world will be to open up the eyes of a great many machine learning researchers and practitioners to the enormous potential that can be accessed by taking these information structures that we've grown so that we could put them to use in these models and agents and recognizing the potential utility and untapped capabilities that may be inherent in these vastly complicated information structures in the form of billions and trillions of tuned parameters.
\n\tMuch like the information and reasoning overhang that is the corpus of the entire internet and which was largely invisible to humans until large language models revealed how much value was inherent there, JEV has revealed an equally or similarly vast overhang of capability inherent in these information structures and shown that such potential can be tapped by reexamining the structures and affordances surrounding these models and approaching them from novel perspectives, without regard to how they were intended to be used. People frequently talk about if or when the machine learning community will have another breakthrough like the transformer architecture. And I've never been anything but convinced that of course we absolutely would. I don't base this out of a deep understanding of the mathematics or other details of how these systems are built. My confidence comes entirely from comparing where the field was in 2017 and its relative share of the intellectual capacity and energy of the human race and its share of the analytical resources created by technology and comparing that to where the field stands today, almost ten years later. A decade ago, a high school senior who didn't know what they wanted to do with their life, but who is immensely talented in the field of mathematics and science, and who, for lack of a better idea, decides to pick their field of study based on what is most lavishly rewarded by society, might have chosen to be a quant at a high-speed trading firm or go to med school and become a specialist. And I don't doubt that the same student today might consider those same options. But to the extent that they are primarily interested in going where the action is, I would have to imagine that the possibility of studying AI has likely also caught their attention, not least because of the myriad reports of seven and eight figure paydays that, in some cases, dwarf even the compensation afforded to professional athletes of the highest degree. While it would have been the rare math or science whiz who would have chosen to study neural networks and machine learning theory in 2017, I have a feeling that that choice would not be nearly unique enough to catch anyone's attention today. Which is to say that while I don't have any figures ready to hand, I feel quite confident in the assumption that the sheer quantity of talent and energy and resources being poured into machine learning today exceed the equivalent quantities from 2017 by orders of magnitude. I'm sure I could do some research and get the numbers for that, but I don't know that any numbers would convince me that something similar to that is not the case, and there are other types of research that I think would contribute more to my thinking, and assist more in predicting what's to come.
\n\tMy prediction, which is non-specific but firmly held, is that the "JEV moment", which recently got so much attention and created so much enthusiasm in the world of machine learning, is neither an isolated event nor even nearly the last such reframing of what machine learning models can be expected to do that we can expect. My prediction is that in retrospect, years later, JEV will come to be understood as the opening salvo of a new Cambrian explosion of machine learning model architectures. While it's true that the machine learning research community has always been exploring alternatives to existing architectures and improvements upon them, it is my suspicion that it has also become somewhat captured by a certain kind of tunnel vision that caused people to think of even alternative architectures in terms of the architectures we have today. For instance, alternatives like Mamba and other state-space architectures are proposed as an alternative to LLMs, but they are an alternative that is also substantially similar to what it seeks to replace. What I foresee is not a replacement of LLMs, but a proliferation of other modalities as the machine learning community recognizes the overhang of capability represented by pre-trained machine learning models.
\n\tWhich is why, to the extent that I did do research before writing this, what I researched was the rate of innovation currently seen in machine learning research. This is somewhat of a murkier statistic than population or GDP. So my thinking was that the most useful labor to establish whether my idea and prediction are actually indicative of reality or are just wishful thinking on my part would be to establish some kind of baseline for what is the current rate of innovation today and in years past, so that I can measure it in the future and determine if it has indeed accelerated as I'm expecting or if the increase is more linear or possibly even non-existent. I don't think that's super likely, but it isn't impossible.
\n\tTo that end, I dispatched my assistant, aka my agent, to inventory and categorize academic publications on the topic of novel machine learning architectures or variations on currently existing architectures. The results are at the very least not inconsistent with my thesis. We estimated 6,197 qualifying papers in 2017, 28,140 in 2023, and 41,368 in 2025, which is definitely an increase, an appreciable one, but certainly the rate of increase resembles the parts of the graph before the hockey stick curve upward. The current baseline for July–September 2026 is an estimated 12,568 qualifying papers per quarter, or about 4,189 per month. These counts cover papers making substantive claims about learned-model computation, architecture, representation, or inference across six arXiv categories, rather than exclusively LLM papers or independent inventions. If I am right, then we should see an inflection point in the data right around now and in the coming months that should be obvious in retrospect as the overhang of intellectual capacity caused by so much more enthusiasm and resources being poured into LLM research will meet with the capabilities overhang represented by these pre-trained architectures and the knowledge about the world that is encoded within them in ways that we hardly even begin to understand. Just to make the prediction concrete, I'm going to say that I think the increase in the innovation rate seen over the last two years will be no more than half of the innovation rate measured in the coming two years. I think the effect might be more than that, but I think that is a strong enough prediction to be falsifiable while being conservative enough that I feel like it has a good chance of being true.
\n\tI can't wait to find out which one it is!
", "author": { "name": "Jefferson Moss" }, "tags": [ ], "date_published": "2026-10-18T19:00:00-05:00", "date_modified": "2026-10-19T13:53:55-05:00" } ] }