I spent the past few days reading through Elon Litman’s annotated walkthrough of JEPA, which is Yann LeCun’s proposal for how to train models to understand the world without labels, and the idea is simple enough to fit in a sentence and subtle enough that I keep finding new corners of it the more I turn it over in my head. The basic move is this: instead of asking a model to predict what pixels come next, which forces it to waste capacity on textures and lighting and JPEG noise, you ask it to predict what the representation of those pixels should be, which forces it to learn the semantic structure underneath. The model sees part of an image and has to guess what the hidden part looks like in embedding space rather than pixel space, and to make that guess the model has to understand what kind of thing it is looking at, a car or a face or a landscape, because you cannot predict the representation of the wheels unless your encoder has figured out that the hood belongs to a car.

The clever part is how they stop the model from cheating, because the obvious way to minimize a matching loss is to output the same vector for everything which drives the loss to zero and the representations to uselessness. I-JEPA prevents this with an asymmetric design where one encoder is updated by gradient descent and the other lags behind via exponential moving average, so the student has to actually learn to predict what the slowly-evolving teacher will output rather than chasing a moving target into collapse. The predictor itself is a small Transformer that takes context tokens and mask tokens and tries to fill in the blank representations, and the whole thing runs on a Vision Transformer architecture with block masking that splits each image into context regions and target regions in a way that forces the model to reason about things that are spatially separated.

What makes JEPA interesting beyond the architecture is the trajectory LeCun has been building toward. The same pattern extends to video with V-JEPA and to planning with V-JEPA 2, where the model learns to predict future states in representation space and an agent can plan by searching through those abstract state trajectories without ever having to hallucinate pixels for hypothetical futures. There is an implicit argument in here about large language models, which is that predicting the next token is a form of prediction in the space of discrete symbols that encode human communication patterns rather than physical reality, and a model that predicts text can describe how a bicycle works without having any internal model of balance or momentum, whereas a JEPA trained on video has to actually learn the structure of persistence and occlusion and dynamics because that is what it takes to predict what happens next in the physical world.

LeCun’s broader vision puts JEPA at the core of a world model that sits inside a larger architecture for autonomous intelligence, with a perception module and a cost module and an actor and a configurator, and the whole thing plans by searching over latent trajectories in representation space. I have spent enough time with this idea to believe the insight is worth understanding even if the autonomous intelligence part turns out to be harder than the papers suggest, because the core move, prediction in a learned abstract space where the representation itself determines what is worth modeling, is one of those things that shifts how you think about the problem even when you are not building the full system. The question it forces you to ask is what your representation space should preserve and what it should discard, which is a question most of us never articulate because we are too busy optimizing the wrong objective. Finding the latent space that captures what matters is the work, and the rest is architecture.

Find me on Twitter if you are chasing the same representation. I am @troysk704.