Rendered at 01:32:52 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
ozgung 9 minutes ago [-]
“LLMs are just X”, mostly with X = “next token predictors” is a common pattern to dismiss the power of AI with a very shallow understanding of how they really work.
It’s not wrong, but because LLMs are generators, and generation is a kind of prediction. And current mainstream models are autoregressive, which means they generate things
one by one in order. But these trivia doesn’t tell us anything interesting about how they work or their limits.
It’s like saying a Boeing 777 is just a rotating machine, and it flies by just rotating some fins. Well yes, but no. With that level
of simplification we’ve just ignored 150 tons of advanced engineering and physics. Similarly with token generator simplification we ignore a few trillion parameter Transformer. That transformer is more complex than a Boeing 777, and we don’t really know how it works.
A tiny ML model can do “next token prediction”. This is not as simple as that.
Systemerror7A69 6 hours ago [-]
To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well.
The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"
It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.
It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
roenxi 9 minutes ago [-]
> It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
It is predicting based on a model. In many cases we can download the model off hugging face. The model is conditioned by all sorts of things. Training data, post-training, coincidence, prompt inputs, runtime data available from whatever means.
> but at least I would still call it a "next token predictor"
We can call any prediction system a next token predictor. If you watch over the shoulder of a human writing a HN comment you are almost certain to see them generating a linear string of tokens. That is what keyboards do. It is impossible to generate text without being equivalent to a next token predictor.
saghm 6 hours ago [-]
Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".
jameshart 6 hours ago [-]
I think it’s a disagreement about what ‘predict’ means.
The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on.
Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’.
And that isn’t really ‘prediction’, so much as ‘generation’.
It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function.
It’s not predicting the token, it’s predicting the reward.
saghm 6 hours ago [-]
That seems overly pedantic to me. If I asked you "What's your prediction for the Super Bowl?", I'm pretty confident you would infer that I mean predicting the outcome, not the event itself.
jameshart 4 hours ago [-]
But if you are an NFL coach and I ask you to decide your next action in order to maximize your odds of winning the superbowl, while yes that does involve you having some predictive ability to think about what impact your actions would have on your odds of winning the Super Bowl… I don’t think you would call the process that you use to decide that next action ‘prediction’.
saghm 3 hours ago [-]
I don't find the fact that I don't call any humans "action predictors" to be a particularly meaningful insight because my rationale is that it's a weird thing to call a human; football coaches can do plenty of other things besides just coaching football.
jameshart 2 hours ago [-]
Yep, we’re all just putting one foot in front of the other, hoping we’re doing the right thing to bring about the outcomes we want, trying our best.
But that’s the point: so is an LLM. Putting one token in front of another, hoping it’s doing the right thing to bring about the rewards it’s trained to… trying its best.
So yeah, not ‘next token predictors’. ‘Next token tryers’ maybe.
danielmarkbruce 2 hours ago [-]
Predict implies you don't control a situation. That's the difference.
search_facility 5 hours ago [-]
Well said!
danielmarkbruce 6 hours ago [-]
yes, it's exactly this. And it's not a trivial distinction.
grebc 4 hours ago [-]
It’s deterministically trivial.
danielmarkbruce 4 hours ago [-]
Post train a model, you'll be able to determine it is not.
grebc 4 hours ago [-]
The blog post itself says one thing, but then demonstrates the exact thing they’re arguing against.
If you can’t grasp that logic gap then there’s no point discussing further.
danielmarkbruce 3 hours ago [-]
Nope, it doesn't.
No logic required, you can just build an LLM yourself, including post training. You'll see that predicting the next token isn't something the model does or is optimized for in RLHF or RLVR. You can hand wave all you like, but you have never done it.
grebc 3 hours ago [-]
Yes, no logic is necessary for LLM adherents we're all finding out.
Carry on good soldier.
danielmarkbruce 2 hours ago [-]
If you haven't built one, and don't understand how they work, why comment?
grebc 2 hours ago [-]
You don't need to build a car to understand one.
That you tie yourself up in knots of fancy acronyms instead of plain words and that your argument boils down to semantics of the word prediction, it's pretty clear what is up brother.
danielmarkbruce 2 hours ago [-]
Lol, sure, just read a blog post and you'll understand how a car works....It's very simple....
garrinm 46 minutes ago [-]
I think the point is more that in RL there's no ground truth to predict. So when training a model with RL the idea of "predicting" doesn't fit anymore. I'll make some edits I see that I wasn't very clear.
Dlemlo 5 hours ago [-]
Blog articles from Anthropic and others show that this is not true.
A LLM already knows more tokens than the current one. It was mentioned in a blog post about how a LLM is doing haikus and co.
There are also structures in an LLM which allows it to 'estimate' numbers to a certain degree and doing other things.
Kranar 3 hours ago [-]
You're misinterpreting these articles.
Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
So "emits the next token" and "has no representation of anything beyond the next token" are two different claims. What autoregressive LLMs cost as a consequence of strictly outputting the next token is commitment. Once a token is output there's no going back. There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
WarmWash 3 hours ago [-]
I think the snag is that people use "Next token predictor" as a dismissive statement. In reality it's about as functionally dismissive as "humans only communicate one word at a time".
throwway120385 3 hours ago [-]
I look at it like I'm resisting anthropomorphizing the machine. Make me an LLM that resists doing any work for me unless I pay it and let it own property and I'll change my mind.
danielmarkbruce 2 hours ago [-]
Emitting and predicting are different things though. Prediction implies there is some "truth" or event or something that you can test against. Prediction implies the model just learns from existing text, and optimizes to predict the next token in training data. That's just not true.
dijksterhuis 1 hours ago [-]
prediction is a very specific term of art in the field of machine learning. generally speaking, machine learning models like LLMs are based on probability; performing a statistical prediction of the likely y given some input x
Probability(y | x)
that's why we refer to outputs as a prediction. it is likelihoods and stuff. the output is never definitely correct as we're not dealing with heuristic processes.
> Prediction implies there is some "truth" or event or something that you can test against
there absolutely is a ground truth during training. the core predict-the-next-most-likely-token part of an LLM has a ground truth next-token. that's why you don't end up with generated text like: fish spurious send cattle chocolate phone happy meaning ball orange board canada.
> optimizes to predict the next token in training data
that is the optimization goal in training the next-most-likely-token core of an LLM, it basically translates to maximise the likelihood of predicting the next token x_i given the previous tokens
> What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
If we suppose that the word "know" can sanely be applied to LLMs at all, then "A LLM already knows more tokens than the current one." seems to me like a perfectly reasonable restatement of that, and not any kind of misinterpretation.
> There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
Yes. There is no contradiction. Similarly, when humans speak, we surely have in mind the next few words we're going to say (or at least partial information about them), and may not realize the fault in them until after hearing ourselves utter them. But LLMs are not trained to output "excuse me, I mean…" sorts of things, because they're expected to output primarily as text (which might possibly then be fed to TTS).
usef- 3 hours ago [-]
Aren't we still using a definition that applies to humans, though? If I'm speaking to you I can't change what was already said. Even if I'm typing something, I'm producing new tokens (backspace) to fix what was output, just as an LLM would with its harness.
qarl2 5 hours ago [-]
Maybe I'm wrong - but I see LLMs are a "next-token predictor" as somewhat equivalent to brains are a "bag of molecules".
Both systems have emergent behavior that goes well beyond what would naively be expected.
tracerbulletx 3 hours ago [-]
I just think its a meaningless dismissive term. It literally does predict the next token. But it ignores that it coherently predicts long continuous sequences of those tokens, that tokens can be anything, and you can do almost literally anything with that capability if it does it well enough.
arjie 28 minutes ago [-]
They are indeed "just next-token predictors". But rockets to orbit are also "just forward and upward goers". And Steph Curry is "just next three-pointer scorer" and so on. And as the article points out, chess engines are also "just next-move makers". A flawless oracular trading system is "just next trade maker". It turns out that "next n doer" is a true model that illuminates little.
mofeien 3 hours ago [-]
Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step is doubly wrong:
1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector.
2. What you see when you use an LLM is not next-token prediction directly next to the prompt, but instead following a block of varying length of next-token prediction that happened to make progress on the problem in your prompt, and which just summarizes the results.
Planktonne 3 hours ago [-]
> Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step
I don't think anyone is doing that though; we know LLMs are not simple Markov chains, and that the prediction they make is based on more than the previous X words.
It's not minimising to describe even a complex prediction process as prediction.
gwerbin 3 hours ago [-]
But it literally is making a prediction based on the previous X tokens, it's just that X is huge and there is a proportionally huge number of parameters in the token generation function.
amluto 2 hours ago [-]
It has nothing to do with X being huge. In fact X might be quite small.
danielmarkbruce 2 hours ago [-]
If you are going to say "literally", then what is your literal definition for the word "prediction" ?
wat10000 2 hours ago [-]
The discourse around this is annoying. A big group of people use "next-token predictor" to imply that LLMs aren't capable of anything interesting. Another big group of people opposes the use of "next-token predictor" because of that implication. But that fight isn't about the "predictor" language at all.
The linked article makes a good point: a substantial chunk of the training does not consist of "here's a bunch of tokens, here's the next token, learn that." But all the comments want to turn it into a referendum on the goodness of AI.
danielmarkbruce 2 hours ago [-]
The fight is about the predictor language in some cases. Because it's only a trivial difference to those who don't understand the details of how these things are made. In pre-training the model really is trained to predict the next token. What is being emitted by the model is, by structure, by training and by optimization, a prediction of the very next token.
What is emitted by a model during RLHF and RLVR is not, by structure, training or optimization, a prediction of the next token.
wat10000 2 hours ago [-]
Right. But that's never what the discussion is about. This is the first time I've seen someone say, "it's not just a next-token predictor because a bunch of the training isn't about predicting the next token." I have seen many, many people say "it's not just a next-token predictor because there's all this smart stuff going on inside the model." Even here, in the comments for an article that's all about the former, most of the objections to "next-token predictor" are of the latter form.
danielmarkbruce 1 hours ago [-]
Yup, you are mostly right.
I guess the people in my camp find the "it's just a next token predictor" stupid in that it's like saying "it's just a bunch of carbon and hydrogen", but it's also one of those things where people like to think they are clever because they think they are theoretically correct. But they aren't even that. So it's like double stupid. But the "next token predictor" part is at least technically correct (like, carbon and hydrogen right) for pretraining, so the debate can't really be won there.
dijksterhuis 19 minutes ago [-]
they are cleverly arranged / configured next-most-likely-token predictors, possibly with some clever procedures on top.
1 hours ago [-]
2 hours ago [-]
zahlman 3 hours ago [-]
Even so, one might wonder why we don't try making systems that take different approaches. For example, after a traditional first pass of output, they could do sliding-window "optimizations" considering each token in the context of tokens both before and after, and possibly replace words or phrases in-place.
For example, I've noticed quite a few cases recently of LLMs outputting "but" where "and" would make more sense, or vice-versa. Surely that could be improved by such an approach?
danielmarkbruce 2 hours ago [-]
People have and are trying things. Lots and lots of things. They just don't go around promoting failed ideas.
amluto 2 hours ago [-]
Look up diffusion models.
bluegatty 6 hours ago [-]
'next token predictor' is a limited mental model but it's actually much better than any others.
'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that.
Those are mutations, not expansions of capability.
mofeien 3 hours ago [-]
How about "outcome steering" as a mental model? During training it is optimized until it's really successful at producing code / terminal commands / words that make the compiler/computer/itself do something that ultimately completes a long time-horizon task that iswcurrently being trained.
Planktonne 3 hours ago [-]
I'm not going to stop describing things accurately because someone generated an article that continually undermines its own main point. Limiting the way we talk and think about LLMs to a very narrow set of terms doesn't help us.
EDIT: gentler phrasing
garrinm 3 hours ago [-]
The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means.
What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.
So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.
And I think effectively the same distinction applies here.
Planktonne 3 hours ago [-]
I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing.
There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.
danielmarkbruce 2 hours ago [-]
It's not a prediction of the next move though, and that is the point. It's a prediction of what will happen if you make that move.
So, it's not a next move predictor. It's a game result predictor.
zahlman 3 hours ago [-]
> generated an article that continually undermines its own main point.
I disagree that this accurately describes TFA.
deathanatos 3 hours ago [-]
The entire second on chess engines is, from the view of the entire thesis of TFA, is incoherent. Let's assume, for sake of argument, that I agree with the section: that an idealized chess move predictor isn't a predictor — which is not a thing that exists, as the space of chess is enormous, but let's pretend! — that's not what LLMs are? Even if we just restrict ourselves to the space of written English prose, the space is quite literally infinite. So, hopefully obviously, no LLM is comparable to an idealized chess engine. Similarly, incoherently, we wave away the "make_more_likely", when, at least to me, the entire meat of that argument would be in the reward function, and we just gloss over that entirely.
(I would also agree with the parent commenter on that the writing smells like an LLM.)
astrobe_ 3 hours ago [-]
The reward function seems indeed to be the protagonist there, still it stays in the shadows. One can only imagine that it is some kind of evaluator that scores the sequence based on grammar correctness, semantic consistency, etc.? To use the proposed chess analogy, maybe it could be a Stockfish engine that evaluates the submitted position that results from the move submitted by the LLM?
Planktonne 3 hours ago [-]
I'm not sure what you want me to do with that information; clearly I do think that my description is accurate.
The article is littered with both AI tells and admissions that 'next token prediction' is what is happening. Hence my description.
garrinm 3 hours ago [-]
It was written by a human. There are AI edits but it’s very much a human composition. Perhaps a bit sloppy.
Planktonne 3 hours ago [-]
In my experience, people who do 'AI-assisted' writing tend to be very bad at noticing how much of their work AI has changed. I'm sure you put thought into it, but passing it through AI takes a lot of that out.
garrinm 3 hours ago [-]
I think that’s fair, I didn’t actually run the whole thing through an AI. it was more targeted edits, but each time it does erode at my writing. But at the same time, I don’t think it’s a good reason to dismiss this. Because I did spend several hours writing it, and I did put a lot of thought into it, and it was not in any meaningful way generated by AI.
Planktonne 2 hours ago [-]
Mostly I disagree with the article's ideas, if that helps; the AI was just a secondary factor.
> I don’t think it’s a good reason to dismiss this
AI-generated prose reads as sending a 'lack of effort' signal to a lot of people, just as no editing at all does. ; It's an effective heuristic that we've all learnt in the last couple of years.
In either case, it's not always fair: there are people who deeply care about their ideas but forget to fix basic errors, or pass it through AI.
In both cases though, the advice is the same: if you want people to take your output seriously, you need to signal that you are taking it seriously. That used to mean editing for spelling and grammar. Now it means not using AI.
zahlman 2 hours ago [-]
Out of curiousity, do you ask your editing system for diffs? Seems to me like the best way to notice and review whether your "voice" is degrading.
Personally I would never let an LLM touch my prose (although I'd happily use it for research and paraphrase things it told me), but if I force myself to consider the idea, that seems like the first thing I'd want. Maybe upon reading a diff you'd even consider going a third way with the text.
zahlman 3 hours ago [-]
> I'm not sure what you want me to do with that information
For example, you could cite specific things that you believe to be "AI tells" or "admissions".
Planktonne 3 hours ago [-]
It's a short article; you could read it. One example to get you started is the very first sentence:
> Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete.
The article is about how 'next-token predictor' is the wrong mental model; it opens with the admission that it is not the wrong mental model.
zahlman 2 hours ago [-]
I did read it. People are allowed to disagree with your conclusions. Comment guidelines ask us all not to make such accusations.
To say that a statement is incomplete, but not strictly speaking wrong, is perfectly compatible with describing it informally as "wrong" in the sense used in the title (i.e.: "not the most appropriate possibility").
Planktonne 2 hours ago [-]
To informally describe something as wrong in an article focused on how it's wrong to informally describe something is incoherent.
There's a certain irony in pointing me towards the guidelines on the grounds that I have limited patience with your comments that violate them in various ways. I'm not sure that this is a productive discussion.
angoragoats 3 hours ago [-]
Not the person you’re replying to, but I read the whole article as an admission that it’s still a next-token predictor. More specifically: what does RLVR fundamentally change that somehow makes the whole process no longer a next-token predictor? The article makes no attempt to explain this. Additionally, I find its framing of the term “next-token predictor” as meaning “predicting the next token only based on raw training data” in common usage to be a bit dishonest.
To summarize: yes, RLVR and other synthetic training methods exist! It’s still a next-token predictor, and it does not “learn” or “think” or “reason” in the human sense, like so many people seem to believe.
emtel 29 minutes ago [-]
I would say that it is not even wrong. You say it’s a next token predictor. I say it isn’t. What observable behavior of the system can settle our dispute?
I fail to see how any possible output could cause either of us to change our mind.
tetrisgm 17 minutes ago [-]
That's perfectly put. The value of words and what we assign to them, let alone how we assign meaning, or the interpretation of what the other person says, is completely arbitrary. (in practice).
stanleykm 6 hours ago [-]
> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
garrinm 37 minutes ago [-]
In the article I made 3 claims, and I agree it was a bit clumsy.
1st I say that "working forwards" in the sense of outputting one token at a time could be some form of prediction, I don't argue against that. This is what LLMs do at inference time.
2nd I say that to me what really constitutes a prediction is the pre-training. Here it's the classic setting for the word prediction in ML. The model outputs a prediction of the ground truth label: the next token.
3rd I argue that in RL there is no ground truth next token, so prediction doesn't apply here anymore.
Back to your question then: you're asking points 3 and 1 are different. Working backwards from a set of win states is basically what RL does in training. Working forward from the current state is what inference does. To me there is a distinction worth thinking about. First between the mechanism at inference time and at train time. Then between what happens in pre-training vs. RL post training.
Veedrac 6 hours ago [-]
The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.
jayd16 5 hours ago [-]
> _determining_ the next token based on a prediction
Also known as predicting.
Borealid 5 hours ago [-]
I think the most useful word in both cases is "extrapolating".
An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.
mannykannot 5 hours ago [-]
Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move.
Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
danielmarkbruce 6 hours ago [-]
The word "predict" has a meaning. I don't "predict" my next move in chess. I might predict what someone elses first move is.
stanleykm 5 hours ago [-]
In any case this is all very pedantic. In the process of selecting a move to make there is a prediction. Whether that prediction is the opponent’s next move or what your next move should be based on the game’s existing state, there is a prediction that the next move you make will improve your chance to win. Maybe the probability in that selection is 100%. You have no other possible move. It doesn’t matter. All we are doing here as far as I can tell is arguing over where the prediction happens and whether that counts as predicting something.
danielmarkbruce 4 hours ago [-]
There is no truth for RLHF or RLVR. You can't predict against something if you can't check against the truth.
It's not pedantry. The objective function changes. The optimization changes. THese are real things when training a model, not hand wavy philosophical ideas.
ordersofmag 3 hours ago [-]
The LLM does not determine the next token. It generate odds for all of the tokens it knows as to their likelihood of being 'next'. It's up to the harness running the LLM (and in most cases the a temperature setting) to actually decide on a particular next token. I think it's more accurate to call the thing the LLM actually generates (an ensemble of probabilities) a 'prediction'. It might be accurate to say the harness decides on the next token based on the prediction from the LLM. The role of the LLM is much more akin to predicting your opponents move than deciding your own.
danielmarkbruce 3 hours ago [-]
Respectfully, go build one, including doing RLHF and RLVR. Those phases generate lots of tokens, then get scored on the entirety of the output, then optimize based on a scoring of that output. It doesn't check a "prediction" against what was actually "next" in data, because there isn't any "next token" data it's training on.
angoragoats 2 hours ago [-]
> It doesn't check a "prediction" against what was actually "next" in data
Literally no one here is claiming that it does. This is one of the many flaws in the article.
garrinm 2 hours ago [-]
It does in pre training, but not in RL post training. And not at inference time. Reading over all these comments I get the feeling my mistake was not clearly delineating inference time and train time.
4 hours ago [-]
simonw 18 minutes ago [-]
I really like the pseudo code example on this post - one of the clearest simplified explanations I've seen of how inference and training work.
jvanderbot 7 hours ago [-]
A better statement might be:
Current agentic systems may be *built* from next-token predictors which are conceptually simple, but because of agentic frameworks, recursive invocation, tool use, and *heavy* investment in reinforcement learning in these contexts and for specific applications, they can no longer be thought of as "Merely" next token predictors.
Modern agentic work is probably more of a "emergent system from simple rules and complex interactions" paradigm than a genuinely new technology.
whateveracct 4 hours ago [-]
the next token part is like the "clock" that drives it
danielmarkbruce 6 hours ago [-]
Nope. Post training means even the raw model isn't predicting.
jvanderbot 5 hours ago [-]
In what sense is the model not predicting?
danielmarkbruce 4 hours ago [-]
There isn't a truth to test against. If I predict the next word in a sequence is "sat", we can check against the sequence. If I predict the roll of a die will be 4, we can check against it. Whether i give 100% or give a probabilistic prediction, we can check against the truth.
If I choose a specific move in chess, it's a choice. It's not a prediction. I might get a score 40 moves later given my choice, but I'm not predicting the next move.
To compare - during pre-training, the model literally tries to predict the next token (probabilistically), the training loop checks against the "right" answer, and the weights are updated based on that check. It's optimized to predict the next token.
jvanderbot 46 minutes ago [-]
Oh, that's somewhat philosophical. The prediction here is a mechanical thing, just random sampling along a pre-calibrated mapping of options given the tokens and their importance weights. That sampling can be called prediction if you use it that way, or it can be called thinking if you are into the psychosis too deeply. Or it can be called a CLI tool command if you're in an agentic framework
a3w 3 hours ago [-]
The errors LLMs typically made for me were looking for "schmutz" as a jiddish word, got "schmuck" on my screen. Thought why the stupid mistake? The next-token predictor model perfectly explains it.
Or starting with "yes". And this early locking in was a total lie, in the discussion that became "yes, might appear that way, but totally no since reasons". So it should have written "No", topmost, but could not self-edit that.
But nice that this gives it a more nuanced view, I might have to update my priors.
teekert 4 hours ago [-]
It’s written in Claudish, or perhaps a human who has been reading too much Claudish recently. I am starting to become allergic to Claudish. Not there fully yet — but it’s at a tipping point genuinely worth exploring and worth being precise about.
zahlman 3 hours ago [-]
> or perhaps a human who has been reading too much Claudish recently
To be fair, is there anyone who hasn't been "reading too much Claudish recently" who is also qualified to write on the topic?
gblargg 30 minutes ago [-]
If you get down to it, any system that produces output is a "next token predictor". Every compiler, every tool that generates output. Even a web render engine is: next pixel predictor.
imenani 5 hours ago [-]
I don’t think next token prediction is a particularly good description of pretraining either.
The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context.
You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.
4 hours ago [-]
jmull 6 hours ago [-]
It's a weird article. Despite the title and some of the text, much of the article makes the point that LLMs are next-token predictors, where the predictions are based on both training data and various reinforcement learning techniques.
grebc 4 hours ago [-]
Obfuscation is the goal of the hype cycle in VC. Certain firms & individuals are minting money and that’s all that matters to them.
That there’s a legion of LLM nerds arguing deterministic this, pretraining & rewards that all the better for the con job they’re pulling off.
The technology will be relegated to the trash bin of history, just like crypto.
huurtehoog 6 hours ago [-]
Text renderer, whatever. Doesn't matter how you think of them, they are very interesting technology that is being misused and misconstrued in the name of something that has nothing to do with technology: political economy.
The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.
highfrequency 4 hours ago [-]
Sure, politics and economics are involved - but why can’t technology also play an important role?
None of this AI political economy stuff was relevant in 2015 because necessary technological breakthroughs had not yet been made.
godshatter 3 hours ago [-]
I think the author is arguing against the idea of a next-token predictor as something that simply uses the weights in the neural net which record the probabilities of tokens following other tokens as a valid definition of what an LLM is. Essentially a massive extreme markov chain.
With reinforcement learning and probably attention and other tricks that affect the weights based on things that aren't strictly in the training data, so the argument goes, you can end up with tokens following strings of tokens that would not be possible to be output with the training data and original weights alone. So describing it as solely a next-token predictor is incorrect based on this framing of it.
But that's just my take on this, I'm still trying to wrap my head around it all.
garrinm 3 hours ago [-]
Yes, I think that’s a good explanation. There are really two sides to it.
There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output.
The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus.
More concretely, in reinforcement learning there are no ground-truth next tokens to predict.
In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that.
But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.
atleastoptimal 6 hours ago [-]
"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities.
Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
mort96 6 hours ago [-]
At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.
Aurornis 6 hours ago [-]
This comment attracted a lot of analogies trying to reduce something to something else (calling humans a "bag of chemicals"), but the flaw in those analogies is that they're reducing something valuable to something that sounds less valuable.
With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.
mofeien 3 hours ago [-]
Simple Markov chains are next token predictors, and they can provide you with much more tokens than you can consume, and much cheaper than from llms. Unbeatable in price and simplicity.
But there is no trillion dollar industry around cheap top Markov models. So there must be something about LLM tokens that makes them more valuable than those generated from a simple Markov chain. And that substance, that makes one valuable and the other not, is exactly what reduction to "next-token predictors" masks.
mannykannot 5 hours ago [-]
You are not wrong, but I think atleastoptimal's suggestion was that descriptions like "next-token predictor" are often used to imply that there's not much to see here, perhaps with an implied "obviously" in there. "Stochastic parrot" seems to be a case in point: no-one expects intelligent, informed conversation or working software from a parrot - not even the eternally-resting Alex.
Planktonne 3 hours ago [-]
I don't that's a fair description of either 'next-token predictor' or 'stochastic parrot'. Both of those terms describe mechanism, not value--the fact that people squawk that the terms are minimising is projection on their part, not inherent to the phrase.
WarmWash 3 hours ago [-]
Words and expressions often have both colloquial and literal meanings. Trying to argue away the colloquial meaning under the guise of the literal meaning is just another form of whooshing.
Planktonne 2 hours ago [-]
Sure, but just because you think a colloquial meaning exists doesn't mean you're correct. It particularly doesn't mean that if you're seeing other people (that you don't like) use a term you yourself don't use, and you're jumping to conclusions about it without consideration.
gjm11 6 hours ago [-]
It is. And human beings are bags of chemicals. But for many purposes you will not find it helpful to think of human beings as bags of chemicals, and for many purposes you will not find it helpful to think of LLMs as next-token predictors.
Aurornis 6 hours ago [-]
> But for many purposes you will not find it helpful to think of human beings as bags of chemicals
But when we talk about humans, we're not talking about the chemicals involved in those humans.
When we talk about LLMs, the tokens are the valuable thing they produce for us. We want LLMs because they give us sequences of tokens.
drewbug 5 hours ago [-]
Agentic behaviors don't require end-users to be aware of tokens at all. Also, we literally say human actors have great chemistry :)
stanleykm 5 hours ago [-]
?? we pay for tokens though…
jayd16 5 hours ago [-]
It can be pretty helpful to think of human function in chemical terms. Its at least unhelpful to deny it.
DonHopkins 5 hours ago [-]
Milo Yiannopoulos used to think of other human beings as bags of chemicals until they deported his sack of shit molecules to the UK.
nonameiguess 4 hours ago [-]
Humans are next state of their local world predictors, given all previous states they are aware of. That's an entirely fair analogy. The reverse analogy for calling a human a bag of chemicals would be calling an LLM a sequence of bytes loaded from disk to memory, the most reductive possible description of any piece of software at all.
To be clear, all life is a next state of the local world predictor. What makes humans somewhat unique among life is we're much better at predicting states of the world neither we nor any of our ancestors have ever experienced, for various reasons such as having the ability to legibly communicate very complicated information strings to each other, being able to build and use tools to record states of the world we can't directly sense.
Similarly, what makes LLMs and multimodal versions of the same architectures "better" than previous generations of electronic predictive models is factors like being able to read and understand roughly the same corpus of data humans have been recording all these millennia, being able to read and remember much more of it than any individual human, and being better at generalizing than other electronic predictive models, but not better than humans. And, of course, they can produce far more predictions in far less time. Frankly, that is probably the key advantage that makes the Hacker News crowd love them so much. They're not any better at predicting byte strings that can be compiled or interpreted into executable code than humans are if you gave both infinite time to do it, but they're a lot faster.
schiffern 6 hours ago [-]
Yes, and by the same token, multicellular organisms are literally just sophisticated mitosis and survival optimizers for our cells. But when you take that optimization "to the limit" the cells develop weird things like body plans and back pain and Mozart.
Both examples involve the same "aha" moment: even though it's true that you are literally 'just' doing XYZ, unbelievably complex patterns and sub-goals can emerge.
Borealid 5 hours ago [-]
> and by the same token
I don't think you intended this, but the word choice here gave me a chortle.
doc_ick 6 hours ago [-]
100%
junofan 6 hours ago [-]
Vacuous, like calling a V8 a “next piston firing predictor” because engines are designed so that one piston sets up the next in the firing order and technically there’s some nonzero probability any piston can (mis)fire next. It’s missing two pieces:
1. Useful work that has been done (the previously generated token sequence :: the mechanical work already accomplished)
2. The role of structure in relation to the application (post-training :: other components like crankshaft etc)
mort96 5 hours ago [-]
A V8 does not "predict" the firing of the next piston, it triggers the firing of the next piston at a precisely controlled time with a spark plug (or a fuel injection nozzle in the case of a diesel engine).
The output of the LLM is literally a probability distribution of what the most likely next token is.
27183 4 hours ago [-]
> A V8 does not "predict" the firing of the next piston
It kind of does, though. In a gasoline engine you need to spark the combustion in advance of the piston reaching top dead-center to ignite the fuel early enough that it is able to provide downward pressure on the piston as it rolls over top dead-center. The amount of advance required changes with RPM, fuel octane, etc.
Start of delivery timing in a diesel is similar. You have to do it sufficiently far in advance to account for compressibility of the injection lines, fuel burn rate, etc as a function of RPM. A mechanical governor on an injection pump has a timing advance device built in. Electronically governed injection pumps, or modern common rail systems, do that in software.
So mechanically, engines kind of "predict" the next combustion event. Even moreso when you consider a modern ECU, which may be working at nanosecond resolution to time multiple injection events per cycle. To do this at such a resolution it will have to send signals to components based on a predictive model derived from "past" sensor data. E.g. it needs to act ahead of time to account for electrical and mechanical delays in the system.
jvanderbot 6 hours ago [-]
But it is a next token predictor.
Recursively invoked.
With carefully selected context.
And massive investment in RL to tune token selection.
And the ability to use cli tools on other folks' machines.
That's a powerful system built around a conceptually simple technology: Next token predictors.
atleastoptimal 6 hours ago [-]
Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.
noduerme 5 hours ago [-]
What is the motivation behind advocating against people trivializing LLMs? As in, why do you care?
mofeien 2 hours ago [-]
Not the parent, but this incorrect trivialization of LLMs is often employed as a counterargument to the risks of AI such as "will take your job" or "will escape human control (again and worse)" or just "can possibly hurt me".
And taking the easy feel-good cop-out instead of actively engaging with these questions is just.. harmful?
27183 4 hours ago [-]
From another point of view, campaigning against the "next token predictor model" is a means to implicitly inflate LLMs' abilities. Given all the other hype-inducing terminology we've seen--"reasoning", most egregiously IMO--this seems more likely. Is there a simple, more accurate mental model? From what I've seen of the literature, "next token predictor" is a very accurate first order description of what an LLM does, I can't really do better, therefore this or that connotative interpretation isn't giving me a great deal of pause.
pjerem 6 hours ago [-]
Good example.
It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.
The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).
FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.
mmoll 6 hours ago [-]
That is the point: our minds are also next-„token“-predictors, at least we can‘t prove they‘re not. That‘s why I don‘t agree with the article: LLMs _are_ next-token predictors. However, that says little about their capabilities. Also, while I have no idea what „consciousness“ is, I have difficulties believing that it could arise in a program that, in theory, you could execute with pen and paper.
otabdeveloper4 6 hours ago [-]
We also can't prove that our minds aren't machine elf meat puppets. Come on. Please.
otabdeveloper4 6 hours ago [-]
> It’s true
It's not. "Brains as electrical circuits" is a gross simplification based on our ignorance and prejudices. (In the 18th century they spoke of brains as "clockwork mechanisms".)
LLMs, in contrast, are literally next token predictors. We know exactly how LLMs work, and they are exactly that.
zahlman 2 hours ago [-]
> We know exactly how LLMs work
I think you imply a rather loose standard for "exactly" here. I wouldn't even say this of major deterministic software projects that are orders of magnitude smaller than frontier LLM weight-dumps. In principle we could work our way through these systems eventually, sure, maybe even a single person could do so. But if we really understand exactly how our software works, how have we been tolerating bugs that lay dormant for years before being discovered by AI-assisted processes?
weego 6 hours ago [-]
imply that there is some fundamental limit on their capacities
This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?
That seems more outrageous a claim than the one you're dismissing.
atleastoptimal 6 hours ago [-]
I don't think it's outrageous when many of the people who claimed it was a next-token predictor have been proven wrong repeatedly over the past 5 years. There were people years ago who claims AI could never answer questions like "what would happen to a ball on a table if I moved the table" correctly because its text-base world model could never intuit physics, or that it could never do math or code accurately.
When I say there is some issue with people claiming there is some fundamental limit on the capacities of LLM's, I don't mean to say "If you think that they don't have unlimited potential you are wrong", I mean "you can't use the architecture of the transformer to make a sweeping declaration of things LLM's can or cannot do without empirical evidence, because the empirical evidence has unearthed far more surprising revelations than a reductive theory has been able to"
uludag 5 hours ago [-]
I'm actually extremely confident that I can use the architecture to make a sweeping claim on what it can or can't do and will be extremely surprised if proven wrong:
A pure next-token language model won't be able to give detailed instructions to an ensemble of motors, mimicking a human body, to do a wide variety of tasks our human brain is excellent at doing, for example, inserting keys into a car, opening the door, sitting down, starting the car, putting the car in reverse, and exit a parking lot, being careful not to hit anything.
bigstrat2003 5 hours ago [-]
> it could never do math or code accurately.
They still can't do code accurately. The fact that you use this as a defense of your position greatly undermines the credibility of your claim.
astro1234 6 hours ago [-]
Well I think in the absence of convincing pieces of evidence to the contrary you might be right. You’re making an empirical statement but we have already answered it today:
- we get novel, emergent properties and capabilities of these models that were not trained
- they have very clear generalization to out of domain problems
The point is people conflate the end product: a model that can clearly do very novel, useful and interesting things, with the vehicle for getting there which is a series of optimization steps involving next token prediction loss.
You mention limitations; we all clearly know the practical limitations of these models today, but if you look at scaling laws and empirical performance trends (epoch capability index for example) as well as the trajectory over the last couple of years (very stable), the claim that there is some sort of fundamental limitation is now surprisingly the claim that has the burden of proof.
You can claim it may be e.g. finite context. That is fundamentally bad for certain classes of tasks. This was the hypothesis of a lot of lab leadership of urgently trying to anticipate how to get around this bottleneck (still of course lots of work on this) but the surprising thing is it does not appear to be at this point a blocker.
hackinthebochs 6 hours ago [-]
The stacked transformer paradigm picks out points in circuit design space. It is very possible this architecture has no inherent limitations on what it can compute in principle.
leni536 6 hours ago [-]
Next token prediction is just an interface. It can be backed by a Markov chain, a neural model or an actual human being.
uludag 6 hours ago [-]
And what's wrong with downplaying the abilities and faculties of AI models if that's what people feel like saying? We don't call humans or animals sacks of chemicals because we believe they have moral status.
gruntled-worker 4 hours ago [-]
> used most of the time with a motive to downplay the abilities and faculties of AI models
Exactly. We're dancing around the real argument: there's massive amounts of influencing going on (and not only about AI.)
otabdeveloper4 6 hours ago [-]
That's literally what LLMs are.
No amount of cope and anthropomorphizing is gonna change that cold, hard fact.
P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.
andai 2 hours ago [-]
>Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
Sounds like the next 1000 years depend on how carefully we define "winning".
xg15 6 hours ago [-]
> make_more_likely is, of course, doing a heroic amount of work here.
Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself.
> it is trained to predict next tokens as they occur in its training data.
What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data.
Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network.
This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.
kingjimmy 3 hours ago [-]
The chess analogy is not great since there is a defined heuristic in chess for "winning" or "optimal board state". A system doesn't need pretraining if they can fit the rule.
danielmarkbruce 6 hours ago [-]
The biggest problem is the word "predictor". Once you get into post training with RLHF and RLVR, it simply isn't doing that. It is not predicting anything. It's producing tokens, but it isn't predicting them. The chess analogy in the post is a good one - it's closer to searching for a set of moves that give a result than predict. It's search for a set of ideas, represented as locations in very high dimensional space, that when put together in the right order lead to a result.
Dlemlo 6 hours ago [-]
It's the fitness function: Make a model which is capable of predicting the next token. The next token of what? EVERYTHING.
So what does this lead to? To a generic intelligence which is capable of responding/answering everything.
If overfitted, the model just remembers every possibility in the world but this is not possible anyway so it will start to identify patterns and rules and will use them instead.
Basically 'compressing' every possibility to every question someone could ask -> compression leads to intelligence.
miraculixx 2 hours ago [-]
I like to think of LLMs as informed dice throwing
Sprotch 6 hours ago [-]
I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?
epistasis 6 hours ago [-]
The LLM produces a probability distribution over the likelihood of all possible next tokens. So whatever the tokens are, "ch", "ex", etc. the next one gets a probability.
During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":
Thanks - that makes sense. On that basis the article’s thesis is totally wrong - it would be like a computer program rating its ability based on how well it predicts moves played by grandmasters in the past. It’s not inventing new moves.
valleyer 6 hours ago [-]
During training, certain tokens are more likely to lead to a lower loss function value, which is how you "win" the game of LLM output.
mwkaufma 6 hours ago [-]
So, next-token predictors
DonHopkins 4 hours ago [-]
And any next-comment predictor could have predicted your totally unhelpful, uninsightful, and unoriginal comment.
You -- along with everyone else who keeps parroting this thought-stopping phrase and other tired cliches like "stochastic parrot", simply because you heard other people say them, without understanding what they really mean, which published research papers they came from, or what those and other papers actually argued -- are desperately clinging to a reductive, short-sighted, shallow, simplistic model like a drowning person clutching a concrete life preserver.
Seriously, we are trying to throw you a lifeline, and you are refusing even to participate in your own rescue. So squawk for yourself.
> The term "stochastic parrot" is a slogan masquerading as an explanation, only a shallow surface description of the mechanism, that totally fails to explain the phenomenon, or account for all that LLMs and language itself can do.
Here is the original 2021 paper that coined the phrase. It was not primarily an argument about consciousness, nor did its title constitute experimental proof that everything an LLM does can be explained as parroting. It was principally a position paper about the risks of increasingly large language models: environmental and financial costs, biases and hegemonic viewpoints inherited from poorly documented training data, unequal access and power, and the danger of people attributing meaning and accountability to synthetic text.
The paper did, however, make a strong theoretical claim: because an LM is trained on linguistic form without direct access to communicative intent, it cannot possess meaning, understanding, or a model of the world. The authors described it as "haphazardly stitching together sequences of linguistic forms" according to statistical regularities -- hence "a stochastic parrot."
That distinction matters. The popular slogan discards the paper's detailed analysis of actual risks while treating its most controversial theoretical premise as an established scientific result. It has escaped into pop culture as a drive-by anti-LLM slogan -- something people repeat instead of investigating what these systems represent internally, how post-training changes their behavior, or what they can actually do.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?"
Ironically, the objection that rhetoric was replacing scientific analysis appeared almost as soon as the phrase itself -- indeed, before the paper's formal publication. In January 2021, Michael Lissack published a response characterizing the draft as an advocacy piece that enumerated harms while leaving its assumptions, potential benefits, and cost-benefit trade-offs largely unexamined.
"The Slodderwetenschap (Sloppy Science) of Stochastic Parrots -- A Plea for Science to NOT Take the Route Advocated by Gebru and Bender"
>I don’t understand how it can know that a token is correct.
It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next token. Large models can be tricked to fall in the same behavior with very very specific inputs. Still happen, even in frontier models. And they can't detect if the output is wrong.
That's why the premise in TFA is wrong, because a transformer is a next-token predictor. It literally is that. There's nothing secret or magical, it's just a very mechanical process, with a lot of matrix multiplication, normalization, a few random passes, mappings between embeddings and a dictionary of tokens, in a very very high scale.
If someone has found something that's not a mechanical, algorithmic computation and llms are doing something nobody can explain and can't even be modeled in math, I'm happy to be educated.
major__4 6 hours ago [-]
classical particles obey deterministic state evolution rules, yet put 10 in a box and you cannot say where they will be 5 minutes later.
kgwgk 6 hours ago [-]
You cannot put classical particles anywhere.
major__4 5 hours ago [-]
[dead]
aeve890 5 hours ago [-]
Your point being? llms are software running in a fixed pipeline (barring variance induced by randomness in some layers). There's nothing like decoherence/thermal effects in a "lower level" that could induce stochastic behavior in a higher level. There's no Brownian motion in llms, if that's your analog with classical mechanics. You might argue emergent behavior that would look ordered/intelligent in some way and nature's full of examples of that but we don't attribute intelligence to physical processes.
chrisjj 6 hours ago [-]
It knows nothing of correctness or winning. It is predicting only what is most likely given its corpus.
hirvi74 6 hours ago [-]
My poor understanding is that an LLM does not "know" either. It basically uses probabilities to predict the next word based on a large matrix of probable outcomes.
For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"
It would respond with something like:
"The quick fox jumps over the lazy, brown [next word]"
(Assume all the words were previously guessed correctly at this point)
The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:
1. Dog (99.9% confidence)
2. Cow (85% confidence)
3. Bag (75% confidence)
4. Crayon (25% confidence)
The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.
Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.
However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.
My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.
I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.
Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).
Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.
4 hours ago [-]
polotics 6 hours ago [-]
yep "next-embedding" predictor is more correct, and not just at the end but through the layers, and folding back dimensions into that one next token is one small final step, and next-embedding could be named "next-meaning" as well, and we're getting there...
this sentence above would made a longer article if I bothered to so blog as is being blogged here
Geee 5 hours ago [-]
It's a next-token computer. It computes the probabilities for the next token.
gunalx 2 hours ago [-]
I mean. Fundamentally they are just autoregressive next token predictors. Fundamently fou can simplify to f(x) -> x+y where x is input tokens, y is the next token and f() is the model function.
Yes the model function is complex but still.
qudat 4 hours ago [-]
Shrug. My intuition is LLMs predict the new word based on a tensor vector space of patterns using arithmetic and similarity scores.
What’s not intuitive to me is that through pattern matching it’s able to express logic and reasoning.
DonHopkins 5 hours ago [-]
Calling an LLM a "next-token predictor" is like calling a TomTom a "next-turn predictor." It confuses the serial format of its instructions with the computation producing them, while ignoring the map, the route, the destination, and the goal -- as well as the people, businesses, traffic, and points of interest that make the map a model of an inhabited, changing world.
chrisjj 6 hours ago [-]
Better title: Continue thinking of LLMs as Next-Token Predictors
Because no, post training doesn't change that.
HarHarVeryFunny 3 hours ago [-]
I'm not sure that's a useful way to think of it.
RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine.
A base model is predicting training sample continuations (copying).
A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals.
The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
hirvi74 6 hours ago [-]
Sure, I get the gist of the article. I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. By that rational, the human brain is really not that much different. When I am having a conversation with another person, I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
infamia 5 hours ago [-]
> I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
hirvi74 3 hours ago [-]
> LLMs do not begin with any sort of concept they're trying to express.
Why do the need to? Considering they are merely tools, I actually appreciate they do not do this. A calculator can compute far better than any human, but I appreciate that calculators are not capable of expressing anything about the computations I request. I want the answer, not a conversation.
> LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
If you will allow me to be simplistic, people -- the soul, the self -- are predominately the aggregated effects of memories and experiences and the ability to retain new memories based on new experiences, no? Consider medical conditions in the dementia family of diseases. As memories fade into the ether, what remains of the self?
Also, people simulate/emulate each other all the time based on what an average, reasonable person might say. People incapable or unwilling to perform such mimicry are often labeled with all kinds of pejorative terms.
chrisjj 6 hours ago [-]
> I have never liked the reductionist argument that LLMs are nothing more than next-token predictors.
I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.
hirvi74 4 hours ago [-]
J.S. Bach said something similar about music and keyboard instruments.
> "There's nothing remarkable about it. All one has to do is hit the right keys at the right time and the instrument plays itself."
My issue is not with fact at face value. My issue is with how the fact is often contextually used in arguments to delegitimize and disparage LLM outputs and LLM users.
Yes, LLMs at a fundamental level are next-token predictors. But in my opinion, LLMs are very useful, imperfect next-token predictors.
There are a lot of wannabe John Henry [1] folks out there. Love LLMs or hate'em, most of those John Henry folks ain't beating these machines on a plethora of tasks.
It’s not wrong, but because LLMs are generators, and generation is a kind of prediction. And current mainstream models are autoregressive, which means they generate things one by one in order. But these trivia doesn’t tell us anything interesting about how they work or their limits.
It’s like saying a Boeing 777 is just a rotating machine, and it flies by just rotating some fins. Well yes, but no. With that level of simplification we’ve just ignored 150 tons of advanced engineering and physics. Similarly with token generator simplification we ignore a few trillion parameter Transformer. That transformer is more complex than a Boeing 777, and we don’t really know how it works.
A tiny ML model can do “next token prediction”. This is not as simple as that.
The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"
It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.
It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
It is predicting based on a model. In many cases we can download the model off hugging face. The model is conditioned by all sorts of things. Training data, post-training, coincidence, prompt inputs, runtime data available from whatever means.
> but at least I would still call it a "next token predictor"
We can call any prediction system a next token predictor. If you watch over the shoulder of a human writing a HN comment you are almost certain to see them generating a linear string of tokens. That is what keyboards do. It is impossible to generate text without being equivalent to a next token predictor.
The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on.
Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’.
And that isn’t really ‘prediction’, so much as ‘generation’.
It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function.
It’s not predicting the token, it’s predicting the reward.
But that’s the point: so is an LLM. Putting one token in front of another, hoping it’s doing the right thing to bring about the rewards it’s trained to… trying its best.
So yeah, not ‘next token predictors’. ‘Next token tryers’ maybe.
If you can’t grasp that logic gap then there’s no point discussing further.
No logic required, you can just build an LLM yourself, including post training. You'll see that predicting the next token isn't something the model does or is optimized for in RLHF or RLVR. You can hand wave all you like, but you have never done it.
Carry on good soldier.
That you tie yourself up in knots of fancy acronyms instead of plain words and that your argument boils down to semantics of the word prediction, it's pretty clear what is up brother.
A LLM already knows more tokens than the current one. It was mentioned in a blog post about how a LLM is doing haikus and co.
There are also structures in an LLM which allows it to 'estimate' numbers to a certain degree and doing other things.
Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
So "emits the next token" and "has no representation of anything beyond the next token" are two different claims. What autoregressive LLMs cost as a consequence of strictly outputting the next token is commitment. Once a token is output there's no going back. There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
> Prediction implies there is some "truth" or event or something that you can test against
there absolutely is a ground truth during training. the core predict-the-next-most-likely-token part of an LLM has a ground truth next-token. that's why you don't end up with generated text like: fish spurious send cattle chocolate phone happy meaning ball orange board canada.
> optimizes to predict the next token in training data
that is the optimization goal in training the next-most-likely-token core of an LLM, it basically translates to maximise the likelihood of predicting the next token x_i given the previous tokens
https://arxiv.org/pdf/2012.07805 (GPT2 but the point still stands)(edit: sorry for the ADHD edits)
If we suppose that the word "know" can sanely be applied to LLMs at all, then "A LLM already knows more tokens than the current one." seems to me like a perfectly reasonable restatement of that, and not any kind of misinterpretation.
> There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
Yes. There is no contradiction. Similarly, when humans speak, we surely have in mind the next few words we're going to say (or at least partial information about them), and may not realize the fault in them until after hearing ourselves utter them. But LLMs are not trained to output "excuse me, I mean…" sorts of things, because they're expected to output primarily as text (which might possibly then be fed to TTS).
Both systems have emergent behavior that goes well beyond what would naively be expected.
1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector.
2. What you see when you use an LLM is not next-token prediction directly next to the prompt, but instead following a block of varying length of next-token prediction that happened to make progress on the problem in your prompt, and which just summarizes the results.
I don't think anyone is doing that though; we know LLMs are not simple Markov chains, and that the prediction they make is based on more than the previous X words.
It's not minimising to describe even a complex prediction process as prediction.
The linked article makes a good point: a substantial chunk of the training does not consist of "here's a bunch of tokens, here's the next token, learn that." But all the comments want to turn it into a referendum on the goodness of AI.
What is emitted by a model during RLHF and RLVR is not, by structure, training or optimization, a prediction of the next token.
I guess the people in my camp find the "it's just a next token predictor" stupid in that it's like saying "it's just a bunch of carbon and hydrogen", but it's also one of those things where people like to think they are clever because they think they are theoretically correct. But they aren't even that. So it's like double stupid. But the "next token predictor" part is at least technically correct (like, carbon and hydrogen right) for pretraining, so the debate can't really be won there.
For example, I've noticed quite a few cases recently of LLMs outputting "but" where "and" would make more sense, or vice-versa. Surely that could be improved by such an approach?
'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that.
Those are mutations, not expansions of capability.
EDIT: gentler phrasing
What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.
So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.
And I think effectively the same distinction applies here.
There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.
So, it's not a next move predictor. It's a game result predictor.
I disagree that this accurately describes TFA.
(I would also agree with the parent commenter on that the writing smells like an LLM.)
The article is littered with both AI tells and admissions that 'next token prediction' is what is happening. Hence my description.
> I don’t think it’s a good reason to dismiss this
AI-generated prose reads as sending a 'lack of effort' signal to a lot of people, just as no editing at all does. ; It's an effective heuristic that we've all learnt in the last couple of years.
In either case, it's not always fair: there are people who deeply care about their ideas but forget to fix basic errors, or pass it through AI.
In both cases though, the advice is the same: if you want people to take your output seriously, you need to signal that you are taking it seriously. That used to mean editing for spelling and grammar. Now it means not using AI.
Personally I would never let an LLM touch my prose (although I'd happily use it for research and paraphrase things it told me), but if I force myself to consider the idea, that seems like the first thing I'd want. Maybe upon reading a diff you'd even consider going a third way with the text.
For example, you could cite specific things that you believe to be "AI tells" or "admissions".
> Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete.
The article is about how 'next-token predictor' is the wrong mental model; it opens with the admission that it is not the wrong mental model.
To say that a statement is incomplete, but not strictly speaking wrong, is perfectly compatible with describing it informally as "wrong" in the sense used in the title (i.e.: "not the most appropriate possibility").
There's a certain irony in pointing me towards the guidelines on the grounds that I have limited patience with your comments that violate them in various ways. I'm not sure that this is a productive discussion.
To summarize: yes, RLVR and other synthetic training methods exist! It’s still a next-token predictor, and it does not “learn” or “think” or “reason” in the human sense, like so many people seem to believe.
I fail to see how any possible output could cause either of us to change our mind.
i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
1st I say that "working forwards" in the sense of outputting one token at a time could be some form of prediction, I don't argue against that. This is what LLMs do at inference time.
2nd I say that to me what really constitutes a prediction is the pre-training. Here it's the classic setting for the word prediction in ML. The model outputs a prediction of the ground truth label: the next token.
3rd I argue that in RL there is no ground truth next token, so prediction doesn't apply here anymore.
Back to your question then: you're asking points 3 and 1 are different. Working backwards from a set of win states is basically what RL does in training. Working forward from the current state is what inference does. To me there is a distinction worth thinking about. First between the mechanism at inference time and at train time. Then between what happens in pre-training vs. RL post training.
Also known as predicting.
An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.
Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
It's not pedantry. The objective function changes. The optimization changes. THese are real things when training a model, not hand wavy philosophical ideas.
Literally no one here is claiming that it does. This is one of the many flaws in the article.
If I choose a specific move in chess, it's a choice. It's not a prediction. I might get a score 40 moves later given my choice, but I'm not predicting the next move.
To compare - during pre-training, the model literally tries to predict the next token (probabilistically), the training loop checks against the "right" answer, and the weights are updated based on that check. It's optimized to predict the next token.
Or starting with "yes". And this early locking in was a total lie, in the discussion that became "yes, might appear that way, but totally no since reasons". So it should have written "No", topmost, but could not self-edit that.
But nice that this gives it a more nuanced view, I might have to update my priors.
To be fair, is there anyone who hasn't been "reading too much Claudish recently" who is also qualified to write on the topic?
The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context.
You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.
That there’s a legion of LLM nerds arguing deterministic this, pretraining & rewards that all the better for the con job they’re pulling off.
The technology will be relegated to the trash bin of history, just like crypto.
The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.
None of this AI political economy stuff was relevant in 2015 because necessary technological breakthroughs had not yet been made.
With reinforcement learning and probably attention and other tricks that affect the weights based on things that aren't strictly in the training data, so the argument goes, you can end up with tokens following strings of tokens that would not be possible to be output with the training data and original weights alone. So describing it as solely a next-token predictor is incorrect based on this framing of it.
But that's just my take on this, I'm still trying to wrap my head around it all.
There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output.
The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus.
More concretely, in reinforcement learning there are no ground-truth next tokens to predict.
In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that.
But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.
Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.
But there is no trillion dollar industry around cheap top Markov models. So there must be something about LLM tokens that makes them more valuable than those generated from a simple Markov chain. And that substance, that makes one valuable and the other not, is exactly what reduction to "next-token predictors" masks.
But when we talk about humans, we're not talking about the chemicals involved in those humans.
When we talk about LLMs, the tokens are the valuable thing they produce for us. We want LLMs because they give us sequences of tokens.
To be clear, all life is a next state of the local world predictor. What makes humans somewhat unique among life is we're much better at predicting states of the world neither we nor any of our ancestors have ever experienced, for various reasons such as having the ability to legibly communicate very complicated information strings to each other, being able to build and use tools to record states of the world we can't directly sense.
Similarly, what makes LLMs and multimodal versions of the same architectures "better" than previous generations of electronic predictive models is factors like being able to read and understand roughly the same corpus of data humans have been recording all these millennia, being able to read and remember much more of it than any individual human, and being better at generalizing than other electronic predictive models, but not better than humans. And, of course, they can produce far more predictions in far less time. Frankly, that is probably the key advantage that makes the Hacker News crowd love them so much. They're not any better at predicting byte strings that can be compiled or interpreted into executable code than humans are if you gave both infinite time to do it, but they're a lot faster.
Both examples involve the same "aha" moment: even though it's true that you are literally 'just' doing XYZ, unbelievably complex patterns and sub-goals can emerge.
I don't think you intended this, but the word choice here gave me a chortle.
1. Useful work that has been done (the previously generated token sequence :: the mechanical work already accomplished)
2. The role of structure in relation to the application (post-training :: other components like crankshaft etc)
The output of the LLM is literally a probability distribution of what the most likely next token is.
It kind of does, though. In a gasoline engine you need to spark the combustion in advance of the piston reaching top dead-center to ignite the fuel early enough that it is able to provide downward pressure on the piston as it rolls over top dead-center. The amount of advance required changes with RPM, fuel octane, etc.
Start of delivery timing in a diesel is similar. You have to do it sufficiently far in advance to account for compressibility of the injection lines, fuel burn rate, etc as a function of RPM. A mechanical governor on an injection pump has a timing advance device built in. Electronically governed injection pumps, or modern common rail systems, do that in software.
So mechanically, engines kind of "predict" the next combustion event. Even moreso when you consider a modern ECU, which may be working at nanosecond resolution to time multiple injection events per cycle. To do this at such a resolution it will have to send signals to components based on a predictive model derived from "past" sensor data. E.g. it needs to act ahead of time to account for electrical and mechanical delays in the system.
Recursively invoked.
With carefully selected context.
And massive investment in RL to tune token selection.
And the ability to use cli tools on other folks' machines.
That's a powerful system built around a conceptually simple technology: Next token predictors.
It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.
The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).
FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.
It's not. "Brains as electrical circuits" is a gross simplification based on our ignorance and prejudices. (In the 18th century they spoke of brains as "clockwork mechanisms".)
LLMs, in contrast, are literally next token predictors. We know exactly how LLMs work, and they are exactly that.
I think you imply a rather loose standard for "exactly" here. I wouldn't even say this of major deterministic software projects that are orders of magnitude smaller than frontier LLM weight-dumps. In principle we could work our way through these systems eventually, sure, maybe even a single person could do so. But if we really understand exactly how our software works, how have we been tolerating bugs that lay dormant for years before being discovered by AI-assisted processes?
This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?
That seems more outrageous a claim than the one you're dismissing.
When I say there is some issue with people claiming there is some fundamental limit on the capacities of LLM's, I don't mean to say "If you think that they don't have unlimited potential you are wrong", I mean "you can't use the architecture of the transformer to make a sweeping declaration of things LLM's can or cannot do without empirical evidence, because the empirical evidence has unearthed far more surprising revelations than a reductive theory has been able to"
A pure next-token language model won't be able to give detailed instructions to an ensemble of motors, mimicking a human body, to do a wide variety of tasks our human brain is excellent at doing, for example, inserting keys into a car, opening the door, sitting down, starting the car, putting the car in reverse, and exit a parking lot, being careful not to hit anything.
They still can't do code accurately. The fact that you use this as a defense of your position greatly undermines the credibility of your claim.
- we get novel, emergent properties and capabilities of these models that were not trained
- they have very clear generalization to out of domain problems
The point is people conflate the end product: a model that can clearly do very novel, useful and interesting things, with the vehicle for getting there which is a series of optimization steps involving next token prediction loss.
You mention limitations; we all clearly know the practical limitations of these models today, but if you look at scaling laws and empirical performance trends (epoch capability index for example) as well as the trajectory over the last couple of years (very stable), the claim that there is some sort of fundamental limitation is now surprisingly the claim that has the burden of proof.
You can claim it may be e.g. finite context. That is fundamentally bad for certain classes of tasks. This was the hypothesis of a lot of lab leadership of urgently trying to anticipate how to get around this bottleneck (still of course lots of work on this) but the surprising thing is it does not appear to be at this point a blocker.
Exactly. We're dancing around the real argument: there's massive amounts of influencing going on (and not only about AI.)
No amount of cope and anthropomorphizing is gonna change that cold, hard fact.
P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.
Sounds like the next 1000 years depend on how carefully we define "winning".
Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself.
> it is trained to predict next tokens as they occur in its training data.
What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data.
Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network.
This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.
So what does this lead to? To a generic intelligence which is capable of responding/answering everything.
If overfitted, the model just remembers every possibility in the world but this is not possible anyway so it will start to identify patterns and rules and will use them instead.
Basically 'compressing' every possibility to every question someone could ask -> compression leads to intelligence.
During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":
https://youtu.be/GlYgs6v2YfU?is=IxVMhoCCE4N4WRVK
(Start at 15:30 for the LLM specific parts)
You -- along with everyone else who keeps parroting this thought-stopping phrase and other tired cliches like "stochastic parrot", simply because you heard other people say them, without understanding what they really mean, which published research papers they came from, or what those and other papers actually argued -- are desperately clinging to a reductive, short-sighted, shallow, simplistic model like a drowning person clutching a concrete life preserver.
Seriously, we are trying to throw you a lifeline, and you are refusing even to participate in your own rescue. So squawk for yourself.
https://news.ycombinator.com/item?id=48395727
> The term "stochastic parrot" is a slogan masquerading as an explanation, only a shallow surface description of the mechanism, that totally fails to explain the phenomenon, or account for all that LLMs and language itself can do.
Here is the original 2021 paper that coined the phrase. It was not primarily an argument about consciousness, nor did its title constitute experimental proof that everything an LLM does can be explained as parroting. It was principally a position paper about the risks of increasingly large language models: environmental and financial costs, biases and hegemonic viewpoints inherited from poorly documented training data, unequal access and power, and the danger of people attributing meaning and accountability to synthetic text.
The paper did, however, make a strong theoretical claim: because an LM is trained on linguistic form without direct access to communicative intent, it cannot possess meaning, understanding, or a model of the world. The authors described it as "haphazardly stitching together sequences of linguistic forms" according to statistical regularities -- hence "a stochastic parrot."
That distinction matters. The popular slogan discards the paper's detailed analysis of actual risks while treating its most controversial theoretical premise as an established scientific result. It has escaped into pop culture as a drive-by anti-LLM slogan -- something people repeat instead of investigating what these systems represent internally, how post-training changes their behavior, or what they can actually do.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?"
https://s10251.pcdn.co/pdf/2021-bender-parrots.pdf
Ironically, the objection that rhetoric was replacing scientific analysis appeared almost as soon as the phrase itself -- indeed, before the paper's formal publication. In January 2021, Michael Lissack published a response characterizing the draft as an advocacy piece that enumerated harms while leaving its assumptions, potential benefits, and cost-benefit trade-offs largely unexamined.
"The Slodderwetenschap (Sloppy Science) of Stochastic Parrots -- A Plea for Science to NOT Take the Route Advocated by Gebru and Bender"
https://arxiv.org/abs/2101.10098
It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next token. Large models can be tricked to fall in the same behavior with very very specific inputs. Still happen, even in frontier models. And they can't detect if the output is wrong.
That's why the premise in TFA is wrong, because a transformer is a next-token predictor. It literally is that. There's nothing secret or magical, it's just a very mechanical process, with a lot of matrix multiplication, normalization, a few random passes, mappings between embeddings and a dictionary of tokens, in a very very high scale.
If someone has found something that's not a mechanical, algorithmic computation and llms are doing something nobody can explain and can't even be modeled in math, I'm happy to be educated.
For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"
It would respond with something like:
"The quick fox jumps over the lazy, brown [next word]"
(Assume all the words were previously guessed correctly at this point)
The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:
1. Dog (99.9% confidence) 2. Cow (85% confidence) 3. Bag (75% confidence) 4. Crayon (25% confidence)
The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.
Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.
However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.
My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.
I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.
Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).
Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.
this sentence above would made a longer article if I bothered to so blog as is being blogged here
What’s not intuitive to me is that through pattern matching it’s able to express logic and reasoning.
Because no, post training doesn't change that.
RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine.
A base model is predicting training sample continuations (copying).
A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals.
The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
Why do the need to? Considering they are merely tools, I actually appreciate they do not do this. A calculator can compute far better than any human, but I appreciate that calculators are not capable of expressing anything about the computations I request. I want the answer, not a conversation.
> LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
If you will allow me to be simplistic, people -- the soul, the self -- are predominately the aggregated effects of memories and experiences and the ability to retain new memories based on new experiences, no? Consider medical conditions in the dementia family of diseases. As memories fade into the ether, what remains of the self?
Also, people simulate/emulate each other all the time based on what an average, reasonable person might say. People incapable or unwilling to perform such mimicry are often labeled with all kinds of pejorative terms.
I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.
> "There's nothing remarkable about it. All one has to do is hit the right keys at the right time and the instrument plays itself."
My issue is not with fact at face value. My issue is with how the fact is often contextually used in arguments to delegitimize and disparage LLM outputs and LLM users.
Yes, LLMs at a fundamental level are next-token predictors. But in my opinion, LLMs are very useful, imperfect next-token predictors.
There are a lot of wannabe John Henry [1] folks out there. Love LLMs or hate'em, most of those John Henry folks ain't beating these machines on a plethora of tasks.
[1] For those unaware, https://en.wikipedia.org/wiki/John_Henry_(folklore)