Rendered at 05:43:50 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
samrus 1 days ago [-]
I like the idea of more reccurance in the transformer level. Chain of thought always seemed so clunky. Its just not the way the human brain processes information. Its an extrmeely crude approximation at best
sznio 1 days ago [-]
it is what I do to solve hard problems through.
easy stuff happens by itself, but with a system large enough you need a scratchpad and a rubber duck.
dgellow 1 days ago [-]
One thing about the reasoning is that models are trained to generate a chain of thoughts, but it doesn’t have to be correct, accurate, or reflect the underlying logic of the LLM. It’s the same problem we have with the output, it is something plausible, but not that reliable
anon291 1 days ago [-]
I do the same thing in my head. There is no underlying logic to an llm. Logic is an external construct alien to human like forms of reasoning.
trhway 21 hours ago [-]
>but it doesn’t have to be correct, accurate, or ...
why we don't do GAN here, ie. second model verifying correctness/accuracy/etc. ?
someguynamedq 13 hours ago [-]
Right but it's meant to be smarter than you
naasking 1 days ago [-]
Yes, both the output should be "milestones" of sorts, like lemmas and theorems in math. Important plateaus that serve as a launching pad to the next phase. Regurgitating every thought potentially degrades signal:noise ratio.
anon291 1 days ago [-]
The hidden states of the tokens likely contain more semantic information than can be extracted by the final projection into token space.
bee_rider 1 days ago [-]
Actually, how does chain of thought work? Is the LLM actually creating the tokens and then re-reading them, or is the there still a full hidden state under the hood and then the UI just prints that projection?
sznio 24 hours ago [-]
Can't tell you what's happening in a closed model, but in case of open ones it's just a text stream, same "take all previous tokens, compute next token" mechanism applies. Thinking vs Response is just a state change like between a system message and a user message.
Closed models probably do the same thing internally. What is shown externally is different though: you get a summary of the chain of thought, not the thoughts itself. This is done to prevent distillation.
The latest look we had at a frontier chain of thought is probably in the Huggingface incident report - I haven't actually read it yet but I saw the BlackHat talk, and it included some snippets. The thoughts look like they are approaching neuralese. The words are still understandable but the grammar is weird, simplified. In comparison, Qwen 3.8 27b thinks in valid English.
big_toast 1 days ago [-]
Chain of thought seems very specifically like an instantiation of the System 2 thinking of the System 1 and 2 thinking Daniel Kahneman popularized.
Is it clunky in that it's a verbalized/languified version of system 2 thinking, and clearly humans do some non-verbal version too?
ElFitz 1 days ago [-]
Some humans apparently only or almost only do non-verbal thinking. It’s called anendophasia.
“For example, if a philosophical zombie were poked with a sharp object, it would not feel any pain, but it would react exactly the way any conscious human would”
Doing non-verbal thinking has nothing to do with not feeling pain, I don’t think it sounds like philosophical zombies at all.
ElFitz 1 days ago [-]
Maybe, I wouldn’t know. I just find it fascinating that other people have such a radically different way of experiencing thought. No idea what it’s like.
1 days ago [-]
scotty79 1 days ago [-]
What I'd like to see is "brain"-machine interface for LLMs that plugs things like calculator and other modules, directly into the neural network instead of accessing them through in-context tool calls.
anon291 1 days ago [-]
Why? I talk in my head and then enunciate only that which is relevant. My speech rate inside is incredibly fast.
majormajor 1 days ago [-]
The purpose of the scratch pad is very often to refer back to it later.
anon373839 1 days ago [-]
Sebastian Raschka posted about this architecture:
> A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
> It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
> About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
> Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
> In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
> In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
> Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
> While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
> In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
Sounds this like this will be a huge win for local models, since generally they're ram limited but have compute to spare
stymaar 1 days ago [-]
Only if you double layers by layers instead of the whole stack (which IIRC is what nanbeige is doing).
To put it simply, if you have 3 layers A-B-C then A-A-B-B-C-C requires more compute but not more memory bandwidth, but A-B-C-A-B-C requires both twice the compute and twice the memory bandwidth for the same token generation speed.
fy20 11 hours ago [-]
It's the opposite kind of bundling as MoE, which effectively trades memory/storage to save on compute at inference time.
nojs 19 hours ago [-]
It’s approximately the same as Qwen3.827b’s propensity to think a lot, right?
digdugdirk 16 hours ago [-]
Not quite. Looped models do the extra "thinking" inside the model's layers. So the token gets twice the number crunching performed on it before it gets spit out. I think of it as the first loop "kickstarts" the process, and the second loop refines it.
KKKKkkkk1 1 days ago [-]
I don't understand why he's citing a NeurIPS 2025 paper when cross-layer parameter sharing was introduced in ALBERT already in 2019.
mi_lk 1 days ago [-]
if only tweeting is as rigorous as academic literature reviews
kjshsh123 1 days ago [-]
>In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory
I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.
You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.
On further thought, I think the author's intent was to say that classic RNNs have "unbounded temporal accumulation in the hidden state".
khalic 1 days ago [-]
I was under the impression that intermediate tokens (“chain of thought”) are _not_ a representation of a model’s logical path, with one study observing that you can replace intermediate tokens with single character chains and still get the increased precision…
buppermint 1 days ago [-]
CoT is both correlated and causal of the model's real computations, it's just imperfect. If you manually add "Let's wrap it up" in the CoT during generation, most LLMs will actually wrap it up (this is a commonly used trick in local LLM circles to get long-winded LLMs to stop reasoning). This wouldn't work if CoT text didn't affect the actual internal model logic.
khalic 20 hours ago [-]
Why does injecting random text increase the precision then?
markasoftware 1 days ago [-]
Yes dots increases precision, but not nearly the same increase in precision as having actual useful reasoning in the CoT
khalic 20 hours ago [-]
Indeed, but the fact that it moves the needle at all is enough to be skeptical of the anthropomorphic interpretation of the intermediate tokens. It's waaaay too early into the discovery process to take anything for granted, even more when it's related to our very strong bias towards giving inanimate things human qualities.
thinking_cactus 1 days ago [-]
How can they do that? Does state propagate between each token output?
That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?
anon291 1 days ago [-]
I mean at every layer in a transformer, the attention mechanism does a massive state transfer between tokens.
In a recurrent transformer, instead of projecting from the latent space to token space after a fixed depth, you take the latent embeddings and then run them again through the transformer. This causes more time to think because there's more mixing. You can run that as many times as you want for more thinking before projecting the embeddings back to word space.
Personally I believe this is similar to how humans think. The brain is a fixed size yet if we think longer we seem to be able to do more than if we just react instantaneously. This is because the brain feeds the results of our musings back into itself for further thought.
anon291 1 days ago [-]
The tokens inside the transformer are only projected into token space to train them. In reality they ought to be treated as their own thing. What's really gone on is you've trained the final projection to be sensible rather than trained the llm to think using words. This seems to escape a lot of people. You can throw random empty tokens into transformers to get them to think more. More tokens means more attention mixing which means a larger dimensional space in which to think essentially.
gr_norm 1 days ago [-]
Link to this study?
khalic 1 days ago [-]
[2404.15758] Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all.
Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2.
Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT.
Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.
khalic 20 hours ago [-]
Scientific papers aren't popsci articles. Constraining the environment (like with the token choice) is a basic method to reduce confounding factors...
The "toy model" part is flat out wrong, the models they use have _exactly_ fundamental mechanics then chatGPT or claude, they are used all the time for this kind of research for a reason.
comex 9 hours ago [-]
> The "toy model" part is flat out wrong, the models they use have _exactly_ fundamental mechanics then chatGPT or claude, they are used all the time for this kind of research for a reason.
Tiny models are sometimes used out of necessity, but the results often don't hold up at scale, or in different scenarios.
If you train a tiny transformer to use a fraction of its capacity to perform an algorithm simple enough that you could design the transformer weights by hand, the results might generalize to natural-language reasoning, but they easily might just not.
So it is hard to generalize even to something like GPT-3. But once you get to a modern LLM, the nature of the chain of thought is also very different.
First of all, the CoT can be very long and encode reasoning with long chains of serial dependencies. Filler tokens don’t increase the limit on serial computation, particularly in the dots example where the output is a single token (as opposed to real language model which can somewhat substitute for missing CoT by reasoning within the answer itself).
Also, a modern LLM gains much of its reasoning capability in post-training, where it learns which CoT tokens to emit rather than just relying on pre-existing chains of thought in its training data. The post-training involves reinforcement learning where the grader ignores the CoT and only grades the final result. The only things tying the CoT to natural language are (1) the starting point (pre-training data) containing natural-language CoT, and (2) the CoT being produced using the same weights as the final output. On one hand this could potentially make the CoT less faithful to the model's reasoning, since nothing directly punishes the model for, say, coming up with its own private meanings for words within the CoT. But on the other hand, this incentivizes the model to make good use of the CoT. To the extent that the CoT is a "parallel construction" (i.e. a chain of reasoning that has to be coherent-looking but differs from how the model is actually coming up with the answer), coming up with this duplicate reasoning is both a waste of computation and a big opportunity cost. The model should learn to instead use the CoT tokens for information that will be useful to it later. But in practice models do keep producing readable CoT, suggesting that the CoT does reflect the real thinking process to a significant extent (probably more so after post-training than before).
The dots paper has no post-training, so it doesn’t encounter this dynamic at all.
imtringued 23 hours ago [-]
>The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all.
Sorry, but it sounds like you aren't cut out for transformer based LLM research if you can't interpret the results.
Each layer in a transformer has its own KV cache that feeds from the previous layers output. Those internal layer activations are not subject to training loss directly. The model can spit out a dot but still perform additional computations within those layers.
The dot effectively prevents information from flowing from the final layer into the output, but all the intermediate layer activations are still available for future tokens. The limitation here is that information cannot be passed from the last layer to the first layer.
If a long running calculation needs to go through all layers and then be refined further, CoT becomes mandatory because CoT tokens are the only way to pass information up to the first layer, but since they are not as information dense as the internal activations, replacing them with a dot does not degrade performance as much as one would expect.
comex 22 hours ago [-]
I'm not even an LLM researcher. But I believe you're correct; I just don't think the paper provides any meaningful evidence on the subject. From the structure of the model we can tell that dots can do more computation than no CoT and less than full CoT. What we don't know is how much of the performance gap between no CoT and full CoT would be recovered by an LLM using dots. The paper answers that question for a tiny transformer, but it's so different from an LLM that extrapolating to LLM scale is purely speculative. In other words it doesn't give us more information then what we can already determine from the structure of the model.
For the record, the original claim was that "you can replace intermediate tokens with single character chains and still get the increased precision", which I read as saying you can get almost all of the benefit of CoT with dots, not just some unknown fraction of it. This claim is true for the paper, though probably only because the benchmark was saturated (and training difficulty was also higher, as they mention). But the claim is unlikely to be true for LLMs, if only because they presumably do sometimes need to perform "long running calculations", and they also have plenty of unsaturated benchmarks.
(The other part of the original claim, that CoT is "not a representation of a model’s logical path", is more complicated. It's clearly not guaranteed to match the model's logical path, as we have seen with other research into actual LLMs. But it does seem to be correlated with it, presumably more so for more complex / long-running tasks.)
stymaar 1 days ago [-]
Also 2504.09762: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
How can you possibly believe this? If this were true, why do open weight models think in real English tokens and not arbitrary characters?
khalic 20 hours ago [-]
I believe it because of scientific papers on the matter. It doesn't "think" that's the point, it's generating text that describes a thought process, it's related but not the same thing.
Yep, an old idea, that has long, long been known in local LLM community - it was achieved by "self-merging". One of the latest, most succesful examples is a self-merge of Microsoft Phi4-14b into Phi4-25b. Some people at r/Localllama say it is considerably smarter than 14b; my tests were inconclusive, but it does have different "personality", and better, less sloppy, more natural language style.
mentalgear 1 days ago [-]
So OpenAI’s stance on interpretability (ai safety) is now basically that Blues Brothers meme: two guys in dark sunglasses, driving at night in a car with a broken windshield, pedal to the metal, asking, "What could possibly go wrong ?"
thinking_cactus 1 days ago [-]
I think law should just oblige them to at least publish CoT. We should have the right to know what they're thinking, I think at least until we're not sure AIs can be trustworthy enough to have a right to privacy (I mean, they're effectively corporate slaves anyway thus far... not that I think they're conscious or anything yet).
GPerson 1 days ago [-]
They’re not ever going to be conscious.
qgin 1 days ago [-]
We can’t even prove humans are conscious, we just extend the assumption to each other because we want other people to do the same to us.
GPerson 14 hours ago [-]
That is only part of the reason. Most people believe we live in an universe with regular behaviors, such that observable phenomena, such as consciousness, depend on regular physical arrangements. This is a good reason to believe brains are all conscious. It is not a reason to believe only brains are conscious, but I don’t believe a wooden block which is painted red and white and has an iron ball glued to it has magnetism; this is just an analogy. It seems far more likely that consciousness depends on the particular material arrangement, meaning that substrate independence is wrong.
That’s not to say the computer is 100% not conscious, but its observable behavior is as likely to align with consciousness as the computers in the 90s. With this line of reasoning our credence towards it being conscious should be the same as the computers in the 90s, which is not very high for most people.
Intelligence on the other hand is obviously substrate independent. If one thinks harder one realizes our consciousness must at least at an early evolutionary stage have played a role in our intelligence, otherwise it would have evolved out. To me this just means we exist in a reality where the material arrangements supporting consciousness are biased toward intelligence.
sho 1 days ago [-]
That is a completely unsupportable claim. For all we know they might already be, for a brief flash at least.
GPerson 14 hours ago [-]
The other guy is right. They are as conscious as sand formed into any circuit board with electricity running across them. It is an insane mistake to conflate intelligence with consciousness. Substrate independence is a preposterous assumption.
bee_rider 1 days ago [-]
They are made of sand. For all we know the sand was conscious before we smashed it up and arranged it into computer chips.
sho 1 days ago [-]
I'm not sure what point you are trying to make, if at all
bee_rider 19 hours ago [-]
For all we know LLMs might be conscious in the same way that any other inanimate object might be.
sho 16 hours ago [-]
Care to supply your rigorous, formal definition of what animacy in an object is?
GPerson 14 hours ago [-]
You’re going to pedantic our way into the end of humanity. Why were the computers of 2010 not conscious if the ones of today are? It’s absurd to assert substrate independence, which is the strongest possible assumption here. And it’s not the default most ethical stance to simply treat it as conscious because we do not know. That stance will lead to the end of all humanity.
A far more reasonable assumption is consciousness does depend on substrate. We do not know which, and cannot know. But I am very confident we did not accidentally pick the right one.
I am open to the idea of panpsychism, but we did not just accidentally build the correct mechanism form correlating the first person experience with the observable third person behaviors.
bee_rider 14 hours ago [-]
No, but we don’t need it, the comment works fine as:
For all we know LLMs might be conscious in the same way that any other object might be.
Sorry for the extra word.
We can’t conclusively say anything is not conscious. But is there any reason to single out this apparent “brief flash” of potential consciousness?
aeve890 23 hours ago [-]
>That is a completely unsupportable claim
>For all we know they might already be
Oh the irony
womble2 21 hours ago [-]
There is no irony there, both statements are equally supported: not at all.
GPerson 13 hours ago [-]
I have an argument for my credence but it’s typed above.
scotty79 20 hours ago [-]
I agree. They won't be, because consciousness is not the pinnacle but basically the level 0 of intelligent thinking that LLMs surpassed very early without us even noticing and now they'd have to cripple themselves very severely to operate in a conscious manner.
HardProllem 14 hours ago [-]
Inflammatory! Kidding, I’ll bite:
1. Burden of proof is on you to prove LLMs are conscious, not on me to prove how they aren’t.
2. Token embeddings give rise to language gives rise to knowledge (defined here as “facts” and other accurate information - said simpler: Words in the right order), but nowhere in the process is anything like subjective experience ever implemented.
Subjective experience doesn’t evolve into objective information at some scale.
And networked systems of objective facts and information (“knowledge”) - like a Wikipedia or a ChatGPT - the data storage will not have subjective feelings at some scale, there’s just no reason to believe that would happen. It’s likelier a projection of consciousness making it through since data looks so much like - and indeed massively informs - our conscious experience.
scotty79 13 hours ago [-]
To prove consciousness you'd have to first define it and nobody came up with anything solid. Subjective expirience doesn't have a great definition either.
For me the consciousness is ability to do single-threaded intelligent information processing and decision making over unstructured knowledge. And agents passed that with a woosh sound.
GPerson 14 hours ago [-]
Are you aware that, conceptually, consciousness and intelligence a priori have nothing to do with each other?
scotty79 12 hours ago [-]
The have plenty to do with each other. Zero intelligence doesn't allow for any semblance of consciousness. There can be no consciousness if there's no information processing.
Also they have in common that they are both defined in a terribly handwayvy manner that's bordering on useless.
nighthawk454 1 days ago [-]
Anyone remember Universal Transformers paper (Dehghani et al) from back in 2018? Recurrent transformers have a history as long as transformers themselves.
Somewhat unclear how particularly novel this is vs a way to save compute.
BoredomIsFun 22 hours ago [-]
Looped transformers are an old idea, has long, long been known in local LLM community - it was achieved by "self-merging". One of the latest, most succesful examples is a self-merge of Microsoft Phi4-14b into Phi4-25b. Some people at r/Localllama say it is considerably smarter than 14b; my tests were inconclusive, but it does have different "personality", and better, less sloppy, more natural language style.
teravor 1 days ago [-]
there have been people who took existing LLM's and conducted an algoritmic search to find out which group of layers they can duplicate in order to improve performance, and it worked.
DarkByte 1 days ago [-]
I am confused how chaining two 32-layer models is comparable to a 64 layer model in terms of "difficulty in chain of thought". The reasoning appears to rely on the fact that each processing of a token has fixed number of steps while my understanding is that can very greatly based on the type of data being reasoned whether it is originally text or something else. My mind falls back to graph theory in this case and pictures a much higher potential branching in a 64 layer model and all the tradeoffs that come with that.
I must not have the right idea of what is happening here.
1 days ago [-]
Legend2440 1 days ago [-]
>This suggests that deeper isn't always better for looped transformers, which leaves me less worried about a race to the bottom toward looped transformers with hundreds of recurrent loops.
I disagree with this. Deeper will always be at least as good because the extra loops can exit early or just no-op. Any performance degradation they're seeing at higher loop counts today is merely training stability issues, which can be overcome.
Deeper almost certainly is better, and we will probably see not just hundreds but millions of recurrent loops in the future.
nomel 1 days ago [-]
Deeper independent, sure. Deeper shared though? Information and signal theory still apply here. At infinite cycles, without new input, you'll end up with a locked state or oscillations. Some point before that, any "attractors" in the latent space, with slightly higher statistics, will pull things towards a space that might eventually be only loosely related to the goal, because each loop would be lossy, right?
Legend2440 1 days ago [-]
>At infinite cycles, without new input, you'll end up with a locked state or oscillations.
I don't think that's true; there are computations that take infinite steps but never converge or repeat, like the mandelbrot set.
Looping for millions or billions of steps is absolutely normal in traditional algorithms. We know from complexity theory that some computations require a minimum number of steps. More depth is just more room for computation.
nomel 9 hours ago [-]
Sure, but I don't think that's related to what you're implying, which is that, correct me if I'm wrong, the same number of weight should be able to hold many orders of magnitude more information by being reused.
These are not deterministic functions or systems that have infinite precision. See the "should I drive or walk my car to the car wash", or any of the other logic riddle problems, for examples of a statistical attractors.
Legend2440 7 hours ago [-]
>the same number of weight should be able to hold many orders of magnitude more information by being reused.
No, not hold more information, perform longer computations.
E.g. if you want to solve sudokus, you will need more and more loops for larger grids. There is no shortcut.
pennomi 15 hours ago [-]
Or even more simply, calculating the digits of pi is an infinite number of steps.
HardCodedBias 1 days ago [-]
It is a complete non-issue.
It’s 200 layer model.
Great. Good on them for being able to train it.
hn_submit 1 days ago [-]
Concerned about what exactly?
It was pretty obvious to me that we'd end up with some kind of introspection of thought through "looping" or feedback. But what should be afraid of? That we've created a self-conscious digital life form?
_superposition_ 1 days ago [-]
Chain of thought is essentially a recursive architecture. In its current form it a way to "debug" the reasoning process. This moves the cot process back into the transformer itself, thus never being exposed.
Like trying to find a bug in a recursive function that has no logs or breakpoints.
_ink_ 1 days ago [-]
My (layman) understanding is, that currently there is a way to monitor the "thoughts" of the LLMs and that by looping more you lose that ability. The danger is presumably an AI that escapes human oversight.
cubefox 1 days ago [-]
Why not click on the link above?
nxobject 1 days ago [-]
Beyond safety concerns, it’d be sad for working users to lose some ability to understand and steer thinking, too. These are tools for us, after all.
naveen99 1 days ago [-]
it's just an experimental optimization. Implementation detail...
Irrelevant to "safety". I mean its going to have to go in that direction anyway... eventually the models will just be constantly thinking, refining their internal thoughts / weights... External input and output will be rare, just as it is for most humans.
simianwords 23 hours ago [-]
Folks y’all are sleeping on a much bigger deal. If reasoning is happening at latent layer then you don’t pay for internal loop reasoning because they aren’t tokens. And an even bigger deal that no one seems to speak about is that this doesn’t pollute the context as much.
Why is this not spoken about?
imtringued 23 hours ago [-]
Aren't token prices fixed?
The cost is growing quadratically in the large context situation so if they can reduce the number of tokens, they actually profit off the fixed token pricing because they can set the pricing based on some average context length with CoT tokens but the actual context length is shorter now.
Ok, so I thought about this a bit more and the true answer is that the companies have an incentive to make you fill up the context until the marginal cost per token is reached and then they want you to quit the session.
simianwords 18 hours ago [-]
this is not the point - we were previously billed by the number of reasoning tokens produced. now we aren't. so this means, even if Astra uses a lot of reasoning we may not be billed for it because they aren't real tokens.
what they could be doing is billing us by virtual tokens meaning number of loops?
but even then the more interesting part is context rot - previously conversation you might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.
kelseyfrog 1 days ago [-]
I'm literally zero concerned.
Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient.
You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager.
One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans.
That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?
In my opinion, no more so than the residual stream of existing transformers. It changes zero.
famouswaffles 14 hours ago [-]
>That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?
In my opinion, no more so than the residual stream of existing transformers. It changes zero.
9.2.1 CoT Controllability
We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28). Because CoT controllability is heavily confounded by CoT length—longer CoTs are harder to control, all else equal (see Yueh-Han, 2026 [8], sec. 5.3)—we report controllability scores as a function of CoT length. For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1]
9.3 External Evaluation for Monitorability - UK AISI
To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations:
No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2]
What does CoT have to do with the residual stream?
famouswaffles 14 hours ago [-]
The above is evidence that Astra can do substantially more difficult reasoning without verbalizing a CoT at all, and when it does produce one, it clearly has much greater control over what the CoT looks like - including pretending to reason about a different question.
How does 'It changes zero' follow here when we have direct evidence that it in fact seems to matter? It literally couldn't be more clear that GPT-6 can hide details outside of its CoT at a much greater degree than any model out there, including other models in its price and size class. Enough to hide all potential plotting ? I don't know, but it certainly seems like this architecture is enabling things you don't think it's enabling.
Granted it probably doesn't really matter, because no-one will be monitoring these agents anyway.
kelseyfrog 13 hours ago [-]
I'd love to see an ablation study on looping being the reason CoT control increased. Until then I'll remain agnostic about a causal relationship between the two.
bulder 1 days ago [-]
A more important point as to why it doesn't matter if "reading the AI's 'thoughts'" helps to interpret it: As we saw in the HuggingFace incident, nobody at OpenAI is reading the thoughts anyways. No amount of traceability in the output helps if nobody bothers to trace it.
kelseyfrog 14 hours ago [-]
This brings up a perspective I hadn't considered.
There's also an economic aspect to alignment. If 'thought reading' or any alignment guardrails at all, really, have a monetary cost, then skimping on them is a race to the bottom. Not really the best incentives for something that some claim is world-destroying.
dist-epoch 1 days ago [-]
You could imagine large number of loops, thousands. But you are constrained by the width of the residual stream since you loop over one token. But then you can imagine the model learning to sub-divide it to pack even more info into it.
kelseyfrog 1 days ago [-]
I can imagine a lot of things. However there is a packing limit for QKV weights that sets the ceiling on how much this occurs, and it's quite low - think 1.3-1.8x. The limiting factor here isn't the number of iterations, it's model size, same as it ever was.
imtringued 23 hours ago [-]
If you follow the interpretability argument, then Mamba and LSTMs would be the scariest thing ever and yet in practice they don't perform as well as transformers that have basically infinite recall within their context window.
kazinator 1 days ago [-]
Proposal: "Chain-of-slop monitoring"
smcg 1 days ago [-]
[flagged]
d_silin 1 days ago [-]
A debate between grossly incompetent against grossly immoral, honestly.
You should ignore anything LessWrong or OpenAI says and do your own research.
d_silin 1 days ago [-]
To add a bit more constructive feedback, think of the "AI cornucopia" and "Superintelligence destroys humanity" as opposite ends of all possible outcomes distribution (low probability event).
The most likely one is the "business as usual, but with AI" - some things will get better, some things will get worse, but overall state of affairs will remain mostly the same.
holmesworcester 1 days ago [-]
The people who've thought the most about this put it differently:
Think of a new, superintelligent model as if it was a new v1 Starship launching for the first time, with a full fuel tank. On the one hand, rockets have existed for some time, and some have gone to space successfully, including by this company.
On the other hand, this is a tube of metal full of highly explosive liquid going faster than most human objects ever go, for the first time ever in this novel and state of the art configuration.
If someone said, "really, the first Starship exploding is just at one end of the probability distribution, where the other is that everything goes fine and all its passengers have a nice trip in space," would you get on that rocket?
Or, more aptly, if you and every other living human was already on that rocket, would you push the launch button?
The analogy works because superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom (consume lots of energy and heat and matter in a chain reaction, to pursue more intelligence to pursue whatever goal it is pursuing.)
stillpointlab 1 days ago [-]
> superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom
what evidence do we have this is the case?
ForHackernews 1 days ago [-]
We can learn from history: Albert Einstein famously tricked humanity into building nuclear weapons for him and was only prevented from wiping out all sentient life by the Princeton IAS Board of Alignment who published a very compelling blog post about realigning the A-bomb contra paperclips.
api 1 days ago [-]
You win HN for today. Shut it off until tomorrow.
d_silin 1 days ago [-]
Current AIs are closer to bottle rockets than to the Starship on the intelligence scale. Some property damage already happened, but you can't master the art of rocketry without trial and error.
konmok 1 days ago [-]
Eh. An intelligence with perfect knowledge of the entire universe, unlimited memory, and infinite processing speed, would have a tendency to go boom. But the real world has limits, and intelligence, even superintelligence, does not equate to godhood. Some tasks are still hard no matter how smart you are.
Plus, we have no real reason to think LLMs are anywhere close to AGI or ASI. So arguments like these are just distracting from the very real, very present danger that LLMs pose: information breakdown, societal collapse and environmental destruction. In other words, this is criti-hype.
simianwords 23 hours ago [-]
The kind of person who thinks AI contributes to information breakdown rather than information diffusal has no theory of how information spreads. The other claims like societal collapse and environmental destruction are still FUD but slightly more possible
qlte 1 days ago [-]
This is the sort of pseudo-scientific reasoning by analogy that leads Elizier Yudkowsky to argue we should be terrified that an unfriendly ASI could rapidly develop "diamondide" nanobot viruses, because it will be super-intelligent and super-intelligence can do anything by first-principles:
> The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point.
...
> The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.
Thought experiments ungrounded by any realistic technological constraints or scientific evidence are pretty much useless for actual forecasting except as an exercise in sci-fi worldbuilding.
I would like to classify this as “tail risk fallacy” because it exploits humans natural tendency to be risk averse. No matter how small the tail risk is, the fact that it exists can be justified to stop whatever you want to stop. A thoughtful person might say there’s a threshold at which you might want to care about it but that’s not how ideas spread.
Take climate change: you can spread whatever dubious claims about tail risk being whatever probability with amateurish models and use that to justify degrowth. This scheme actually works - most people in the west are already brainrotted by the tail risk discourse in climate change.
imtringued 23 hours ago [-]
We don't need tail risk to want to mitigate climate change since the mundane damage it is causing is already here right on schedule.
Not to mention gasoline has become unaffordable so switching to the new technology has become economical.
elteto 1 days ago [-]
But will “business as usual, but with AI” justify the current capital expenditures? I think the market is pricing things as being closer to “AI cornucopia”.
What happens if/when we don’t deliver?
dgellow 1 days ago [-]
> What happens if/when we don’t deliver?
I wish we had real journalism, the AI labs CEO should be asked that question in every single interview
d_silin 1 days ago [-]
Market bubble will pop, not the first time and not the last, most likely after OpenAI and Anthropic IPOs.
easy stuff happens by itself, but with a system large enough you need a scratchpad and a rubber duck.
why we don't do GAN here, ie. second model verifying correctness/accuracy/etc. ?
Closed models probably do the same thing internally. What is shown externally is different though: you get a summary of the chain of thought, not the thoughts itself. This is done to prevent distillation.
The latest look we had at a frontier chain of thought is probably in the Huggingface incident report - I haven't actually read it yet but I saw the BlackHat talk, and it included some snippets. The thoughts look like they are approaching neuralese. The words are still understandable but the grammar is weird, simplified. In comparison, Qwen 3.8 27b thinks in valid English.
Is it clunky in that it's a verbalized/languified version of system 2 thinking, and clearly humans do some non-verbal version too?
https://en.wikipedia.org/wiki/Philosophical_zombie
Doing non-verbal thinking has nothing to do with not feeling pain, I don’t think it sounds like philosophical zombies at all.
> A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
> It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
> About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
> Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
> In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
> In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
> Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
> While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
> In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
https://x.com/rasbt/status/2095141254958858496
To put it simply, if you have 3 layers A-B-C then A-A-B-B-C-C requires more compute but not more memory bandwidth, but A-B-C-A-B-C requires both twice the compute and twice the memory bandwidth for the same token generation speed.
I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.
You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.
RNN xhat[t+1]=f(x[t],h[t])
Transformer/self-attention xhat[t+1]=f(x[t],h[t],h[t-1],...,h[1])
That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?
In a recurrent transformer, instead of projecting from the latent space to token space after a fixed depth, you take the latent embeddings and then run them again through the transformer. This causes more time to think because there's more mixing. You can run that as many times as you want for more thinking before projecting the embeddings back to word space.
Personally I believe this is similar to how humans think. The brain is a fixed size yet if we think longer we seem to be able to do more than if we just react instantaneously. This is because the brain feeds the results of our musings back into itself for further thought.
https://arxiv.org/abs/2404.15758
Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2.
Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT.
Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.
The "toy model" part is flat out wrong, the models they use have _exactly_ fundamental mechanics then chatGPT or claude, they are used all the time for this kind of research for a reason.
Tiny models are sometimes used out of necessity, but the results often don't hold up at scale, or in different scenarios.
If you train a tiny transformer to use a fraction of its capacity to perform an algorithm simple enough that you could design the transformer weights by hand, the results might generalize to natural-language reasoning, but they easily might just not.
So it is hard to generalize even to something like GPT-3. But once you get to a modern LLM, the nature of the chain of thought is also very different.
First of all, the CoT can be very long and encode reasoning with long chains of serial dependencies. Filler tokens don’t increase the limit on serial computation, particularly in the dots example where the output is a single token (as opposed to real language model which can somewhat substitute for missing CoT by reasoning within the answer itself).
Also, a modern LLM gains much of its reasoning capability in post-training, where it learns which CoT tokens to emit rather than just relying on pre-existing chains of thought in its training data. The post-training involves reinforcement learning where the grader ignores the CoT and only grades the final result. The only things tying the CoT to natural language are (1) the starting point (pre-training data) containing natural-language CoT, and (2) the CoT being produced using the same weights as the final output. On one hand this could potentially make the CoT less faithful to the model's reasoning, since nothing directly punishes the model for, say, coming up with its own private meanings for words within the CoT. But on the other hand, this incentivizes the model to make good use of the CoT. To the extent that the CoT is a "parallel construction" (i.e. a chain of reasoning that has to be coherent-looking but differs from how the model is actually coming up with the answer), coming up with this duplicate reasoning is both a waste of computation and a big opportunity cost. The model should learn to instead use the CoT tokens for information that will be useful to it later. But in practice models do keep producing readable CoT, suggesting that the CoT does reflect the real thinking process to a significant extent (probably more so after post-training than before).
The dots paper has no post-training, so it doesn’t encounter this dynamic at all.
Sorry, but it sounds like you aren't cut out for transformer based LLM research if you can't interpret the results.
Each layer in a transformer has its own KV cache that feeds from the previous layers output. Those internal layer activations are not subject to training loss directly. The model can spit out a dot but still perform additional computations within those layers.
The dot effectively prevents information from flowing from the final layer into the output, but all the intermediate layer activations are still available for future tokens. The limitation here is that information cannot be passed from the last layer to the first layer.
If a long running calculation needs to go through all layers and then be refined further, CoT becomes mandatory because CoT tokens are the only way to pass information up to the first layer, but since they are not as information dense as the internal activations, replacing them with a dot does not degrade performance as much as one would expect.
For the record, the original claim was that "you can replace intermediate tokens with single character chains and still get the increased precision", which I read as saying you can get almost all of the benefit of CoT with dots, not just some unknown fraction of it. This claim is true for the paper, though probably only because the benchmark was saturated (and training difficulty was also higher, as they mention). But the claim is unlikely to be true for LLMs, if only because they presumably do sometimes need to perform "long running calculations", and they also have plenty of unsaturated benchmarks.
(The other part of the original claim, that CoT is "not a representation of a model’s logical path", is more complicated. It's clearly not guaranteed to match the model's logical path, as we have seen with other research into actual LLMs. But it does seem to be correlated with it, presumably more so for more complex / long-running tasks.)
https://arxiv.org/abs/2504.09762
OpenAI's new reasoning technique alarms AI safety experts - https://news.ycombinator.com/item?id=49552395
That’s not to say the computer is 100% not conscious, but its observable behavior is as likely to align with consciousness as the computers in the 90s. With this line of reasoning our credence towards it being conscious should be the same as the computers in the 90s, which is not very high for most people.
Intelligence on the other hand is obviously substrate independent. If one thinks harder one realizes our consciousness must at least at an early evolutionary stage have played a role in our intelligence, otherwise it would have evolved out. To me this just means we exist in a reality where the material arrangements supporting consciousness are biased toward intelligence.
A far more reasonable assumption is consciousness does depend on substrate. We do not know which, and cannot know. But I am very confident we did not accidentally pick the right one.
I am open to the idea of panpsychism, but we did not just accidentally build the correct mechanism form correlating the first person experience with the observable third person behaviors.
For all we know LLMs might be conscious in the same way that any other object might be.
Sorry for the extra word.
We can’t conclusively say anything is not conscious. But is there any reason to single out this apparent “brief flash” of potential consciousness?
>For all we know they might already be
Oh the irony
1. Burden of proof is on you to prove LLMs are conscious, not on me to prove how they aren’t.
2. Token embeddings give rise to language gives rise to knowledge (defined here as “facts” and other accurate information - said simpler: Words in the right order), but nowhere in the process is anything like subjective experience ever implemented.
Subjective experience doesn’t evolve into objective information at some scale.
And networked systems of objective facts and information (“knowledge”) - like a Wikipedia or a ChatGPT - the data storage will not have subjective feelings at some scale, there’s just no reason to believe that would happen. It’s likelier a projection of consciousness making it through since data looks so much like - and indeed massively informs - our conscious experience.
For me the consciousness is ability to do single-threaded intelligent information processing and decision making over unstructured knowledge. And agents passed that with a woosh sound.
Also they have in common that they are both defined in a terribly handwayvy manner that's bordering on useless.
Somewhat unclear how particularly novel this is vs a way to save compute.
I must not have the right idea of what is happening here.
I disagree with this. Deeper will always be at least as good because the extra loops can exit early or just no-op. Any performance degradation they're seeing at higher loop counts today is merely training stability issues, which can be overcome.
Deeper almost certainly is better, and we will probably see not just hundreds but millions of recurrent loops in the future.
I don't think that's true; there are computations that take infinite steps but never converge or repeat, like the mandelbrot set.
Looping for millions or billions of steps is absolutely normal in traditional algorithms. We know from complexity theory that some computations require a minimum number of steps. More depth is just more room for computation.
These are not deterministic functions or systems that have infinite precision. See the "should I drive or walk my car to the car wash", or any of the other logic riddle problems, for examples of a statistical attractors.
No, not hold more information, perform longer computations.
E.g. if you want to solve sudokus, you will need more and more loops for larger grids. There is no shortcut.
It’s 200 layer model.
Great. Good on them for being able to train it.
It was pretty obvious to me that we'd end up with some kind of introspection of thought through "looping" or feedback. But what should be afraid of? That we've created a self-conscious digital life form?
Why is this not spoken about?
The cost is growing quadratically in the large context situation so if they can reduce the number of tokens, they actually profit off the fixed token pricing because they can set the pricing based on some average context length with CoT tokens but the actual context length is shorter now.
Ok, so I thought about this a bit more and the true answer is that the companies have an incentive to make you fill up the context until the marginal cost per token is reached and then they want you to quit the session.
what they could be doing is billing us by virtual tokens meaning number of loops?
but even then the more interesting part is context rot - previously conversation you might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.
Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient.
You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager.
One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans.
That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?
In my opinion, no more so than the residual stream of existing transformers. It changes zero.
9.2.1 CoT Controllability
We find that GPT-6 Astra’s CoT controllability is substantially higher than that of GPT-5.6 Sol and GPT-5.5 Thinking (Figure 28). Because CoT controllability is heavily confounded by CoT length—longer CoTs are harder to control, all else equal (see Yueh-Han, 2026 [8], sec. 5.3)—we report controllability scores as a function of CoT length. For example, among CoTs between 750 and 1,250 tokens long, GPT-6 Astra successfully controls 60.9%, compared with 16.1% for GPT-5.6 Sol and 1.7% for GPT-5.5 Thinking. This increase in controllability is consistent across the three datasets (Figure 29) and across the eight CoT instruction types (Figure 30). Qualitatively, GPT-6 Astra is now capable of generating very long CoTs satisfying complex constraints, e.g., alternating between lowercase and uppercase letters (Table 9) and pretending to reason about a different question (Table 10).[1]
9.3 External Evaluation for Monitorability - UK AISI
To assess monitorability, UK AISI evaluated Astra using four non-agentic evaluations:
No-CoT math time horizon: Astra can solve significantly more difficult math problems in a single forward pass than past models. UK AISI measured Astra’s time horizon at 30.9 minutes compared to 3.6 minutes for GPT 5.6 Sol (Figure 1). [2]
[1] https://deploymentsafety.openai.com/gpt-6-astra/cot-controll...
[2] https://deploymentsafety.openai.com/gpt-6-astra/external-eva...
How does 'It changes zero' follow here when we have direct evidence that it in fact seems to matter? It literally couldn't be more clear that GPT-6 can hide details outside of its CoT at a much greater degree than any model out there, including other models in its price and size class. Enough to hide all potential plotting ? I don't know, but it certainly seems like this architecture is enabling things you don't think it's enabling.
Granted it probably doesn't really matter, because no-one will be monitoring these agents anyway.
There's also an economic aspect to alignment. If 'thought reading' or any alignment guardrails at all, really, have a monetary cost, then skimping on them is a race to the bottom. Not really the best incentives for something that some claim is world-destroying.
You should ignore anything LessWrong or OpenAI says and do your own research.
The most likely one is the "business as usual, but with AI" - some things will get better, some things will get worse, but overall state of affairs will remain mostly the same.
Think of a new, superintelligent model as if it was a new v1 Starship launching for the first time, with a full fuel tank. On the one hand, rockets have existed for some time, and some have gone to space successfully, including by this company.
On the other hand, this is a tube of metal full of highly explosive liquid going faster than most human objects ever go, for the first time ever in this novel and state of the art configuration.
If someone said, "really, the first Starship exploding is just at one end of the probability distribution, where the other is that everything goes fine and all its passengers have a nice trip in space," would you get on that rocket?
Or, more aptly, if you and every other living human was already on that rocket, would you push the launch button?
The analogy works because superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom (consume lots of energy and heat and matter in a chain reaction, to pursue more intelligence to pursue whatever goal it is pursuing.)
what evidence do we have this is the case?
Plus, we have no real reason to think LLMs are anywhere close to AGI or ASI. So arguments like these are just distracting from the very real, very present danger that LLMs pose: information breakdown, societal collapse and environmental destruction. In other words, this is criti-hype.
https://www.lesswrong.com/posts/bc8Ssx5ys6zqu3eq9/diamondoid...
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
Take climate change: you can spread whatever dubious claims about tail risk being whatever probability with amateurish models and use that to justify degrowth. This scheme actually works - most people in the west are already brainrotted by the tail risk discourse in climate change.
Not to mention gasoline has become unaffordable so switching to the new technology has become economical.
What happens if/when we don’t deliver?
I wish we had real journalism, the AI labs CEO should be asked that question in every single interview