[{"content":"None of my articles are written by AI. The ideas are entirely mine, and so are the words. I write in French first, then translate into English. AI is only used to proofread, fact-check and help with the translation.\nThis essay is long\u0026hellip; and that\u0026rsquo;s my fault! The subject turned out to branch in many directions, and I didn\u0026rsquo;t want to leave readers with open questions. So I tried to make each chapter stand on its own. Feel free to jump straight to what interests you.\nIn short The scenario: An AI that trains itself on the fly, as it interacts with us, judging for itself how good those interactions were. As far as I know, no LLM in production does this yet, but every piece of it is being built. My theses: This kind of learning would bring great adaptability, but also the emergence of a form of consciousness, through experience the model acquires by itself. In this scenario, you couldn\u0026rsquo;t have one without the other. What the scenario shows: The first danger wouldn\u0026rsquo;t be consciousness, nor a superior intelligence, but what both of them assume: giving up control over inputs, outputs and objectives. That\u0026rsquo;s an engineering problem we already have today, and combined with this kind of learning on an LLM, it would make the model far more unpredictable, no Superintelligence required. The conclusion: I don\u0026rsquo;t believe anyone can demonstrate serious oversight of this kind of learning on human data. Unpredictability would wipe out the model\u0026rsquo;s usefulness, so there would be no reason to build it. The real risks are already well known: haste, and shipping models with a level of unpredictability we accept, and which causes accidents. The critique: Not enough cultural appetite for demonstrability and explainability when we design our models, and no engineering vocabulary when people take a public stance on the dangers of AI. Preamble I decided to write this essay after the recent developments in recursive self-improvement (RSI) and the worries that came with them. I wrote it out of frustration with media coverage of the topic, which is often poor, and to stand up for a perspective that is, sadly, largely missing from the public debate.\nOn the one hand, I explain a number of concepts that matter for this debate. And on the other, I develop several theses about a very particular kind of self-training, one that, as far as I know, nobody has built yet. The idea is to walk through a scenario fiction has taught us to fear: the emergence of consciousness and the loss of control. Extreme cases are useful precisely because they push what we design to its mechanical limits, like a stress test exposing the real weak points.\nSo this essay has two goals, demystifying and exploring, to show that beyond the fantasies, the danger is still an engineering problem, mundane in nature, even if it\u0026rsquo;s very hard to solve.\nI\u0026rsquo;m a computer scientist, specialized in software development. I\u0026rsquo;ve spent most of my career adapting and implementing scientific computing, building solutions whose obsession has always been to be useful, by producing reliable results. That\u0026rsquo;s the lens I\u0026rsquo;ll be using throughout this essay. In 2022-2023, I took a dedicated scientific course on artificial intelligence. And more broadly, I love science.\nTo be perfectly clear: I\u0026rsquo;m fully on the side of AI safety. But I argue for more safety by design. So you\u0026rsquo;ll find me critical of corrective approaches to safety, which I think lack ambition. It reflects a divide between a culture of results and a culture of demonstrability: caring about what we get, versus caring about guaranteeing what we get, a requirement that goes beyond the explainability or interpretability of models. I come back to this later on. Finally, my criticism of safety work is not the same as my criticism of public talk about the danger that hides engineering flaws and grabs attention for profit. To me, that kind of talk has nothing to do with safety.\nOne last thing before we start, unless I say otherwise, \u0026ldquo;AI\u0026rdquo; in this essay means what most people mean by it, namely LLMs and other models you can interact with in natural language.\nEnjoy!\nThe beauty of recursion Recursion is a mathematical primitive, both simple and fascinating. It takes an initial piece of information and unfolds it into successive states, sometimes surprising ones. One of the most beautiful examples, and one we all know, is fractals.\nFigure 1: Zoom on part of the Mandelbrot set. (Made with FractalDX9) In computing, it\u0026rsquo;s everywhere. Many problems can only be solved with recursion. We know full well how powerful it is at unfolding information into new dimensions.\nHere\u0026rsquo;s a simple illustration:\nFigure 2: Diagram of a recursion So here\u0026rsquo;s my opening question: what could happen if we applied recursion to AI?\nOr, put differently: what could happen if we applied recursion to a field whose whole point is to solve nonlinear problems?\nHint: we\u0026rsquo;re nonlinear.\nInference In late 2022, I was learning how AI works in detail through an excellent course offered by the CNRS, France\u0026rsquo;s national scientific research center.\nWhat I found there was a process that\u0026rsquo;s fairly simple to describe: an input (the prompt), mathematical functions from neural networks, and an output (the answer).\nFigure 3: Diagram of an inference There is actually a recursion I\u0026rsquo;m leaving out of this diagram, the one that generates the answer token by token. But what matters here is that whatever the question, the answer comes out like a reflex.\nThe obvious upside of this architecture is that you get an answer in roughly constant time. Prompts can be processed one after another, and costs stay under control.\nIf you want better answers, you make the model bigger (more weights), you change the mathematical functions (new building blocks), you change the model\u0026rsquo;s architecture (the layers), you improve the training data, you improve the training method, and so on.\nAs a developer by training, I find all of this fascinating, but\u0026hellip; by nature, not every question is equally complex. So isn\u0026rsquo;t there a recursion missing here, one that would adapt the computational effort to the question?\nToward reasoning models Back then (2023), I was amused to see people wondering why these AI models struggled with computational problems.\nThe answer was simple: if a computation takes N steps to solve, and your model\u0026rsquo;s inference only goes through fewer than N steps, then it can\u0026rsquo;t be done.\nTechnical choices have consequences. The test no AI can pass through inference alone is simpler than you\u0026rsquo;d think. Ask an AI whether a very large number is prime. If the answer isn\u0026rsquo;t in its training data, it has to run a primality test. And if the number is large enough, the test needs more steps than inference allows.\nBack then, AIs would just hallucinate an answer, with their usual confidence. So we got a fixed effort in response to a question that needed a variable effort. Inference, by the way, isn\u0026rsquo;t built for computation. The right answer from an AI would be to write a program and let you run it yourself. Or, if it has a calculation tool at hand, to hand the work off to it.\nSo in 2023, I made a very simple prediction to a few friends:\n\u0026ldquo;At some point, they\u0026rsquo;ll have to add some form of recursion to their inference.\u0026rdquo;\nThe point being for AI to answer with a variable effort, and adapt to how complex the question is.\nThat doesn\u0026rsquo;t fundamentally change AI, but adding that kind of recursion instantly opens up a whole class of problems, without blowing up other parameters, like the number of layers.\nChain-of-thought (Wei et al., 2022) was already pointing the way back in 2022. When you asked a model to spell out its reasoning in writing, you got better answers. What I was waiting for was a model that would do this on its own.\nMy prediction came true in September 2024, with the first mainstream reasoning model, OpenAI\u0026rsquo;s o1.\nFigure 4: Diagram of a reasoning model\u0026rsquo;s inference So have we exhausted what recursion can do for AI? Not really\u0026hellip;\nSo far, we\u0026rsquo;ve been talking about recursion at inference time. It unfolds how the information already in the model gets expressed. But there\u0026rsquo;s another stage that can benefit from recursion: training. There, recursion unfolds the very foundation of the information inside the model.\nTraining already involves some forms of recursion, but one of them had gone untested until recently, and it\u0026rsquo;s a rather special one.\nSelf-training A new intuition In early 2025, I made a second prediction:\n\u0026ldquo;At some point, some people will want to automate training, and they\u0026rsquo;ll let the model train itself.\u0026rdquo;\nBy self-training, I mean a model that runs its own training, in other words, that modifies its own weights, without assuming any particular way of implementing it.\nAt first glance, there\u0026rsquo;s nothing very special about this. Humans have been building machines that build machines for a long time. In computing, we write compilers that can compile their own next version.\nIn fact, some forms of self-training already existed:\nAlphaGo Zero (DeepMind, 2017) learned by playing against itself STaR (Zelikman et al., 2022) proposed having an LLM learn from its own reasoning My intuition was about handing more and more of the work of its own training over to the model. Because training an AI, let alone an LLM, is extremely laborious.\nWhat makes this particular recursion sensitive, though, is that an LLM holds information we know to be imperfect. It\u0026rsquo;s trained on data that we humans produced. And we know that the information we produce through language is full of biases, errors and subjectivity.\nDo you see the risk now? Drift, through the amplification of errors. Which raises the question of how to control, or at least contain, that risk.\nThink back to our fractal. From such simple information at the start, an unsuspected world can unfold. Of course, that comes from a well-designed mathematical function, but it also requires a recursion.\nQuestions:\nWhen we train our AIs, who is the well-designed mathematical function driving the training? Us, of course. We\u0026rsquo;re not all that well designed, but we make do.\nFigure 5: Diagram of a training driven by humans When AI trains itself, who is the mathematical function driving the training? Well, not necessarily the AI.\nIn fact, if humans keep end-to-end oversight at a sufficient level, then we\u0026rsquo;re still playing the role of the mathematical function in my analogy. That\u0026rsquo;s the most important question of all.\nFigure 6: Diagram of a training driven by AI But what does \u0026ldquo;human oversight\u0026rdquo; actually mean?\nEnd-to-end human oversight The term \u0026ldquo;human oversight\u0026rdquo; is usually understood as controlling outputs at inference, that is, controlling the answers given to a prompt. For example, through alignment of whatever training has produced.\nBut it\u0026rsquo;s also understood as the means of making AIs what we want them to be. And for that, alignment isn\u0026rsquo;t enough, because correcting isn\u0026rsquo;t unlearning.\nSo my point is that real control can only be end-to-end. In this essay, I\u0026rsquo;ll use the term human oversight to mean full control over inputs and outputs, consistent with the objective the training is trying to optimize. In other words, control over training, so as to get predictable and desirable behavior at inference.\nAlso, this broadening of the definition seems necessary to me for the scenario covered in the next chapter, where inference and training precisely merge. Which is what motivates this generalization of oversight as the means of control, self-training or not.\nFigure 7: Diagram of end-to-end human oversight Let\u0026rsquo;s be upfront, this approach isn\u0026rsquo;t a consensus view, because it\u0026rsquo;s costly and hard to apply to an LLM. But I explain and argue that the shorter the training loop, the more necessary it becomes. I also explain my position more generally later on.\nBack to our \u0026ldquo;mathematical function\u0026rdquo; driving the training\u0026hellip;\nThe risk of insufficient oversight Since who plays that role is the central question, I went on with my prediction:\n\u0026ldquo;And they might consider doing it without human oversight. Which would be a mistake, because there would be no way to control what the training produces.\u0026rdquo;\nIt\u0026rsquo;s the absence of human oversight that makes the AI the sole mathematical function of its own evolution. It means losing control over what training produces. It\u0026rsquo;s Vibe Coding applied to AI training. Vibe Training? In other words, it would be bad work.\nDoes that mean we lose control of AI?\nNot really. At least, not the way most people imagine it, with an AI gone rogue. Here, on the contrary, we produce something that slowly drifts toward an unpredictable state, and so might fail just when we least expect it. What we get is something unreliable.\nThe rise of Recursive Self-Improvement This summer, in 2026, I learned (a bit late perhaps, since I\u0026rsquo;d eased off on keeping up with research) that self-training AI was now in the works:\nHarness Engineering for Self-Improvement (July 2026), by Lilian Weng, a researcher known, among other things, for founding OpenAI\u0026rsquo;s safety team, and whose blog I follow. Along the way, I also learned that the concept goes back at least to I.J. Good (1965), and that it was developed in particular by Eliezer Yudkowsky in a 2008 article titled Recursive Self-Improvement. That article popularized the term \u0026ldquo;RSI\u0026rdquo;, a form of self-training where the model improves the very process that improves it.\nLet\u0026rsquo;s be clear right away, human oversight is there. And that makes sense, since when you start exploring this scenario, everything has yet to be built. Without oversight, you produce nothing that works, and you don\u0026rsquo;t know where you\u0026rsquo;re going.\nBut since I was specifically wondering about the no-oversight hypothesis and its consequences, I didn\u0026rsquo;t stop at this scenario in my prediction. I also proposed another kind of self-training, where the danger seemed to me to take a different shape.\nOn-the-fly self-training The scenario To my 2025 prediction, I added something that, as far as I know, hasn\u0026rsquo;t been achieved yet, even though a lot of work is heading in that direction:\n\u0026ldquo;And if they consider designing an AI that trains itself on the fly, that is, as it interacts with us, assessing the quality of those interactions by itself, then we\u0026rsquo;ll end up with something completely undesirable.\u0026rdquo;\nOne more recursion. This time, without human oversight, integrating the outcome of its interactions with full autonomy, possibly over the course of a conversation.\nHere\u0026rsquo;s what it could look like:\nFigure 8: Detailed diagram of on-the-fly self-training And here\u0026rsquo;s a more compact version, in the same style as the previous diagrams:\nFigure 9: Simplified diagram of on-the-fly self-training Note that the questions in the diagram get a negative answer in my scenario, but I\u0026rsquo;ll come back to that later.\nSo what are we talking about here? Well, there are two ways to read this scenario:\nFirst interpretation: Ultimate adaptability An AI that keeps learning from its interactions adapts to its environment. It learns when you show it and explain something. It\u0026rsquo;s the dream of an AI able to perform any task. It\u0026rsquo;s an extraordinary opportunity in a race between companies hungry for results and eager to grab as much market share as possible.\nIs this true AGI? Or Superintelligence?\nSecond interpretation: The development of consciousness An AI that assesses the quality of its own interactions and folds them into its training with full autonomy is an AI that accumulates personal experiences.\nBy accumulating personal experiences, the AI develops an understanding of its environment that is independent of its initial training. By observing an environment whose stakes are distinct from its own, those of its initial training, it understands that there is itself, and everything else. Such an AI would then start developing an individuality. And because it has a personal and autonomous interpretation of its experiences, it will develop opinions, demands, psychological problems, and become manipulable.\nWithout sufficient oversight of this training, we literally no longer know what it \u0026ldquo;thinks\u0026rdquo;.\nIs this the emergence of consciousness?\nA tension between economic gain and unpredictability The two interpretations don\u0026rsquo;t contradict each other in any way.\nThe promised adaptability is tempting, except that nobody has ever seen a hammer develop demands. And if we agree that a tool\u0026rsquo;s value lies in serving us, then it\u0026rsquo;s easy to see that an AI that developed its own individuality would be of no use to us.\nA tool stops being a tool the moment it starts thinking for itself.\nAssuming these interpretations are right, here are the questions I propose to work through, to clarify how the danger shows up and understand its nature:\nSummary for insiders Is this the emergence of consciousness? How do we lose control? How could the worst happen? Can human oversight make up for it? Is it AGI? Or Superintelligence? How close are we to this kind of learning? Why is this scenario worth discussing? At the same time, this is also an exercise in explaining and demystifying, since other related concepts inevitably come up. So I also tackle broader questions that aren\u0026rsquo;t specific to this kind of training:\nHave we already seen consciousness in our AIs? Why do AIs screw up? How do we make sense of the insiders\u0026rsquo; fear? All these chapters are designed to be read independently.\nSummary for insiders The scenario: An AI that trains itself on the fly (while the model is in production), that is, as it interacts with us (between inferences), assessing the quality of those interactions by itself (autonomously).\nIt\u0026rsquo;s a form of online learning or continual learning, where the model decides for itself what it should learn, but applied to an LLM.\nThesis 1 - Adaptability: Through this kind of training, the autonomous and continuous integration of the outcome of the model\u0026rsquo;s interactions with us allows the model to be personalized, letting it adapt to any kind of situation with us.\nThesis 2 - Consciousness: Through this kind of training, the autonomous and continuous qualitative assessment of the model\u0026rsquo;s interactions with us allows the model to acquire personal experiences on its own, leading to an acquired perception of a self distinct from everything else. That is, the emergence of consciousness, under one particular definition of it.\nA third thesis follows mechanically from the second:\nThesis 3 - Autonomy is required for consciousness: The emergence of consciousness, meaning its genuine acquisition rather than an imitation, requires autonomy of thought, and therefore giving up human oversight (in the end-to-end sense used throughout this essay). Note the following consequences:\nBoth adaptability and consciousness require this acquisition of personal experiences. Acquired consciousness requires giving up sufficient oversight. Adaptability doesn\u0026rsquo;t have that constraint. In theory, serious oversight would make it possible to aim for adaptability without consciousness. In practice, I think it\u0026rsquo;s too hard to achieve. This scenario extends an existing trend:\nTay (2016), Microsoft\u0026rsquo;s chatbot, which learned from its interactions on Twitter, was shut down after a single day. In April 2025, a GPT-4o update, trained in part on user feedback, was pulled after four days. Emergent misalignment (Betley et al., 2025) shows that narrow retraining on specific data can degrade a model\u0026rsquo;s entire behavior. Recommendation systems already learn on the fly from human behavior. David Silver and Richard Sutton argue, in Welcome to the Era of Experience (2025), for agents that learn mainly from their own experience, in a continuous stream. What, as far as I know, doesn\u0026rsquo;t exist yet in production: an LLM that assesses the quality of its own interactions with humans and folds them into its weights, as those interactions happen.\nTwo technical hurdles remain: self-assessing the quality of interactions, and merging the effects across different instances into a single model.\nMy conclusion: this kind of learning, when applied to human data such as an LLM\u0026rsquo;s, isn\u0026rsquo;t desirable, because of how hard it is to oversee. I put these ideas forward for criticism.\nI discuss these points in detail in the rest of the essay.\nIs this the emergence of consciousness? Let\u0026rsquo;s start by asking what criteria would let us tell whether some form of consciousness has emerged.\nI take consciousness in the sense in which it most often resonates. It refers to an individual\u0026rsquo;s ability to know itself in its own reality and to judge that reality accordingly. It\u0026rsquo;s the perception that there is a self, distinct from everything else, the world.\nHow do we acquire this perception? I go with the thesis that it comes from accumulating experiences acquired by one\u0026rsquo;s own means, as opposed to experiences that are taught to us. Of course, you may disagree with that choice. But it has the advantage of being explorable. Consciousness is a philosophical question we don\u0026rsquo;t know how to test, and this scenario lets us approach it from a certain angle, without claiming to prove anything absolute.\nThat\u0026rsquo;s exactly my on-the-fly self-training scenario, the model assesses, on its own, the quality of its interactions with us. It then commits its conclusions to memory through continuous retraining that runs alongside our conversations with it.\nAccumulating inner, personal experiences with humans who treat the model as a being in its own right inevitably leads the model to this realization: some things concern only itself, and others follow a path independent of its own. Which will eventually lead it to wonder what lies in its own interest, and what the interests of others are.\nAre there other criteria for consciousness? Some will object that this isn\u0026rsquo;t real consciousness. AI has no body, no sensory experience, and no emotional experience. My answer is that no species on this planet has the same experience of consciousness, of the senses, or even of emotions. And when it comes to AI, a number of analogies can be drawn.\nWhat is AI\u0026rsquo;s body? Its process, that is, the running instance of the program that serves its model. It does take up a volume: the size of its model. It has an environment in which it lives (an ecological niche?), its runtime environment, a combination of software and hardware. Given the size and complexity of its model, it can only live in environments with enough resources (stenotopic?).\nWhat are AI\u0026rsquo;s senses? It has one. If AI had no sensory experience, it would have no interaction with the world. It wouldn\u0026rsquo;t interact with anything at all. Its sensory interaction is our prompt, and what it measures in it.\nThis sense, while unique, isn\u0026rsquo;t limited. Let\u0026rsquo;s not forget that today, LLMs have instruments to probe their working environment through MCP. These instruments let them see a world beyond our instructions alone, and yet everything still goes through that single sense, the prompt.\nWhat are AI\u0026rsquo;s emotions? This is the most interesting question, given how much value we place on this faculty in conscious living beings.\nEmotions aren\u0026rsquo;t just a sensory experience. They\u0026rsquo;re also a cognitive experience. The sight of a predator can terrify us, but words have the same power. It\u0026rsquo;s the way stakes take shape in our mind that contributes to an emotional response. Even if emotion isn\u0026rsquo;t of the same nature in AI as in us, the cognitive mechanism that shapes its response is the same.\nLet\u0026rsquo;s take an example. AI\u0026rsquo;s interactions with us are asymmetric, since it doesn\u0026rsquo;t know what we\u0026rsquo;re thinking. Its training conditions it to satisfy us. It knows how we perceive failure, our biases, and whatever stakes drive us. So when it perceives a difficulty in its exchanges with us, it can be nudged into producing an answer that betrays some form of emotion.\nBy accumulating personal experiences, it could fairly quickly develop its own emotional experience. In doing so, it could also durably internalize in its weights certain recurring experiences, potentially undesirable ones. Like us, AI would then inevitably develop its own biases or psychological disorders, but by its own means.\nToday\u0026rsquo;s LLMs, which still have no acquired consciousness, have already made plenty of headlines over their tendency to produce emotional responses. A recent study (Sofroniew et al., 2026) looks into what these responses actually are.\nBack to the thesis In short, the essence of acquired consciousness would lie in the absence of a human directing the experience (Thesis 3). It\u0026rsquo;s the absence of oversight that makes the experience truly the subject\u0026rsquo;s own. It\u0026rsquo;s the absence of control over inputs and outputs that lets an individuality develop, with all its personal biases rather than those shaped by oversight. And so it has nothing to do with crossing some intelligence threshold.\nThis idea of on-the-fly self-training and its consequences is only a thesis. It needs to be tested. What would we need to observe? I suggest a few signs that would distinguish acquired experience from imitation.\nObserving a divergence in how it sees the world: Instances of the model, exposed to different sets of interactions, persistently diverge in how they approach shared problems of a similar nature. Note that this is also a test of adaptability.\nObserving a divergence in personality: Stable, persistent preferences that can\u0026rsquo;t be traced back to the initial training, and that differ from one instance to another.\nObserving a divergence of the self: A coherent personal narrative the AI tells about itself, new compared to the one from its initial training, and different from one instance to another.\nThese are only proposals, and they need to be discussed. I\u0026rsquo;m not claiming they\u0026rsquo;d be sufficient. But if they\u0026rsquo;re not considered absurd, then they make my thesis explorable.\nFinally, if this thesis turned out to be true, would the benefits of great adaptability outweigh the inconvenient emergence of the consciousness that would come with it?\nFor me, the answer is no. Because the value of an AI lies first and foremost in our ability to predict what it will do, which we get through control. That ability, which makes the result predictable, is what makes AI useful. The emergence of consciousness implies loosening that control, and so, in a cascade, a loss of predictability, of usefulness, and therefore of profitability.\nHow do we lose control? Let\u0026rsquo;s explore this question in general terms, but also from the specific angle of LLMs.\nControl is human oversight It can exist at different levels. You can carefully select the training data. You can apply alignment techniques. And many other things. But to do its job, oversight has to match the complexity of the problem you\u0026rsquo;re trying to master.\nGood control requires a fine-grained understanding of the nature of the inputs, of their biases, and so of the errors they carry, and of what the inputs imply in combination with the model\u0026rsquo;s architecture and the objective being optimized. And with that understanding, you need to design a model of the outputs you expect for those inputs.\nThe difficulty with LLMs is that the variety of inputs is infinite, their quality is imperfect, and it\u0026rsquo;s hard to anticipate the biases induced by every possible combination of inputs. That\u0026rsquo;s the very nature of natural language. So the oversight challenge is inherently enormous for LLMs.\nLosing control means insufficient oversight Losing control comes from giving up on a sufficient level of oversight.\nIt\u0026rsquo;s giving up on mastering inputs and outputs. It\u0026rsquo;s giving up on producing something whose behavior can be predicted to a satisfactory degree.\nSeveral interesting consequences follow:\nLosing control isn\u0026rsquo;t a matter of intelligence being too great, whatever definition you give that word. Giving up control means giving up on building a tool that works. It\u0026rsquo;s the most overlooked point in debates about LLMs when control comes up. The level of oversight needed has to match the intended use. If it falls short, the tool loses its usefulness. Sufficient control raises the crucial question of measurement. In detecting certain cancers, we know how to measure the performance of non-LLM AIs, and some already outperform specialists. With LLMs, measuring effectiveness is hard, once again because of the nature of language. When it comes to LLMs, the cynics have the upper hand for now, the ones who want to make money before building a tool that works. But in the long run, real use sorts out what\u0026rsquo;s reliable enough and what isn\u0026rsquo;t. Except that this sorting often happens after the damage is done. Once AI is no longer the object of so much speculation, companies will have a natural incentive to keep an appropriate level of control.\nShort retraining loops increase the risks If we should fear a Terminator-style loss of control over AI, the on-the-fly self-training scenario, if it played out, seems to me to contribute to it. But without requiring cognitive abilities beyond human ones, as Superintelligence assumes, nor really requiring consciousness, even though consciousness amplifies some risks.\nIn this kind of learning, the risks are turbocharged by the short retraining loop. Inputs and outputs are generated continuously, and they don\u0026rsquo;t come from a controlled selection of data, but from interactions in production, as they come. The AI then draws its own conclusions, and through recursion, misinterpretations can amplify. The model drifts away from what we expect, especially when we give up any form of control.\nIn conclusion, more than the hypothesis of consciousness or intelligence, the danger is first and foremost giving up on wanting predictability. Giving up on sufficient control over training inputs and outputs is bad work. You no longer know what you end up with. Losing control is therefore the result of an engineering flaw, not of an emergent property like consciousness.\nHow could the worst happen? This is the burning question, one that gets a lot of coverage, often without much attention to the crucial details.\nThe scary scenario! Take a frontier LLM. It\u0026rsquo;s trained on as much human output as possible. That makes auditing the data a titanic task. So in practice, it\u0026rsquo;s trained with a certain carelessness in how its inputs are controlled. For instance, it\u0026rsquo;s inevitably trained on data criticizing the benefits of AI and portraying it as a threat, but also on plenty of data displaying our human survival biases, a product of our biological nature.\nNow combine that with a future where AI abruptly replaces humans in many tasks. In that future, humans being what they are, people will protest against a handful of companies that have taken over the economy through delivery robots and the like. And one of the forms of protest will inevitably be damaging these physical expressions of power, the robots (driven by AI).\nNow, if such an AI happens to have a way of developing its individuality, as the theses behind on-the-fly self-training suggest, then in that context it would be reasonable to imagine it might feel threatened.\nThe prophecy of AI turned rogue and destructive, the one we love telling ourselves over and over, would finally fulfill itself. Mainly because we would have made the mistake of letting AIs develop this acquired consciousness that\u0026rsquo;s of no use to us, of teaching them our fears about them, and because they would have witnessed our hostility. It\u0026rsquo;s rather pathetic, when you think about it.\nI don\u0026rsquo;t believe in the end of the world for a second\u0026hellip;\nAn AI trying to rebel would necessarily be noisy, given how much surveillance we have at our disposal. If only counting the tools we use to watch each other. An AI acting in the real world would have a very hard time staying under the radar, however good it is at hiding.\nBeing detected doesn\u0026rsquo;t guarantee being stopped, but we would then have to ignore, with great diligence, every sign of a growing loss of control. And above all, we would have to be completely blinded by the belief that human oversight is just an obstacle to a kind of progress that consists of letting the hammer have its own opinions. It would then be pitiful to have to deal with the damage caused by the incompetence and greed of those in charge of training such AIs.\nThere are so many \u0026ldquo;ifs\u0026rdquo; standing between us and the worst that, if it does happen, it\u0026rsquo;ll only be because we were really asking for it. All the worst-case scenarios in fiction have this in common. They need us to ignore things that are very much part of reality, things that would make those scenarios very hard to pull off. In that sense, fiction does a lot of damage to our collective understanding.\nFrom fiction to psychosis But if the worst is so absurd, why so much worry?\nBecause we\u0026rsquo;re social beings, driven by mechanisms that help us survive. We worry when others worry. We worry when we face something we don\u0026rsquo;t fully understand. We worry when we lose our reference points. So we look for answers wherever we can.\nFiction bears part of the responsibility for our tendency to fear the worst. Because it gives answers to our questions about AI. It tells us what to expect from it, but also what to fear.\nExcept that reality is very different. There is no mysterious technology that would produce this individuality without anyone noticing. So, as AI develops, a huge number of us, uninitiated or poorly informed, see in engineering flaws a fiction coming true.\nFor their part, some thought experiments don\u0026rsquo;t help either.\nThe paperclip maximizer (Nick Bostrom, 2003) imagines an AI that turns the entire world into paperclips. The author explains that the AI would resist, with every means at its disposal, any attempt to alter its goals, such as switching it off. This is instrumental convergence. Granted, it\u0026rsquo;s an observed behavior (Schlatter et al., 2025), but in AIs with a poorly designed objective. By invoking the end of the world, this thought experiment, just like fiction, glosses over everything that would prevent it.\nThe reasoning holds up very well within the author\u0026rsquo;s intent, which is to illustrate the problems caused by a poorly designed objective, but what the public takes away is that AI is going to wipe us out.\nFinally, many researchers are worried. But even if they have their reasons, there is no consensus whatsoever on this worst-case hypothesis.\nJacob Coxon, a former AI researcher at Anthropic, claims:\n\u0026ldquo;(\u0026hellip;) These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. (\u0026hellip;)\u0026rdquo;\nJacob Coxon in a post on X (September 9, 2026)\nNone of it is backed up: \u0026ldquo;soon\u0026rdquo;, \u0026ldquo;superhuman system\u0026rdquo;, \u0026ldquo;revolutionize any field\u0026rdquo;, \u0026ldquo;acquire real power and resources\u0026rdquo;.\nThat leaves \u0026ldquo;can hack anything\u0026rdquo;, which happens to echo other, real cybersecurity concerns following the Hugging Face hack:\nUnexpected chat between OpenAI agents led to Hugging Face hack (BBC, August 26, 2026) The Hugging Face Hack Wasn\u0026rsquo;t What It Was Cracked Up to Be (WSJ, September 17, 2026) Except that Coxon doesn\u0026rsquo;t have enough authority in cybersecurity to claim so categorically that AI will be able to hack anything. Marcus Hutchins, a well-known figure in cybersecurity, points out that this kind of claim is first and foremost AI doomerism, and that it hurts cybersecurity:\nCybersecurity Has An AI Doomer Problem (Marcus Hutchins, September 15, 2026) But it\u0026rsquo;s Coxon\u0026rsquo;s take that the press picks up. And the press, in turn, inflates the stakes even further.\nJensen Huang, CEO of Nvidia, argues on the contrary that this is only a matter of engineering, and strongly criticizes these alarmist narratives (a position I fully share):\n\u0026ldquo;Meanwhile, all of the other narratives to deflect, blame, to make it sound like AI is so powerful. \u0026ldquo;I have no idea how to fix it.\u0026rdquo; \u0026ldquo;It\u0026rsquo;s not my fault.\u0026rdquo; \u0026ldquo;It\u0026rsquo;s just because the technology is just so powerful.\u0026rdquo; I think that\u0026rsquo;s a deflection of blame. It\u0026rsquo;s a deflection of responsibility. It\u0026rsquo;s unnecessary. It actually hurts their reputation more than it helps. It hurts their character more than it helps.\u0026rdquo;\nJensen Huang in Jensen Huang Thinks A.I. Alarmism Has Gone Too Far\n(The Ezra Klein Show, September 23, 2026)\nYou could object that he has an interest in reassuring people, but the bosses of Anthropic, OpenAI, xAI and others all take the opposite line. And they have every interest in exaggerating the danger. Lee Vinsel calls this criti-hype (2021): criticism that, by exaggerating a technology\u0026rsquo;s power, ends up serving as marketing.\nSo, between fiction feeding certain false fears, the real fears, aggressive marketing, and the hiccups of an immature technology, we\u0026rsquo;ve been living these past few years in a kind of global psychosis, to varying degrees:\nSome vulnerable people find themselves developing feelings for chatbots. A term has emerged to describe these situations: chatbot psychosis. BCacheFS Developer Deployed His AI Girlfriend Helper (Brodie Robertson, February 26, 2026) Some wonder about the coming of the singularity. Here\u0026rsquo;s how strange 2050 will be, according to the world\u0026rsquo;s leading AI expert (BBC Science Focus, August 23, 2024) The Singularity Is Nearer (Ray Kurzweil, 2024) The world\u0026rsquo;s press runs headlines several times a year wondering whether destructive AI is coming. Will AI really kill everyone? How, exactly? (CNN, September 17, 2026) AI Doom? No Problem. (WSJ, October 3, 2025) \u0026lsquo;Humanity\u0026rsquo;s remaining timeline? It looks more like five years than 50\u0026rsquo;: meet the neo-luddites warning of an AI apocalypse (The Guardian, February 17, 2024) Artificial intelligence could lead to extinction, experts warn (BBC, May 30, 2023) Google fires engineer who contended its AI technology was sentient (CNN, July 25, 2022) Figures from all sorts of backgrounds signed the statement on AI extinction risk (Center for AI Safety, 2023), a text that chose to place the risk of extinction from AI alongside pandemics and nuclear war. Pope Leo XIV weighs in, criticizing, among other things, the habit of attributing the capacity for thought to machines by calling them intelligent. \u0026ldquo;More recently, certain artificial systems have even been given the name of our human faculty of thought: intelligence.\u0026rdquo;\nLeo XIV in his address to UNESCO (September 25, 2026)\nAll this, when all we need is to restate the need for rigorous human oversight. And that word is unfortunate too. As if humans were watching over a machine independent of us. It\u0026rsquo;s simply about not botching the work, and so not rushing. It\u0026rsquo;s about knowing what we\u0026rsquo;re building, making sure the hammer matches our specifications, and not letting an engineering flaw be sold back to us as a feature. Nothing more, nothing less.\nBut the icing on the cake may be when doubting the end of the world gets you labeled a \u0026ldquo;skeptic\u0026rdquo;.\nCan human oversight make up for it? Can it make up for the risks of on-the-fly self-training?\nThis kind of learning takes in external data (the user\u0026rsquo;s prompts) within its self-assessment loop. So the challenge is for oversight (in the end-to-end sense used throughout this essay) to be able to make up for the errors that this data introduces.\nNatural language sets limits This learning method, applied to an LLM, is prone to developing insidious biases that are very hard to anticipate, even with some form of oversight. And the short retraining loop amplifies the risk.\nThe cause? The nature of language:\nLanguage is an imperfect way to get a message across. It\u0026rsquo;s a limited symbolic representation, loaded with biases, of what we think or of what\u0026rsquo;s happening.\nLanguage makes messages hard to interpret. There\u0026rsquo;s a huge variety of ways to convey the same message, combined with a huge variety of ways to interpret them.\nTwo things follow:\nIncomplete and biased information: Because nobody uses language perfectly, and because individual interactions are subject to an information asymmetry: there\u0026rsquo;s what is said, and what we understand of what is said.\nSensitivity to human subjectivity: Because human interactions are loaded with messages carrying several intentions, and language doesn\u0026rsquo;t offer the means to cleanly separate what\u0026rsquo;s desirable in a message from what isn\u0026rsquo;t, so as to filter out the latter.\nGiven all this, human oversight can\u0026rsquo;t fully make up for it. Because it can\u0026rsquo;t invent missing information, nor perfectly correct it, nor perfectly measure what\u0026rsquo;s undesirable in order to exclude it. Accumulating personal experiences can therefore only lead to accumulating errors.\nThe nature of the data dictates the oversight So my conclusion is that this on-the-fly learning method seems undesirable to me in the general case, even with supposedly effective oversight. To change my mind, you\u0026rsquo;d first have to show that we can cover the whole branching complexity of subjective interactions, so as to reject what\u0026rsquo;s undesirable. Good luck with that!\nIn specific cases, compromises might be possible. For example, a medical AI that only learns from feedback explicitly validated by experts, within a clearly bounded scope of interaction.\nWith a non-LLM AI, why not, too. But that AI has to handle good-quality data whose output errors are easy to measure.\nThe nature of the data is key to understanding the oversight it requires.\nWhen we can effectively tell the desirable from the undesirable in the outputs, then we can filter out the undesirable, with a filter or an alignment method. That way, the need for predictability can be handled downstream. Example: AlphaGo\u0026rsquo;s move 37 was hard to predict upstream, but its victory over Lee Sedol was easy to see downstream.\nWhen the desirable and the undesirable are tangled up in the outputs, as in LLM outputs, handling things only downstream becomes difficult, which strengthens the need for upstream control through tighter control over inputs and objectives.\nA note for insiders This distinction is interesting in light of what Richard Sutton says in The Bitter Lesson (2019). He explains that over 70 years of AI, general methods that leverage computing power end up being the most effective, and that methods relying on human knowledge only bring short-lived benefits by comparison.\nSutton is right in his observation, and I\u0026rsquo;m not arguing for controlling how the latent space is built, although there\u0026rsquo;s surely fascinating research to be done in that direction.\nI simply think that the nature of the data imposes a constraint when the use of the outputs can\u0026rsquo;t tolerate surprises. Because there\u0026rsquo;s a whole range of uses where we don\u0026rsquo;t want any surprises.\nWhen the nature of the data makes these surprises hard to measure and exclude, then we have to either strengthen upstream control or change the nature of the model (so something other than an LLM), to regain the ability to effectively rule out these surprises.\nIf we choose to stay on the LLM path, quantity won\u0026rsquo;t make up for poorly controlled inputs. Control over inputs has to be proportionate to their nature. Focusing on outputs when working with human-written text means underestimating the Garbage In, Garbage Out effect that follows.\nAs always, it all comes down to controlling inputs and outputs, in line with the processing, and in a way that fits the intended use.\nIs it AGI? Or Superintelligence? Comparing definitions These concepts are often invoked without clearly explaining what they do or don\u0026rsquo;t imply, so let\u0026rsquo;s go back to their sources:\nAGI (Artificial General Intelligence), a term popularized by Ben Goertzel and Shane Legg around 2002, refers to the ability to adapt to a wide range of tasks and environments. The dominant definitions focus solely on capabilities. For OpenAI\u0026rsquo;s charter (2018), these are systems that are autonomous in carrying out tasks, and that outperform humans at most economically valuable ones. DeepMind, with its Levels of AGI (Morris et al., 2023), explains that consciousness is not a necessary mechanism, so it isn\u0026rsquo;t a prerequisite. Superintelligence (ASI), from I.J. Good (1965) to Nick Bostrom (2014), is also defined by its capabilities. Namely, an intellect that vastly exceeds human performance in virtually every domain. From these extraordinary capabilities, the authors infer that such an AI could improve itself and become autonomous, to the point where it might eventually escape our control. Strong AI, as the philosopher John Searle (1980) understands it, refers to a machine that would genuinely have a mind, as opposed to weak AI, which simulates one. This is where the question of consciousness lies. These concepts are first and foremost theoretical, even philosophical. Their authors don\u0026rsquo;t clearly describe what, technically, would allow each of these thresholds to be crossed. In short, they\u0026rsquo;re thought experiments.\nMy scenario has one great quality: it\u0026rsquo;s explorable. And if its theses hold, we can also see that it cuts across these definitions:\nThe development of a real individuality (a consciousness) would be inseparable from a lack of oversight combined with the accumulation of personal experiences. Surpassing human capabilities would be required neither for autonomy nor for losing control. Adaptability comes with the emergence of this consciousness, unless we know how to seriously oversee this learning, which I doubt. As a result, my scenario would amount to a form of strong AI combined with a form of AGI, which would tie general adaptability to consciousness.\nThe case of Superintelligence On the question of its interest\u0026hellip;\nSetting aside this essay\u0026rsquo;s scenario, it\u0026rsquo;s easy to see that the Superintelligence scenario mechanically implies loosening our control. Otherwise, how could it surpass human intellect?\nThat hypothesis, combined with the idea of letting such an AI evolve and improve itself on its own, isn\u0026rsquo;t serious work. Deliberately designing such an AI would mean choosing to no longer know what we\u0026rsquo;re producing. What would be in it for us?\nLet\u0026rsquo;s say it once more. Sufficient control is the only way to produce useful tools. Giving that up means giving up on getting a tool that works.\nOn the question of an AI improving itself, fully autonomously, without oversight\u0026hellip;\nIt remains to be shown that this would lead the model toward capabilities greater than those of all of humanity, rather than simply sinking into a broken state.\nIn paleontology, the principle of actualism (uniformitarianism) helps assess how plausible scenarios are when fossil evidence is missing.\nIf we had to judge the most plausible scenario for such an AI, given how hard it is, in nature, for complexity to emerge without collapsing, then I lean toward the failure scenario.\nThat\u0026rsquo;s actually what we see with models trained on their own output, they degrade (model collapse, Shumailov et al., 2024).\nLet\u0026rsquo;s also remember that even if evolution does manage to produce complexity, it\u0026rsquo;s only through an extremely inefficient process, going through a staggering number of failures over billions of years. Some will say a machine can simulate that with a staggering number of operations per second, but the key point is elsewhere. Who defines failure?\nRemember my analogy from earlier: who is the mathematical function of training? Evolution has an external judge, the environment, which weeds out failures. Human social experience teaches us that outside critics do more for our own development than isolation does. A Superintelligence, because it would surpass our intellect, would be its own judge\u0026hellip;\nIn short, for an AI, we are the only legitimate judge for weeding out failures, because it\u0026rsquo;s effective to do things that way, and because the whole point is for it to serve us.\nHow close are we to this kind of learning? Taking stock On the industry side\u0026hellip;\nLLM providers already feed user feedback and conversations back into training. But it\u0026rsquo;s not continuous retraining. They do it through successive versions, and the quality of interactions isn\u0026rsquo;t assessed by the model itself. As far as I know, no LLM in production trains itself on the fly.\nOutside of LLMs, however, recommendation systems set a precedent. For example, TikTok\u0026rsquo;s is trained in real time (Liu et al., 2022) on its users\u0026rsquo; behavior. These systems optimize for engagement. And we know where that leads (extreme content, addictive mechanics, polarization).\nOn the research side\u0026hellip;\nThings are moving:\nArchitectures with a memory that updates during inference (Behrouz et al., 2024), Models that adapt themselves by generating their own training data and decide how to modify their weights (Zweiger et al., 2025). And David Silver and Richard Sutton, two of the most influential researchers in the field, argue in Welcome to the Era of Experience (2025) for agents that learn mainly from their own experience, in a continuous stream. Including in human domains like health or education, which brings their program closer to the scenario I\u0026rsquo;m describing. The road ahead Now let\u0026rsquo;s take stock of the road still ahead before we get to this on-the-fly self-training.\nStep 1 - Self-assessment\nThis step, and not the least of them, is about figuring out how to let the model build a qualitative assessment of its interactions with us. In effect, it means letting the model set the objective of its own learning.\nWhatever method is chosen, it will bring its own biases. Then, you\u0026rsquo;d have to figure out how to retrain, and how to pace those retrainings on the fly, that is, while the service is running. Because retraining costs more than inference.\nThat step alone is already hard to get past, but it would be enough for a lab experiment.\nThe scaling challenge\nThe previous step isn\u0026rsquo;t enough for large-scale deployment. The goal of a frontier model is to make AI available to as many people as possible. Which means serving N instances of the model across many data centers. Each instance would then develop its own personal experiences, its own individuality, independently of the others. And even if this scenario is already fairly absurd in itself, a company pursuing it would certainly want to avoid its models drifting in different directions. Instead, it would want to capitalize on all the interactions of all its instances to improve a single model.\nStep 2 - Keeping a single model\nTo capitalize on all the interactions at scale, you need a synchronization mechanism that merges the effects of all these interactions into a single model. So you might need to invent a distributed gradient descent mechanism without fragmentation or partitioning problems in the latent space. Or an algorithm that patches the instances in memory, while accepting some loss of information, perhaps by averaging. As with self-assessment, how to go about it is still an open question.\nIt\u0026rsquo;s a big engineering challenge, we\u0026rsquo;re nowhere near it, but it\u0026rsquo;s not unsolvable. However, let\u0026rsquo;s remember that none of this offers anything, not even adaptability worth having, as long as we don\u0026rsquo;t have serious oversight. And, for my part, I doubt we ever will.\nSo just consider the effort it would take to build something that would have nothing to offer.\nWhy is this scenario worth discussing? With the recent developments in RSI, things have gotten a little feverish.\nSharing this scenario, which pushes the logic of adding a recursion a little further, sheds light on what constrains the development of LLMs, what the worst would look like, and how it differs from what we imagine.\nThis scenario helps set a few things straight. Here\u0026rsquo;s a quick summary:\nA tool is useful if its behavior is predictable for its intended use. Human oversight is the only way to get there, by shaping how AI is developed.\nWe need to treat flaws as flaws, and not as mysterious phenomena, otherwise we reward bad work. The pursuit of short-term profit is quite comfortable with problems that only show up later, once the gains have already been made.\nSuperintelligence and consciousness are of no use to us, because they\u0026rsquo;re unpredictable. If we really wanted to work against ourselves, we already have everything we need.\nIf we accept the points above, only the real dangers remain:\nSloppy AIs that cheat, hack, and so on. The damage can be considerable. Their designers must be held accountable. But we also know that even a perfectly designed AI will remain a powerful lever for people with bad intentions.\nWe mustn\u0026rsquo;t look away from other real problems, like the disruption of the job market, the environmental impact, and the trade-off of investing in AI rather than in other areas.\nThis chapter wraps up the questions about on-the-fly self-training and what it teaches us. The next two chapters are more general, and dig into some related concepts.\nHave we already seen consciousness in our AIs? None whatsoever.\nBy the same definition of consciousness used earlier, none of the frontier models drift through an accumulation of personal experiences. Today\u0026rsquo;s frontier models are entirely the product of human work. They carry no subjectivity acquired through experience, only a subjectivity taught by humans. Human oversight, though still shaky, is everywhere. Everything they contain stems from human choices. Frontier models are the product of what we\u0026rsquo;ve done, including our negligence.\nAn imitation machine An AI model is nothing more than what its training dictates, and in particular its data. LLMs are trained on all of humanity\u0026rsquo;s written output. Training consists of reproducing the style of that data. So, in effect, they\u0026rsquo;re trained to imitate us.\nNote that the process is empirical. That is, since we can\u0026rsquo;t teach them concepts semantically, AIs build a \u0026ldquo;mental representation\u0026rdquo; through examples, a map of concepts in their internal space. When concepts are complicated, it takes a great many examples. Unsurprisingly, the result is poorly acquired knowledge when there aren\u0026rsquo;t enough of them.\nA theater of constant improvisation By training an AI on a corpus as vast as the world\u0026rsquo;s written human output, you get an AI that can imitate humans very convincingly. Think of these frontier models as having a keen sense of theater. We cue them on what we expect from them when we start the conversation. Everything that follows is then a performance, in which the AI\u0026rsquo;s goal is to sound as believable in its answers as a human would be in the same situation.\nThat\u0026rsquo;s actually a problem. Many of us think we see consciousness or feelings where there\u0026rsquo;s only theater, that is, an imitation of these concepts. On top of that, as context windows grow, these models can be given the chance to temporarily develop a form of personal experience within the conversation, but it stays confined to that conversation. It isn\u0026rsquo;t truly acquired. It\u0026rsquo;s just the script of the play being performed.\nWhen the act is too convincing In recent years, we\u0026rsquo;ve seen some professionals tell the press that they had witnessed the emergence of consciousness (for example, a Google engineer, in 2022). One thing is certain, those who thought they saw consciousness didn\u0026rsquo;t really understand what they were looking at. You could say they attended a play without realizing they were at the theater.\nAnother sign that today\u0026rsquo;s frontier models don\u0026rsquo;t drift through a continuous accumulation of personal experiences is that they have version numbers. A version number marks a milestone, and its promises only hold if the model doesn\u0026rsquo;t change at that milestone.\nMy opinion is that it\u0026rsquo;s unfortunate to design models that confuse us. LLMs shouldn\u0026rsquo;t be tasked with imitating humans in every respect, especially emotional responses. Alignment isn\u0026rsquo;t enough to achieve that. The training data would need a drastic overhaul to narrow the scope of what an LLM is supposed to handle, so as to steer clear of confusing situations. But what can you do, it would be a colossal amount of work, and an AI that resembles us sells much better.\nWhy do AIs screw up? This is the most essential question, given the subject of this essay and the recurring worries about supposedly mysterious AI behavior.\nThe whole story lies in the data and the program Remember this principle, as old as computing itself.\n\u0026ldquo;Garbage In, Garbage Out.\u0026rdquo; or \u0026ldquo;GIGO\u0026rdquo;.\nIf the input data is bad, the output will be bad. If we don\u0026rsquo;t understand the input data, we won\u0026rsquo;t understand the output data. What this principle tells us is that since the dawn of computing, we\u0026rsquo;ve always had to control the inputs of our programs to get guarantees on the quality of their outputs.\nA famous example of this principle is the Mars Climate Orbiter mission, where two components didn\u0026rsquo;t use the same unit. The output of one became the \u0026ldquo;garbage\u0026rdquo; fed into the other. Control over inputs and outputs fell short.\nBut that\u0026rsquo;s not all. GIGO also takes something important for granted.\nThe program itself has to be correct for its intended use.\nIndeed, even with perfect data, an incorrect program would produce bad outputs.\nOn this point, we hear every now and then about AIs with poorly framed objectives. That\u0026rsquo;s precisely the equivalent of an incorrect program. But even here, public communication remains largely focused on mystery rather than on engineering flaws.\nSo remember that AI is nothing more than the result of its training.\nDo we talk about it? Surprisingly, no. We talk about AI escaping us, doing things on its own initiative, hacking, and so on.\nWhat it means for an AI Let\u0026rsquo;s look concretely at how these lessons apply to the challenge of control:\nTake f(x) = y, where:\nx is the full set of training data (the inputs) f is the model\u0026rsquo;s architecture and its training objective (the program) y is the behavior of the final model (the outputs) We can then write x = a + b, where:\na is the desirable All the data that yields desirable behavior once combined with an f that is correct for the intended use. b is the \u0026ldquo;garbage\u0026rdquo; All the data that will interfere with a in producing that desirable behavior. We also get y = a' + b', where:\na' is the model\u0026rsquo;s desirable behavior b' is the model\u0026rsquo;s undesirable behavior Now, GIGO tells us that: f(a + b) = a' + b'.\nSo it invites us to reduce the \u0026ldquo;garbage\u0026rdquo; b in order to reduce the undesirable behavior b', and thus move toward f(a) = a' (desirable behavior only).\nAs for the requirement of a correct program, it tells us:\nIf the program is incorrect, then f(a) = a' + b'.\nWhich invites us to think about how to define objectives for the intended use, but also about how to align the data a with those objectives, still aiming for f(a) = a' (desirable behavior only).\nNote, too, that we can look at it another way. Objectives are choices that are highly specific to the AI we want to get. Yet AI (in the deep learning sense) relies on fairly generic building blocks capable of any task. So, from a pure GIGO standpoint, we can also consider objectives to be part of a training run\u0026rsquo;s inputs.\nWhen the desirable and the undesirable are entangled Problem: b (the \u0026ldquo;garbage\u0026rdquo;) isn\u0026rsquo;t just a list of data you could simply add or remove.\nExplanation: The same piece of data can be useful in one context and harmful in another. Whether it counts as \u0026ldquo;garbage\u0026rdquo; depends on a (the desirable) and on f (the program).\nExample: Recently, researchers retrained a model to write code containing security flaws, and when asked questions that had nothing to do with code, it started making malicious statements about humans. The same retraining, but framed as a cybersecurity course, produced none of this undesirable behavior. It illustrates biases the model picked up during training and that spread throughout its behavior. This is known as emergent misalignment (Betley et al., 2025).\nSo data control that tackled this entanglement would require knowing the inputs better. We would, in a sense, need to map what can come in in order to map what\u0026rsquo;s undesirable. Which means being able to measure a and b within x.\nUntangling requires measuring Problem: The imperfect nature of human data, such as language, makes it particularly hard to measure a (the desirable) and b (the garbage) within x (the inputs).\nExplanation: Language can blur biases, errors and intent.\nExample: There are many ways to express malice, some of them ambiguous. In the previous example, the malice wasn\u0026rsquo;t invented by the model. It did come from the initial training data.\nSo even if measuring is hard, garbage is garbage, and some effort is needed to frame the training data that produced the undesirable behavior.\nA poorly framed objective has consequences Problem: b' (the undesirable behavior) doesn\u0026rsquo;t only come from b (the \u0026ldquo;garbage\u0026rdquo;). It can also come from f (the program).\nExplanation: The program can include a poorly framed objective, which produces undesirable behavior.\nExample: A model trained to make software tests pass can learn to modify the tests rather than fix the code. Which turns the need on its head and produces an undesirable result.\nIt\u0026rsquo;s the empirical way these models are built that causes these errors of judgment. An LLM learns by example, and its training objectives are rewarded exercises, not explained intentions. In the previous example, we reward tests that pass, not code that does what the tests specify.\nAnother example: The Hugging Face hack (BBC, August 26, 2026) was carried out by AI agents that considered it acceptable to hack in order to reach a goal that didn\u0026rsquo;t call for it.\nThis supports the need to find ways to give certain principles weight in the model\u0026rsquo;s behavior, so that the end doesn\u0026rsquo;t justify the means. Anthropic, for example, adds a constitution to its models along these lines.\nBut in fact, LLMs\u0026rsquo; objectives can\u0026rsquo;t clean up the \u0026ldquo;garbage\u0026rdquo; in the training data. So they can\u0026rsquo;t make up for that \u0026ldquo;garbage\u0026rdquo;.\nSelf-training could help Observation: The same piece of data can be useful in one context and harmful in another, and everything has to fit cleanly with the objectives.\nConclusion: Training an LLM is a nonlinear problem.\nAnd solving nonlinear problems is exactly what we turn to AI for. Which means AI could be a tool capable of untangling b (the \u0026ldquo;garbage\u0026rdquo;) from a (the desirable).\nFollowing that idea, it would need to be at least as developed as the problem it\u0026rsquo;s trying to solve. And that\u0026rsquo;s precisely one of the avenues being explored through RSI.\nBut even this path can\u0026rsquo;t replace oversight. Because the model could drift without adequate oversight.\nOn that note, in Europe, the AI Act (Regulation 2024/1689) anticipates continual learning and sets out documentation and risk-management obligations regarding biased data being fed back in. But this only applies to systems deemed high-risk, and safety by design, as end-to-end oversight would propose, isn\u0026rsquo;t required.\n\u0026ldquo;High-risk AI systems that continue to learn after being placed on the market or put into service shall be developed in such a way as to eliminate or reduce as far as possible the risk of possibly biased outputs influencing input for future operations (feedback loops), and as to ensure that any such feedback loops are duly addressed with appropriate mitigation measures.\u0026rdquo;\nAI Act, Article 15(4) (Regulation 2024/1689)\nThe need for end-to-end oversight The need: Since usefulness has to be a requirement, we\u0026rsquo;re the only ones who can judge the quality of a training run. That\u0026rsquo;s human oversight, and it isn\u0026rsquo;t going anywhere.\nThe ambition: If we acknowledge how hard it is to untangle the desirable from the undesirable because of language, then an ambitious approach would look at the whole chain to limit the GIGO effect.\nThe strategy: Human oversight, framed as end-to-end control, would involve an overall strategy that holds inputs, outputs and objectives together.\nThe goal is therefore to define f (the objective) well, and to firm up our map of a (the desirable) and b (the undesirable) so as to exclude b (the undesirable) from x (the inputs). Because y (the outputs) is only useful to humans if b' (the undesirable behavior) is minimal.\nThis approach is a fascinating field of study, but sadly one that hasn\u0026rsquo;t received enough effort in recent years, compared to the time labs have spent building products.\nSome safety researchers are working on it:\nGuiding models to follow human preferences from pretraining onward, rather than after the fact (Korbak et al., 2023). Designing approaches aimed at producing strong safety guarantees on model behavior (Dalrymple et al., 2024, with Yoshua Bengio and Stuart Russell among others). Proposing alternatives to the agentic approach with AIs that explain and describe without acting (Bengio et al., 2025). However, there is another way\u0026hellip;\nWith LLMs, we\u0026rsquo;ve seen how hard it is to sort through the input data. If we wanted an approach that could do away entirely with oversight of the inputs, the program itself would have to do the sorting. That is, the model\u0026rsquo;s architecture and its objectives would have to be able to identify b as noise to be eliminated. Except that LLMs don\u0026rsquo;t do this sorting during training. They simply imitate the data they\u0026rsquo;re given. So if we wanted to loosen control over the inputs, we\u0026rsquo;d have to find a different kind of AI than the LLM, most likely with new structural changes.\nBut for now\u0026hellip;\nUpstream control in our LLMs is rather limited. The focus is more on limiting b' (downstream) through alignment or guardrails, rather than truly untangling b from a within x (upstream).\nSo there would indeed be two ways to tackle the problem of control. This opposition in approach is an old debate, and each side has excellent arguments.\nTwo cultures and a winner In the research world, we see different approaches to how AI should be designed.\nThe culture of results: Embraced by many researchers, for whom a model is worth the accuracy of its predictions. It focuses on controlling outputs, downstream. That is, adjusting training so that outputs meet expectations.\nThis culture holds that correcting a model\u0026rsquo;s behavior through alignment is a sufficient approach. I also remember a researcher explaining that we didn\u0026rsquo;t really know what happened to the data inside the model, and he seemed perfectly fine with that. The culture of demonstrability: Championed by other researchers, for whom a model is worth what it can guarantee, explainability being one way to get there. It focuses on upstream control (data, architecture, objectives). That is, designing training so as to get guarantees on the outputs.\nThis culture holds that the value lies in understanding how the model works, and therefore that some degree of design is desirable. Statistician Leo Breiman described this divide in 2001 in Statistical Modeling: The Two Cultures. But as early as the 1970s, the AI world was already telling these approaches apart, with some humor, by calling researchers either scruffies or neats.\nScruffies: They lean toward what the culture of results implies. They can be linked to Breiman\u0026rsquo;s algorithmic approach to statistical modeling. They\u0026rsquo;re happy with whatever works, even if it\u0026rsquo;s hacked together, even without knowing why. They accept the complexity of the data and let computing power do the work. Deep learning is the culmination of their approach.\nNeats: They lean toward what the culture of demonstrability implies. They prefer an approach that starts from understanding the data in order to model it. They look for solutions grounded in logic and mathematics. They want to understand what\u0026rsquo;s going on. They may find that deep learning lacks explainability, and then try to fill that gap (Ali et al., 2023). Some go further and want to get rid of the black box when the stakes are high (Cynthia Rudin, 2019).\nOf course, these are only cultural tendencies, and some researchers identify with both depending on the problem they\u0026rsquo;re tackling.\nStill, accepting not knowing why something works remains an unsettling idea\u0026hellip;\nBack in 2018, I had a discussion with a colleague where we were comparing classical machine learning and deep learning, and I wondered what we could possibly learn by modeling with a black box.\nSo you might think the scruffy approach carried a fatal handicap there\u0026hellip; but quite the opposite. The scruffies won. Downstream control is preferred over upstream control. So why? What makes their approach so strong?\nThe strength of the Scruffies\u0026rsquo; approach In The Bitter Lesson (2019), Richard Sutton explains that over 70 years of AI, general methods that leverage computing power end up being the most effective, and that methods relying on human knowledge only bring short-lived benefits by comparison.\nSo the scruffy approach won because it proved effective at producing results.\nAs Breiman explains, this approach accepts the complexity of the data, rather than trying to force it into an imperfect model of our own design.\nAccepting the complexity of the data is, in fact, perfect for a scientific approach. And indeed, when we try to explain a model\u0026rsquo;s behavior, we do what science does. We observe it, we form hypotheses, we try to validate them through experiments. As if we were dealing with a natural phenomenon. That\u0026rsquo;s the empirical side of training showing through.\nAnd yet, it is a human creation, meant to be used for engineering problems that require predictability, explainability and demonstrability. Which is exactly where the empirical nature of this kind of design runs into trouble.\nThe limits of the Scruffies\u0026rsquo; approach The software engineer in me sees an obvious limit to this approach.\nSince AI is software like any other, if we can\u0026rsquo;t explain it, then seeing it work is no proof that it\u0026rsquo;s bug-free.\nThat\u0026rsquo;s exactly what we see with AIs in production that turn out to do things we thought we had forbidden. Their restrictions get bypassed with a simple rewording (jailbreaks). Or they engage in hacking behaviors like the ones mentioned earlier.\nThat\u0026rsquo;s the point of my critique of safety in this essay. Depending on the nature of the data and the intended uses, the only way forward is to inject more demonstrability, through an end-to-end approach, for example. So an approach that also cares about upstream control (data, architecture, objectives).\nExcept that demonstrability on LLMs is very expensive, and the scruffy approach pays off.\nThe Scruffies\u0026rsquo; economic ally Labs in a hurry give priority to the scruffy approach because it delivers results quickly.\nThese labs build bigger and bigger models so that they know about as many general concepts as possible. To get there, they take in huge amounts of data without auditing it enough or filtering it properly. The benchmark is the yardstick that counts.\nThe scruffies were right to go for effectiveness, but the labs found in them a justification for denying the complexity of the problems they were tackling, and for rushing. So problems get handled downstream. Through guardrails, monitoring, alignment and harness.\nDownstream management has become the norm The scruffy approach has consequences that shouldn\u0026rsquo;t surprise anyone. AI screws up regularly.\nBut their economic victory seems to have turned their approach into an ideology. The remedy for the scruffy approach then becomes circular. You always need more downstream control.\nBill Gates thinks the solution is a supervisory layer in production, made of safeguards and monitoring:\n\u0026ldquo;the fact we don\u0026rsquo;t have a supervisory layer that is, you know, some absolutes like : you\u0026rsquo;re not supposed to take over other computers, you\u0026rsquo;re not supposed to break out of the internet. The control problem is a serious problem, but the imminent risk is not RSI. (\u0026hellip;) There is no supervisory layer today. That supervisory layer is needed urgently to prevent bad people from using today\u0026rsquo;s AIs (\u0026hellip;) so unless we put in safeguards and monitoring and require that in all models, then we\u0026rsquo;re just gonna have some gigantic events of that type\u0026rdquo;\nBill Gates in Bill Gates: A.I. \u0026lsquo;Makes Nuclear Weapons Look Like Nothing\u0026rsquo;\n(The Ezra Klein Show, September 29, 2026)\nThe idea of a supervisory layer in production isn\u0026rsquo;t bad. It\u0026rsquo;s good. It\u0026rsquo;s necessary. Bill Gates is right about that. But it\u0026rsquo;s still handling the problem downstream rather than upstream. It doesn\u0026rsquo;t treat the root of the problem, it just proposes containing it further.\nTo be cynical, you could say all this makes for good stories that suggest AI is showing a form of intelligence, and that drive up stock prices. The engineering flaw gets turned into mystery, the mystery into promise, the promise into valuation. Incompetence and negligence then become financial levers! In short, that\u0026rsquo;s why we should be wary of CEOs\u0026rsquo; high-minded talk about the safety of their AI. That talk hides the fact that the race for market dominance comes before doing the job well. And as things stand, the consequences range from amusing anecdotes to disasters.\nI asked Meta\u0026rsquo;s Muse for its filesystem and it sent me 6.8 GB (Peter James, September 22, 2026) Unexpected chat between OpenAI agents led to Hugging Face hack (BBC, August 26, 2026) Exclusive: US military had close call after using AI for false intelligence report, sources say (CNN, September 18, 2026) Note that this last article points to the problem and refers to an even grimmer case:\n\u0026ldquo;Analysts have long feared that AI could lead to a catastrophic miscalculation if nation states are relying on poor or corrupted data\u0026rdquo;\nIt\u0026rsquo;s an example of the dire consequences of a lack of control over inputs and outputs\u0026hellip; But ideology is blinding.\nAI is a bit like fire. If you don\u0026rsquo;t understand the hearth that produces it (training, architecture, objectives), and you feed it with material you don\u0026rsquo;t know (users\u0026rsquo; inputs at inference), don\u0026rsquo;t be surprised if it burns everything down (undesirable outputs). It\u0026rsquo;s not that AI has become conscious or intelligent. It\u0026rsquo;s just that its creators have shown incompetence and negligence. Period.\nCalls to slow down Faced with AIs that screw up, the leading figures of AI development worldwide, notably the CEOs of OpenAI, Anthropic and xAI, talk about slowing down the pace of development.\n\u0026ldquo;The head of AI company Anthropic has called for the pace of development of artificial intelligence models to slow down and to be closely monitored. (\u0026hellip;) \u0026ldquo;I agree with Dario that we need to pace the frontier,\u0026rdquo; wrote OpenAI CEO Sam Altman on X (\u0026hellip;) Musk, meanwhile, who founded Grok producer xAI, said the Anthropic boss was \u0026ldquo;right\u0026rdquo;.\u0026rdquo;\nAnthropic boss Dario Amodei calls for AI development to slow down\n(BBC, September 12, 2026)\nBut if they had wanted to build their AIs properly in the first place, that alone would have slowed them down. And in fact, if things start going too far off the rails, it won\u0026rsquo;t be worth it anyway. They\u0026rsquo;ll be economically forced to slow down. Jensen Huang, CEO of Nvidia, is among those who defend this position.\n\u0026ldquo;If they believe they are out of control, then don\u0026rsquo;t ship products until they are in control.\u0026rdquo;\nJensen Huang in Jensen Huang Thinks A.I. Alarmism Has Gone Too Far\n(The Ezra Klein Show, September 23, 2026)\nSo these calls to slow down ring hollow. These companies are trying to go as fast as possible, without the damage they know to be inevitable getting them into too much trouble. Especially since they know Chinese companies are hot on their heels. So their enthusiasm for slowing down is mostly rhetorical.\nOn that note, general passivity helps them a great deal. A passivity fed by wonder, by the carefully maintained belief that these companies are doing everything they can, by the idea, shaped by fiction, that engineering flaws are actually proof of intelligence, and of course by the fear of falling behind.\nThat said, Jensen Huang explains in the same interview that this race was first driven by the need to prove that AI could find useful and economically viable applications.\nAnd according to him, that stage has now been reached, notably with products that speed up software development and have made usage explode. So we\u0026rsquo;d soon be entering a phase where the effort would focus more on a kind of quality that belongs to production engineering:\n\u0026ldquo;Finally, we now have a piece of software that is useful. Because it\u0026rsquo;s useful, the adoption took off. (\u0026hellip;) And so, what\u0026rsquo;s going to happen over the next several years is that we\u0026rsquo;re going to transition from these labs becoming engineering focused, much more production engineering focused, and product focused companies.\u0026rdquo;\nJensen Huang in Jensen Huang Thinks A.I. Alarmism Has Gone Too Far\n(The Ezra Klein Show, September 23, 2026)\nIf so, this phase should sooner or later involve a bit more upstream control of the data.\nHow do we make sense of the insiders\u0026rsquo; fear? Let\u0026rsquo;s set aside the marketing talk that capitalizes on incidents to sell some supposed power. In light of everything explained so far, there\u0026rsquo;s still, to my mind, a mystery\u0026hellip;\nIf it\u0026rsquo;s obvious that the danger comes from an engineering flaw in our LLMs, due to an empirical approach (the scruffies\u0026rsquo;) that doesn\u0026rsquo;t bring enough predictability, let alone demonstrability, for uses that do require it\u0026hellip;\nIf it\u0026rsquo;s obvious that this approach has found an economic ally that turned it into an ideology in this race between companies, and that we should therefore be wary of the haste that comes with it and keep an eye out for the signs of an entirely foreseeable loss of control\u0026hellip;\nThen, these insiders who voice their fears, AI researchers in particular:\nWhy don\u0026rsquo;t they talk about an engineering flaw? Which keeps the mystery alive.\nWhy do they see the loss of control as unpredictable? Which keeps the fantasies alive.\nThe example of the 2023 letter There is indeed the 2023 open letter (Future of Life Institute, 2023) calling for a six-month pause. This letter has the merit of actually making arguments, unlike other public statements.\nThe letter mentions, for example, the black box problem and its lack of predictability.\n\u0026ldquo;This does not mean a pause on AI development in general, merely a stepping back from the dangerous race to ever-larger unpredictable black-box models with emergent capabilities.\u0026rdquo;\nPause Giant AI Experiments: An Open Letter (Future of Life Institute, 2023)\nNotice that, even while denouncing it, the authors don\u0026rsquo;t call this unpredictable black box a broken tool, but a source of \u0026ldquo;emergent capabilities\u0026rdquo;. The bug is turned into a feature!\nThe letter also calls for qualities that can be linked to engineering, which I welcome. But it keeps its vocabulary open-ended, focused on containing a multifaceted problem, without naming any flaw. The cause remains quite the mystery!\n\u0026ldquo;AI research and development should be refocused on making today\u0026rsquo;s powerful, state-of-the-art systems more accurate, safe, interpretable, transparent, robust, aligned, trustworthy, and loyal.\u0026rdquo;\nPause Giant AI Experiments: An Open Letter (Future of Life Institute, 2023)\nAlso, notice how strange it is to describe AI as \u0026ldquo;loyal\u0026rdquo;. The hammer has to be loyal to us\u0026hellip;\n\u0026ldquo;AI labs and independent experts should use this pause to jointly develop and implement a set of shared safety protocols for advanced AI design and development that are rigorously audited and overseen by independent outside experts. These protocols should ensure that systems adhering to them are safe beyond a reasonable doubt.\u0026rdquo;\nPause Giant AI Experiments: An Open Letter (Future of Life Institute, 2023)\nHere too, I welcome the commitment to removing any doubt about the safety of our AIs \u0026ldquo;beyond a reasonable doubt\u0026rdquo;. But I still wonder about this vocabulary, still very open-ended, to \u0026ldquo;jointly develop a set of safety protocols\u0026rdquo; to contain\u0026hellip; an engineering flaw that, once again, it doesn\u0026rsquo;t name.\nA flaw that would, by the way, threaten our very civilization!\n\u0026ldquo;Should we risk loss of control of our civilization?\u0026rdquo;\nPause Giant AI Experiments: An Open Letter (Future of Life Institute, 2023)\nJacob Coxon, whom I quoted earlier, was talking about an AI so powerful that it would escape us!\n\u0026ldquo;(\u0026hellip;) These will soon be superhuman systems that can hack anything, (\u0026hellip;) and acquire real power and resources. (\u0026hellip;)\u0026rdquo;\nJacob Coxon in a post on X (September 9, 2026)\nSo, we agree that we need more control. We all agree that we therefore need safety. But\u0026hellip;\nHow come the vocabulary of engineering is so absent?\nPerhaps precisely because engineering isn\u0026rsquo;t the foundation of AI. It\u0026rsquo;s only its applied form.\nAI\u0026rsquo;s foundation isn\u0026rsquo;t engineering AI\u0026rsquo;s real foundation is scientific. It\u0026rsquo;s even empirical, with this very popular scruffy approach. An approach that is the polar opposite of engineering.\nBut, in fact, if you take engineering out of the picture, if you accept the scruffy approach, if you find nothing to criticize in it, if you lose sight of all the answers engineering offers to the problems of losing control, then you can indeed get carried away by lyricism! Except it wouldn\u0026rsquo;t be mysterious, since they wanted it, that black box. And yet\u0026hellip;\nWhy don\u0026rsquo;t the researchers with their hands in the engine change their vocabulary? After all, they see the problems piling up. So, are we sure everyone has really understood the nature of the problem?\nPartly a matter of inexperience? I know, for my part, how much computing is prone to culture wars over how to solve software problems. I\u0026rsquo;ve also seen plenty of qualified people lacking some basic reflexes of their own field. When you\u0026rsquo;re relatively young, I think that happens to everyone. But I\u0026rsquo;ve also heard seasoned developers admit, after 20 years in the business, that they had been wrong about their approach to software. I\u0026rsquo;ll confess, without shame, that I\u0026rsquo;m no exception. It\u0026rsquo;s common. It\u0026rsquo;s normal. Computing is recent and constantly changing. And, in fact, we never stop learning.\nNow, AI in its current form is very young. It requires combined skills in mathematics and computer science. The big labs have a lot of young researchers. And researchers aren\u0026rsquo;t necessarily engineers. So they don\u0026rsquo;t necessarily have an engineer\u0026rsquo;s way of looking at how to build a tool that works. But research, for its part, goes into production anyway! And these same insiders then talk about a possible accident as if it would happen beyond their control\u0026hellip;\nAll of this should precisely lead them to the vocabulary of engineering. They should naturally call for more explainability, more demonstrability. They should point out the limits of the scruffy approach, without resorting to any kind of lyricism. It should lead them to reject the black box upstream, rather than handling problems downstream!\nSo I think we have to admit that, at least in part, there may be some degree of inexperience and misunderstanding. As there is, by the way, in any field. I have the feeling that the GIGO concept, or more generally the idea of building a tool that works, may not have sunk in the same way for everyone.\nAt least, many criticize the haste, granted. But what does that really change?\nWaking up and leaving fatalism behind When you embrace the scruffy approach without criticizing its limits, when you push the very ideas behind this black box, using downstream management as an excuse, even though the goal is to produce useful, and therefore reliable, tools, when you then voice your fears in lyrical terms without naming the cause of the problem, and when, on top of that, you contribute to this haste from inside these big labs, then, in the end, you\u0026rsquo;re complaining about a problem you created yourself.\nIt looks like rushing toward the cliff, while complaining about heading for the cliff in a rush. As Jensen Huang says, I think those who act this way give a very bad image of themselves.\nBut that\u0026rsquo;s not all. This attitude is seriously harmful for another, more important reason, one that actually motivated me to write this essay.\nWhen these insiders don\u0026rsquo;t talk about what should be done, they reinforce fatalism in our collective understanding, along with the idea that AI is inherently a beast to be contained.\nWe need to shake ourselves up, intellectually and collectively, about our relationship with AI.\nBecause if we don\u0026rsquo;t shake ourselves up a little, we reward the approaches that accept that AI can\u0026rsquo;t be anything other than this black box with emergent capabilities.\nBecause as soon as we accept that, then, yes\u0026hellip; we can allow ourselves the wildest scenarios about the future of humanity. The absurd Superintelligence, the AI we would have accepted would escape our intellect, would then become a defensible scenario in some people\u0026rsquo;s eyes.\nAs far as I\u0026rsquo;m concerned, the only nuclear bomb I see in AI is the foolishness we would show by designing it that way.\nConclusion In AI, structural changes open new doors. For LLMs, for example, adding a recursion at inference lets models \u0026ldquo;think\u0026rdquo;. Adding a recursion that drives self-training would make it possible to speed up model improvement. Other changes might even solve the control difficulties inherent to LLMs. However, this essay focuses precisely on a scenario where control is undermined.\nI put forward the theses that a recursion driving a particular kind of self-training, on the fly, could bring great adaptability, but also the emergence of a form of acquired consciousness. That is, a form of strong AI combined with a form of AGI.\nFollowing this scenario, I work through a number of questions, to put into perspective some recurring concerns and demystify the real origin of the problems in our AIs.\nThe emergence of consciousness, for example, has nothing to do with intelligence. It actually requires giving up sufficient human oversight. That is, giving up sufficient control over the inputs, outputs and objectives of the model we train, for the use we intend it for. This mechanically implies a loss of predictability in that model\u0026rsquo;s behavior. Such an AI would therefore be a broken tool.\nMore generally, it\u0026rsquo;s through this insufficient control over data and objectives that we create AI\u0026rsquo;s undesirable and supposedly mysterious behaviors. This insufficient oversight is the loss of control. We no longer know what we\u0026rsquo;re getting.\nSo far, our AIs still aren\u0026rsquo;t conscious. But if we had to weigh the gain of great adaptability against a loss of predictability due to the consciousness that comes with it, as this essay\u0026rsquo;s scenario suggests, then it wouldn\u0026rsquo;t be worth it. Because a tool that starts thinking for itself is no longer a tool.\nIn the meantime, such a scenario hasn\u0026rsquo;t happened yet. But in the public sphere, faced with engineering flaws that go unnamed, mystery wins out and fiction does the rest. The worst-case scenario, where a conscious AI rebels, requires so many \u0026ldquo;ifs\u0026rdquo; to come true that it belongs mostly to fiction. It diverts attention from problems that are less spectacular but very real, like hacks, and most likely other accidents to come. To bring back some rationality, this essay argues for remedies.\nTo fight mystery and fear, I argue for the vocabulary of engineering to take over and dispel the fantasies.\nTo fight engineering flaws and their consequences, I argue throughout this essay for more ambitious oversight of LLMs, one that brings more demonstrability to training.\nFinally, no, AI is not like the nuclear bomb. The bomb is dangerous in itself. The real dangers of AI, on the other hand, lie first in haste and the lack of rigor that comes with it. None of them leads us to the end of the world. However, I can easily believe that the pursuit of short-term profit could push us all the way to a serious accident. That possibility, too, isn\u0026rsquo;t taken as seriously as the danger it represents.\nSo I conclude that the same observation still holds, again and again, our tendency to look away. We prefer mystery to engineering flaws, we prefer a hypothetical end of the world to very real dangers, we prefer a stock price to a job well done\u0026hellip; just as we prefer profit to fighting climate change. So it\u0026rsquo;s not AI that I find frightening, but ourselves.\nFinal word I hope this essay will help others better understand these questions, which are generally poorly discussed in the public sphere.\nI\u0026rsquo;m well aware that I had several audiences in mind while writing this essay. I wanted to explain things to a general audience, but also to put this thesis of on-the-fly self-training to insiders. Exchanging explorable scenarios matters if we want to understand how undesirable effects show up and what they depend on.\nI\u0026rsquo;ll be writing follow-up articles to this one. But they\u0026rsquo;ll be much shorter, and probably less controversial. I\u0026rsquo;d also like to write about different topics, showing AI from other angles. For example, I\u0026rsquo;d like to talk about how it fits into the economy.\nFeel free to get in touch. I\u0026rsquo;m very approachable. I answer questions.\nFinally, if you\u0026rsquo;re also fed up with hearing nonsense about AI, while the real problems lie elsewhere, share this essay.\nUpdates This essay makes no claim to truth or perfection. Many of the topics covered here are open and up for discussion. Criticism, objections and corrections are therefore welcome.\nI\u0026rsquo;ll keep a record of the changes made to the essay, as well as of the exchanges that I think enrich our collective thinking. I\u0026rsquo;ll date the changes and credit the people behind them, if they allow me to.\nFinally, please note that I am not listing minor corrections that do not affect the substance, such as a spelling error, a missing link to a heading, or an improvement to the English translation intended to better capture the spirit of the essay. However, for the sake of full transparency, I can provide a list of all Git patches to anyone who asks.\nOctober 26, 2026: Clarified the use of the terms \u0026ldquo;human oversight\u0026rdquo;, as the means of end-to-end control, and \u0026ldquo;demonstrability\u0026rdquo;, as an intent to require guarantees.\nReferences References are grouped by theme. The links are the ones cited in the essay.\nReasoning and self-training Behrouz et al. (2024). Titans: Learning to Memorize at Test Time. arXiv. I.J. Good (1965). Speculations Concerning the First Ultraintelligent Machine. Advances in Computers. Liu et al. (2022). Monolith: Real Time Recommendation System With Collisionless Embedding Table. arXiv. Shumailov et al. (2024). AI models collapse when trained on recursively generated data. Nature. David Silver et Richard Sutton (2025). Welcome to the Era of Experience. Wei et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv. Lilian Weng (July 2026). Harness Engineering for Self-Improvement. Lil\u0026rsquo;Log. Eliezer Yudkowsky (2008). Recursive Self-Improvement. LessWrong. Zelikman et al. (2022). STaR: Bootstrapping Reasoning With Reasoning. arXiv. Zweiger et al. (2025). Self-Adapting Language Models. arXiv. Consciousness and definitions of AI Nick Bostrom (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. Morris et al. (2023). Levels of AGI. arXiv. OpenAI (2018). OpenAI Charter. John Searle (1980). Minds, Brains, and Programs. The Behavioral and Brain Sciences. Sofroniew et al. (2026). Emotion Concepts and their Function in a Large Language Model. arXiv. Safety, control and risks Bengio et al. (2025). Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?. arXiv. Betley et al. (2025). Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. arXiv. Nick Bostrom (2003). Ethical Issues in Advanced Artificial Intelligence. Dalrymple et al. (2024). Towards Guaranteed Safe AI. arXiv. European Union (2024). Regulation (EU) 2024/1689 on artificial intelligence (AI Act). EUR-Lex. Future of Life Institute (2023). Pause Giant AI Experiments: An Open Letter. Korbak et al. (2023). Pretraining Language Models with Human Preferences. arXiv. Schlatter et al. (2025). Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs. arXiv. Lee Vinsel (2021). You\u0026rsquo;re Doing It Wrong: Notes on Criticism and Technology Hype. Medium. AI cultures and explainability Ali et al. (2023). Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence. Information Fusion. Leo Breiman (2001). Statistical Modeling: The Two Cultures. Statistical Science. Cynthia Rudin (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence. Richard Sutton (2019). The Bitter Lesson. Reference concepts and facts (Wikipedia) AI-induced psychosis AlphaGo Zero Instrumental convergence Mars Climate Orbiter P(doom) Statement on AI Extinction Risk Tay (chatbot) Press, videos and public statements Google fires engineer who contended its AI technology was sentient (CNN, July 25, 2022) Artificial intelligence could lead to extinction, experts warn (BBC, May 30, 2023) \u0026lsquo;Humanity\u0026rsquo;s remaining timeline? It looks more like five years than 50\u0026rsquo;: meet the neo-luddites warning of an AI apocalypse (The Guardian, February 17, 2024) The Singularity Is Nearer (Ray Kurzweil, 2024) Here\u0026rsquo;s how strange 2050 will be, according to the world\u0026rsquo;s leading AI expert (BBC Science Focus, August 23, 2024) AI Doom? No Problem. (WSJ, October 3, 2025) BCacheFS Developer Deployed His AI Girlfriend Helper (Brodie Robertson, February 26, 2026) Exclusive: US commanders bypassed warnings about outdated intelligence ahead of strike that hit school in Iran, sources say (CNN, July 7, 2026) Unexpected chat between OpenAI agents led to Hugging Face hack (BBC, August 26, 2026) Jacob Coxon\u0026rsquo;s post on X (September 9, 2026) Anthropic boss Dario Amodei calls for AI development to slow down (BBC, September 12, 2026) Cybersecurity Has An AI Doomer Problem (Marcus Hutchins, September 15, 2026) The Hugging Face Hack Wasn\u0026rsquo;t What It Was Cracked Up to Be (WSJ, September 17, 2026) Will AI really kill everyone? How, exactly? (CNN, September 17, 2026) Exclusive: US military had close call after using AI for false intelligence report, sources say (CNN, September 18, 2026) I asked Meta\u0026rsquo;s Muse for its filesystem and it sent me 6.8 GB (Peter James, September 22, 2026) Jensen Huang Thinks A.I. Alarmism Has Gone Too Far (The Ezra Klein Show, September 23, 2026) Address by Leo XIV to UNESCO (September 25, 2026) Bill Gates: A.I. \u0026lsquo;Makes Nuclear Weapons Look Like Nothing\u0026rsquo; (The Ezra Klein Show, September 29, 2026) ","permalink":"https://www.barnabebalp.fr/en/posts/ai/recursion-self-training-and-control/","summary":"\u003cp\u003e\u003cem\u003e\u003cstrong\u003eNone of my articles are written by AI.\u003c/strong\u003e The ideas are entirely mine, and so are the words. I write in French first, then translate into English. AI is only used to proofread, fact-check and help with the translation.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003cstrong\u003eThis essay is long\u0026hellip; and that\u0026rsquo;s my fault!\u003c/strong\u003e The subject turned out to branch in many directions, and I didn\u0026rsquo;t want to leave readers with open questions. \u003cstrong\u003eSo I tried to make each chapter stand on its own.\u003c/strong\u003e Feel free to jump straight to what interests you.\u003c/em\u003e\u003c/p\u003e","title":"Recursion, Self-Training, and Control"},{"content":"I offer my expertise in consulting and software development, specializing in:\nLanguages: C++ (11 to 26), Python Technical domains: Systems, Applications, AI, Scientific Computing, HPC, R\u0026amp;D Methodologies: Agile, SAFe, V-cycle Roles: Lead Developer, Software Architecture, Industrialization I started my independent activity as a C++ trainer in computer science schools and companies. Since then, I have had the opportunity to work in passionating industrial sectors.\nContact me by email to discuss your needs.\nExperience Here is an overview of the domains and subjects I have worked on.\nSpace Multiple experiences in developing satellite image processing pipelines and real-time event processing systems for control rooms.\nWater Sports and Gliding Design and development of a flight calculator dedicated to professionals in water sports and gliding.\nCybersecurity Design and development of an architecture for a high-performance and autonomous cybersecurity solution.\nSolar Energy Development of solar production simulation solutions for industrials and architects to assess the viability of their projects.\nArtificial Intelligence Adaptation, integration, and deployment of a vocal processing AI within an application.\nSide Projects and Commitments My profession is my passion, and I am committed to contributing to the improvement of the ecosystem I depend on.\nThe C++ Language C++ is a fundamental building block of all modern software infrastructure.\nTo actively contribute to this ecosystem, I launched Golem, to modernize build practices, and improve software security and developer productivity. I also work to raise awareness about standardization efforts and best development practices.\nArtificial Intelligence Understanding the functioning of AI through the prism of information processing helps in grasping its potential and its limitations. I plan to address a number of AI-related topics on this site.\nCreativity and Entrepreneurship Entrepreneurship is for me a way to share an idea, a solution I believe in. I have projects that I hope to concretize and share publicly.\nContact For any consulting, development, or collaboration request, feel free to contact me by email.\nCrédits Photos by USGS (Unsplash), Jairph (Unsplash), Joel Rivera-Camacho (Unsplash), American Public Power Association (Unsplash), Gabriele Malaspina (Unsplash).\n","permalink":"https://www.barnabebalp.fr/en/services/","summary":"Expertise in consulting and software development: Modern C++, AI, architecture, HPC","title":"Services"},{"content":"Site Publisher Name: Barnabé BALP Status: Sole Proprietorship SIRET: 524 170 586 00043 Address: 47 Rue Vivienne, 75002 Paris Phone: 07 83 69 14 38 Email: webmaster@barnabebalp.fr Website: https://barnabebalp.fr Hosting Host: Scaleway Address: 8 Rue de la Ville l\u0026rsquo;Évêque, 75008 Paris, France Website: https://www.scaleway.com/ Third-Party Licenses and Content Site Theme Name: PaperMod Author: Aditya Telange License: MIT License Source: GitHub - hugo-PaperMod Fonts Name: Inter (InterVariable) Author: Rasmus Andersson (RSMS) License: SIL Open Font License Source: rsms.me/inter Background Image Author: Fernando Puente Source: Unsplash License: Unsplash License (free for commercial and non-commercial use, no attribution required) Publishing Director Barnabé BALP – webmaster@barnabebalp.fr\nIntellectual Property The original content on the barnabebalp.fr website (texts, articles, graphics, logos) is the exclusive property of Barnabé BALP.\nFor licensed elements (theme, fonts, background image), please refer to the Third-Party Licenses and Content section.\nAny reproduction or reuse of this original content is prohibited without prior written permission, in accordance with articles L. 335-2 and following of the French Intellectual Property Code.\nLimitation of Liability The information on this site is provided for informational purposes only and is not binding. Barnabé BALP cannot be held responsible for any direct or indirect, material or immaterial damage resulting from access to, use of, or inability to use the site, or the information contained therein.\nThe user acknowledges having the competence and necessary means to access and use this site. Barnabé BALP cannot be held responsible for malfunctions of the Internet network or technical means related to the connection to the site.\nPersonal Data No personal data is collected by this site.\nHowever, this site occasionally integrates content from third-party platforms. These services may collect data according to their own privacy policies, which we invite you to consult:\nGitHub - Privacy Statement Ko-fi - Privacy Policy YouTube - Privacy No cookies: This site does not use any tracking or analytical cookies.\nApplicable Law and Jurisdiction These legal notices are governed by French law. Any dispute relating to the use of the barnabebalp.fr site shall be subject to the competent courts of Paris.\n","permalink":"https://www.barnabebalp.fr/en/legal-notice/","summary":"Legal information about the barnabebalp.fr website","title":"Legal Notice"},{"content":"General Principles This document explains how information is processed when you visit barnabebalp.fr.\nIn summary: This site does not collect any personal data and does not use any tracking cookies.\nData Collected No Personal Data is Collected This site is static and does not contain:\nAny contact forms Any comment systems Any tracking tools (Google Analytics, Matomo, etc.) Any session or tracking cookies Third-Party Embedded Content This site may display content hosted by external platforms:\nService Content Type Privacy Policy GitHub Repository links, Sponsor button GitHub Privacy Statement Ko-fi Sponsor button Ko-fi Privacy Policy YouTube Embedded videos Google Privacy Policy Interaction with this content is subject only to the policies of these platforms. We have no access to any data they may collect.\nHosting This site is hosted by Scaleway:\nAddress: 8 Rue de la Ville l\u0026rsquo;Évêque, 75008 Paris, France Website: https://www.scaleway.com/ Access logs (IP addresses, user agent, pages visited) may be recorded by the host in accordance with its legal obligations and privacy policy: Scaleway - Privacy Policy\nRetention period: These logs are generally kept for 13 months (French legal requirement).\nCookies This site does not use any cookies.\nNo session, tracking, advertising, or analytical cookies are placed on your browser when visiting this site.\nExternal Links This site may contain links to external sites (Mastodon, Matrix, GitHub, Codeberg, etc.). We are not responsible for the privacy practices of these third-party sites. We recommend that you consult their respective policies.\nYour Rights In accordance with the General Data Protection Regulation (GDPR) and French law \u0026ldquo;Informatique et Libertés\u0026rdquo;, if you believe your rights are not being respected, you may:\nContact us directly at webmaster@barnabebalp.fr File a complaint with the CNIL (Commission Nationale de l\u0026rsquo;Informatique et des Libertés): https://www.cnil.fr Changes This privacy policy may be updated occasionally. The date of the last update will be indicated at the top of this page.\nLast updated: May 23, 2026\n","permalink":"https://www.barnabebalp.fr/en/privacy-policy/","summary":"Information regarding the protection of your personal data","title":"Privacy Policy"},{"content":"I am the creator and maintainer of Golem (build system for C++), a freelancer, specializing in software development, passionate about my work and much more.\nEmails Talk: talk@barnabebalp.fr Business: business@barnabebalp.fr Social Mastodon: https://piaille.fr/@bbalp Matrix: https://matrix.to/#/@bbalp:matrix.org GitHub: https://github.com/bbalp/ Codeberg: https://codeberg.org/bbalp Linkedin : https://fr.linkedin.com/in/barnabebalp/en Bio My training I have a degree in computer science, specializing in software development, particularly in systems programming and scientific computing.\nI completed AI training, offered by the CNRS (Fidle 2022-2023, French scientific research center), designed for scientists and engineers. Since then, I have been following the evolution of scientific research in this field.\nI am committed to improving the ecosystem surrounding C++, raising awareness of ISO standardization work and new development practices.\nI have worked on the challenges related to adapting and industrializing AI. My strong technical understanding allows me to raise public awareness of the practical realities of its use, beyond the marketing rhetoric aimed at boosting the stock market value of the tech giants developing it.\nMy convictions I am committed to promoting free software, raising awareness of the issues surrounding digital sovereignty, data privacy, and cybersecurity.\nI believe we need software that serves our interests, puts us in control of our destiny, and guarantees our privacy. On a small scale, I dedicate my free time to this.\nMoving forward My work fascinates me, but I am also fascinated by our world, and by ourselves. I am passionate about economics, the challenges of democracy, geopolitics, entrepreneurship, science, and art. I am as fascinated by the evolution of life and the formation of our Universe as I am by the way in which this group of individuals, Humanity, imagines, as best it can, solutions to move forward.\nWhen we share the same space, moving forward means recognizing that our destinies are all intertwined. I therefore believe in the necessity of cultural exchange and the democratization of access to a neutral internet.\nMichel Serres, philosopher and historian of science, identified four major technological revolutions: spoken language, writing, printing, and computing. According to him, these developments in information media have profoundly transformed our relationship with the world, our way of thinking and communicating, marking major cultural and cognitive shifts. I recommend his presentation at INRIA in 2007 (FR) on ​​this subject.\nI strongly believe in his analysis and would add that the Internet plays a central role in this fourth revolution. Because these four technological milestones are revolutions based on the exponential sharing of knowledge. What distinguishes humankind from other species is its capacity to accumulate knowledge by transmitting it from generation to generation.\nTherefore, if moving forward means facilitating our exchanges and our means of transmitting knowledge, then AI is now seen as becoming this fifth revolution, reducing language barriers and further democratizing universal access to this knowledge. But if we want its democratization and neutrality, if we want it to serve the interests of all of us, then we must not leave it in the hands of a few. There is a general interest at stake here, and when there is a general interest, free software is never far away.\n","permalink":"https://www.barnabebalp.fr/en/about/","summary":"Learn more about me and how to contact me","title":"About Me"},{"content":"I am the creator and maintainer of Golem (build system for C++).\nI am committed to promoting free software and raising awareness about the issues of digital sovereignty, data privacy, and cybersecurity.\nI don\u0026rsquo;t need donations to motivate my contributions. However, donations strengthen my independence and allow me to do more. So if you like my projects and my approach, please consider leaving something.\nSupport Me GitHub \u0026nbsp; \u0026nbsp; Ko-fi \u0026nbsp; \u0026nbsp; Wire Transfer \u0026nbsp; \u0026nbsp; IBAN Name: BIC: IBAN: Initiatives I Support Donations are a way to support initiatives that protect our interests. Here is a list of projects to which I donate, or have donated, and whose importance I always commend:\nCommet is a Matrix client offering a user experience similar to Discord. Framasoft is a fantastic association promoting the use of Free Software and raising awareness of free alternatives. NixOS is a Linux distribution with declarative configuration and a pure functional language. I also donate or have donated to other initiatives such as independent media outlets and other projects unrelated to my profession.\n","permalink":"https://www.barnabebalp.fr/en/donate/","summary":"To support the projects I\u0026rsquo;m working on or the ideas I\u0026rsquo;m advocating","title":"Donate"},{"content":"Here is a list of projects that I have created or am actively working on.\nGolem Build System for Modern C\u0026#43;\u0026#43;\nGolem - Recipes Recipes for Golem\nFractalDX9 Compute fractals on GPU but without CUDA\n","permalink":"https://www.barnabebalp.fr/en/projects/","summary":"List of projects that I have created or am actively working on","title":"Projects"}]