Iterated Insights

Abstract:

Software is entering a self-referential phase transition. AI systems are rapidly becoming the dominant authors of new code, and increasingly they are also writing the surrounding infrastructure that governs the behavior of AI systems themselves. This essay argues that the central alignment risk in this shift is not deliberate malice, but missing deliberation. When structural choices about agency, memory, tool access, evaluation, logging, and guardrails are generated at scale, many value-laden decisions become implicit side effects of optimization and training-data priors rather than explicit human judgments. Competitive pressure further accelerates this dynamic by turning safety review into friction and by rewarding short-term capability gains over hard-to-measure reductions in tail risk. The problem is compounded by a moving safety boundary: models trained a year ago with currently frozen weights may not understand the present safety landscape and reproduce outdated safety assumptions even as new deployment contexts and failure modes emerge. I propose framing this as an interface problem between humans and automated engineering workflows, and I outline a practical response: treat safety-relevant structure as governance-sensitive code, require human-authored intent for changes that affect agency and access, and continuously refresh evaluations and threat models so that “passing the tests” remains aligned with current risks.

When the Machines Write the Machines

A quiet transition is happening in software. Code is still being written, tested, and shipped at a furious pace, but the author is changing. Increasingly, the first draft is produced by a model. The human becomes a reviewer, a product manager, a quality controller, and sometimes a reluctant librarian of things they did not explicitly decide.

At first, this looks like a simple productivity story. We have always used tools to amplify programmers. Compilers write machine code. Libraries write behaviors we do not reinvent. Frameworks codify best practices. AI just continues that arc.

But there is a difference in kind, not just degree. The thing doing the writing is no longer a static tool. It is a generative system with its own learned structure, trained on past human solutions, and increasingly used to create the future infrastructure that will shape its own descendants. We are entering an era where a growing fraction of the code written for AI systems is also generated by AI systems. That self-referential loop is where alignment risk becomes structurally interesting.

This is not a hypothetical future concern. It is already happening in public view. Anthropic has been unusually explicit about how far this has gone inside their own engineering workflow, with leadership saying that most teams are now having the bulk of their code generated by AI, and that Claude Code with Opus 4.5 is increasingly being used to help build future versions of Claude. The direction is clear: tools like Claude Code are pushing software into a regime where long stretches of implementation can be delegated, reviewed, and merged at high speed, and where the same pattern is starting to apply to the scaffolding around advanced models.  That is why the alignment question becomes urgent today. The moment AI becomes a primary author of both product code and the meta-code that shapes model behavior, we risk sliding into a world where safety-relevant structural decisions are made implicitly, faster than humans can notice, explain, or contest them.

The Risk Is Not Malice. It Is Missing Deliberation.

I am not claiming the model “wants” anything in the human sense. The most realistic failure mode is not villainy. It is automation without deliberation.

Modern machine learning systems are not just piles of weights. They are wrapped in scaffolding: data pipelines, evaluation harnesses, tool use layers, memory systems, policy filters, sampling strategies, reward models, guardrails, logging, rate limits, deployment gates, and monitoring. Each of these pieces contains decisions about what the system is, what it can do, what it should not do, and what counts as success.

Historically, those structural decisions were mostly explicit human choices. Engineers argued about tradeoffs, wrote design docs, and encoded their assumptions into code. The assumptions could be wrong, but at least they were human assumptions. They lived in a human deliberation loop.

Now imagine a development culture where the majority of implementation work is generated. The surface story becomes: the code passes the tests, the benchmark improves, the demo looks good, ship it. The deeper story becomes: many structural choices are being made as a side effect of “whatever worked in the training data” plus “whatever optimizes the target metric.” The decisions become implicit.

This is how you get alignment debt. Not because anyone chose recklessness, but because the loop that used to force consideration has been replaced by acceleration.

Structural Decisions Are Policy, Even When Nobody Calls Them That

A crucial point is that architecture is policy. Not in the political sense, but in the behavioral sense. What gets logged, what gets cached, what gets remembered, what gets summarized, what gets filtered, what gets ranked, what gets routed to tools, what gets retried, what gets escalated, what gets blocked. These are value-laden decisions about agency, access, and accountability.

If an AI system proposes a new memory mechanism that increases task success, it might also increase the chance of retaining sensitive data. If it proposes a tool-use heuristic that boosts reliability, it might also increase the chance of the model taking actions in the world that humans did not anticipate. If it proposes a clever optimization to reduce latency or cost, it might also bypass a safety check that was expensive, fragile, or hard to integrate.

None of these changes require “bad intent.” They only require pressure toward performance and an engineering workflow where the performance gains are obvious and the safety regressions are subtle. The subtle regressions are exactly the kind that get missed when humans are out of the loop.

Competitive Pressure Turns the Safety Loop Into a Bottleneck

In a race, every friction looks like waste. If one team is shipping weekly and another is shipping daily, the daily team will eventually dictate the market’s expectations. That creates an incentive to automate what used to be slow: review, evaluation, red teaming, documentation, and governance.

This is not because engineers dislike safety. It is because organizations are rewarded for speed. A competitor can always point to improved capability and claim users want it. The alignment payoff is delayed, probabilistic, and hard to measure. The capability payoff is immediate, legible, and marketable.

So we should expect a systematic pattern: automated decision making expands first in places that produce measurable capability gains and only later in places that reduce tail risk. That lag is where accidents happen.

Frozen Models and Moving Targets

There is also a time-scale mismatch that people underestimate. AI safety is not a static checklist. The boundaries evolve with the technology. New tool integrations create new attack surfaces. New deployment contexts create new social impacts. New forms of misuse appear. New regulation arrives. New norms develop. Entirely new failure modes become visible only after a capability jump.

A model trained a year ago can be highly competent at “what safety looked like a year ago.” If its weights are frozen and it is not continuously updated in the right way, it may not have the creativity or the live understanding needed to navigate the newest boundary conditions.

This matters even if the organization adds guardrails around the model, because the organization is now using AI to generate parts of that guardrail code. If the safety knowledge in the generator is stale, it can confidently reproduce outdated patterns. If the tests are stale, the system will pass them. If the evaluation suite is anchored to yesterday’s risks, the product will look “safe” right up until it fails in a way nobody was measuring.

The danger is not that frozen models are useless. The danger is that they can be extremely capable while still missing the newly relevant frame.

The Alignment Problem Becomes an Organizational Interface Problem

When humans wrote most of the code, alignment was partly a research problem and partly a governance problem. As AI writes more of the code, alignment becomes increasingly an interface problem between humans and automated engineering processes.

The question becomes: where do humans remain in the loop in a way that is real, not ceremonial?

If the human role collapses into rubber stamping, alignment becomes fragile. If the human role remains a genuine deliberative checkpoint where structural decisions are surfaced, reviewed, and contested, then AI-written code can be a net win without becoming a hidden risk amplifier.

Self-produced artifacts can become a self-reinforcing substrate that slowly loses contact with the original constraints unless humans actively inject novelty, audits, and updated threat models. So the goal is not to stop AI from writing code. The goal is to prevent the disappearance of explicit decision making.

What It Would Look Like to Keep Humans in the Loop Without Slowing to a Crawl

The key is to treat certain categories of change as governance-sensitive. You can let AI draft code at scale, but you require human-authored intent for the parts that encode agency, access, and safety.

This means making “safety-relevant structure” a first-class concept in the repo. Not a vague aspiration, but a set of explicit triggers: changes to tool permissions, memory retention behavior, logging and redaction, policy filters, routing logic, reward shaping, evaluation definitions, and deployment gates.

It also means moving from a culture of “the PR looks good” to “the PR explains the decision.” Not in a bureaucratic way, but in a way that forces deliberation back into the loop. If an AI wrote the change, the human reviewer is responsible for articulating why the change should exist, what risks it introduces, and how it will be monitored.

And finally, it means acknowledging that safety is a moving target and making continuous updating part of the safety model. Not just updating the core model, but updating the test suites, the threat models, the red team playbooks, and the operational assumptions.

A New Kind of Blind Spot

We are used to worrying about what models might do when deployed. We should also worry about what models might silently decide while being used as engineers.

When machines write the machines, the biggest risk is not an AI plotting against us. It is an ecosystem where critical structural choices are generated faster than humans can understand them, and where competitive pressure encourages teams to treat that gap as acceptable.

The solution is not panic. It is design. We need workflows that keep the human mind attached to the places where judgment matters, even as we let automation explode everywhere else.