12. The Distance Between an Order and a Death

Decisions before the trigger

Imagine a room far from a battlefield. On a screen, vehicles appear as symbols. A system recommends a target. An operator can accept, reject, or request more information. The display looks orderly. The people represented by the symbols occupy a world of dust, noise, interrupted communications, uncertain intentions, and bodies that can be injured only once.

This is a hypothetical scene, not a report about a particular conflict. Its purpose is to expose a distance that matters even when no one in the room is malicious. A decision can become easier to execute while remaining difficult to justify. An interface can make an action feel precise without making the underlying judgment equally secure.

The phrase autonomous weapons often brings to mind a machine making every decision for itself. That image can obscure the many decisions already made by people: which system to buy, what information to trust, which target categories to permit, where to deploy, how long to operate, and what to do when communication fails. Autonomy at one point in the sequence does not erase human choices elsewhere.

Nor does the presence of human choices prove that the resulting control is adequate. A person can authorize a broad process without understanding the particular force it will apply. We need to examine what has been delegated and whether the remaining decisions still provide a meaningful connection between judgment and consequence.

The International Committee of the Red Cross describes autonomous weapon systems in terms of selecting and applying force to targets without further human intervention after activation. That functional description focuses attention on the point at which a machine’s processing determines an application of force. It does not mean that every remotely operated weapon is autonomous, or that every military use of AI performs this function.¹

These distinctions are essential. A remotely piloted aircraft can remain under direct human control over a strike. An AI system may help with logistics, translation, maintenance, or medical evacuation without choosing a target. A decision-support tool may propose a target while a person retains the formal final decision. Each arrangement raises questions, but they are not identical questions.

Decision support deserves particular scrutiny because it can transfer practical authority without changing the official organization chart. If the system produces recommendations faster than operators can examine them, and the institution expects rapid approval, the human role may shrink to endorsing the machine’s selection. Calling the result a human decision does not explain how much judgment actually occurred.

The central moral issue is therefore more specific than whether computers should be used in war. It is whether people can remain responsible for an application of force when the process makes its grounds, limits, or likely effects difficult for them to understand and govern.

A signature at the beginning of a process cannot authorize every unforeseeable consequence simply by existing. Yet responsibility cannot be assigned only to the person nearest the final action. That person may have inherited a system, a tempo, and a set of constraints designed elsewhere. Serious accountability follows the chain in both directions.

The question is not merely who pressed a button. It is who made the button consequential, what judgment it was supposed to represent, and whether the conditions allowed that judgment to be made.

Taking the protection argument seriously

The strongest case for military automation is not that machines are impressive. It is that some functions may protect people better than the available alternatives. A system could detect an incoming threat quickly, reduce exposure during a dangerous task, or help identify an error before force is used. If those benefits can be demonstrated, they deserve moral weight.

Consider a hypothetical defensive installation facing incoming munitions. The interval between detection and impact may be too short for a person to approve each response individually. A carefully bounded automated defensive function could plausibly save lives. The relevant judgment would include the environment, the objects it may engage, the consequences of an incorrect classification, and the ability to confine its operation.

Compare that with a hypothetical system sent into a populated area to identify and attack people whose behavior fits a broad profile. The difficulty of interpreting conduct, recognizing changed circumstances, and protecting those who must not be attacked is much greater. Calling both systems autonomous tells us something useful, but it does not establish that they should receive the same authorization.

This comparison is conceptual rather than an endorsement of a named product. It shows why the use, environment, and limits matter alongside the technical label. A technology can perform acceptably within a narrow setting and become unacceptable when its mission expands. A successful demonstration in one environment does not grant permission to operate everywhere.

There is also a serious argument from human failure. People can be exhausted, frightened, vindictive, distracted, or prejudiced. A system does not become morally inferior merely because it lacks a human face. If a carefully tested tool reduces a particular kind of preventable harm, insisting on unaided human performance could itself be irresponsible.

But the comparison must be honest. The alternative to an automated system is not always an exhausted individual making an immediate decision with inadequate information. Other alternatives may include delaying action, improving intelligence, changing the mission, using a less harmful method, or declining the operation. A favorable comparison with the weakest imaginable human arrangement does not establish necessity.

The accounting must also extend beyond harm per engagement. Suppose, hypothetically, that a system reduces the likelihood of an error in each use but makes force so inexpensive and convenient that it is used far more often. Its total effect could differ from its performance in a single encounter. This is not a prediction that automation necessarily increases war. It is a reason to ask whether tactical improvement changes political incentives.

Protection of one’s own personnel is a legitimate concern. It does not remove duties toward civilians or make every transfer of risk to others acceptable. A decision that is safer for an operator thousands of miles away may remain dangerous for the people among whom the weapon acts. Moral assessment has to see both positions.

The same is true of claims about precision. The ability to strike exactly where directed is valuable only after the direction has been justified. Precise force against a wrongly identified person remains a grave failure. A narrow physical effect cannot compensate for a careless account of who may be attacked.

For these reasons, a protective claim should arrive with evidence specific to the proposed use. What harm is expected to decrease? Under which conditions? What new failure becomes possible? How will independent reviewers distinguish genuine improvement from favorable test conditions? The goal is to make the strongest argument for protection answer the strongest questions about it.

Human control that can change the outcome

Meaningful human control is an attractive phrase. It becomes useful only when translated into conditions under which a person’s judgment can affect what happens.

One condition is understanding. The operator does not need to know every mathematical detail of a system, just as a pilot need not manufacture an engine. But the operator needs an adequate grasp of its purpose, limitations, relevant uncertainty, and behavior under foreseeable changes. If a system can produce recommendations for reasons nobody can evaluate in the circumstances of use, the institution must ask whether that opacity is compatible with the decision being delegated.

A second condition is time. An interface may offer a reject button while giving the operator too little time to assess the recommendation. The button supplies formal participation without substantive choice. Where events move too quickly for intervention, the earlier decision to activate the system carries more weight and requires tighter limits on what can follow.

A third condition is authority. A person who can technically stop a system but expects punishment for doing so may be less able to exercise independent judgment than the procedure suggests. Institutions need rules that support interruption when conditions fall outside the approved use. The person raising uncertainty should not have to prove that a disaster will occur before being allowed to prevent a poorly understood action.

A fourth condition is effective reach. An instruction to stop has little value if it cannot arrive in time or if the system continues operating after losing communication. Communication failures, ambiguous commands, and changes in mission conditions should be considered before deployment. An emergency intervention mechanism cannot be the only protection when the emergency may disable the mechanism itself.

These conditions apply across stages. A commander may exercise control by imposing limits before activation. An operator may exercise it during use. An engineer may make those limits technically enforceable. A reviewer may prevent a system from entering service until its behavior is adequately established. The result depends on how these roles fit together, not on placing a human somewhere in a diagram.

Consider a hypothetical system authorized to engage a particular kind of military equipment inside a defined area for a short period. During that period, civilians enter the area unexpectedly. A credible control arrangement would identify what the system is capable of recognizing, what information reaches operators, and what action follows. A written statement that the system respects civilians is not an answer to those operational questions.

There is a related danger in multiplying information while reducing comprehension. An operator can receive many indicators without understanding why the recommendation is reliable. A polished explanation may be generated after a decision rather than faithfully report its actual basis. What matters is whether the explanation supports a real assessment, not whether it sounds professional.

Institutions should therefore test the human and machine arrangement together. Can people recognize the kinds of errors the system makes? Do they become overconfident after long periods without visible problems? Can they identify when the conditions of use have changed? Do exercises include cases in which the correct action is to stop, wait, or decline?

These questions express the author’s practical standard for responsibility. They do not claim that human control is easy or that any single mechanism guarantees lawful conduct. They insist that responsibility must have the conditions it needs to function.

A person should be able to explain the decision in terms more substantial than the system’s confidence. If no one can connect the evidence, the permitted purpose, the expected effects, and the choice to proceed, adding another approval box will not close the gap.

The law is already present

Discussion of new weapons can create the impression that technological novelty places conduct outside existing law. It does not. International humanitarian law regulates the conduct of hostilities, including rules on distinction, proportionality, and precautions. Distinction requires separating those who may lawfully be targeted from protected civilians and separating military objectives from civilian objects. Proportionality prohibits attacks expected to cause incidental civilian harm excessive in relation to the concrete and direct military advantage anticipated. The legal assessment concerns the applicable rules and circumstances, not merely the technology’s label.²

Those principles should not be reduced to a simple algorithmic score. Whether an object qualifies as a military objective involves the circumstances. Whether someone is protected can depend on facts that change. Expected civilian effects and military advantage require a judgment before the action, based on the information reasonably available. A numerical output may help organize information, but it cannot relieve the responsible people and institutions of the need to make and justify the assessment.

Nor should this account be misunderstood as a claim that every civilian death automatically proves a war crime, or that compliance becomes established whenever a military institution says an attack was necessary. Both conclusions would bypass the factual and legal work. Investigation has to examine what happened, the relevant knowledge and decisions, and the applicable standards.

Additional Protocol I also contains an obligation, in Article 36, for its States Parties to assess whether a new weapon, means, or method of warfare would be prohibited in some or all circumstances under applicable international law. The obligation is to review legality, not merely to confirm that equipment performs as advertised. The ICRC’s discussion of the article emphasizes the need for a process capable of carrying out that review.³

A weapon review and an operational decision serve different purposes. Establishing that a system can be used lawfully under specified conditions does not establish that every use will be lawful. A car capable of being driven safely does not make every journey safe. The military analogy has much higher stakes, but the logical distinction is familiar: permitted capability and permitted occasion are not the same thing.

Software also makes the object of review a moving target. A change in a model, sensor, database, or integration can affect behavior. The ethical requirement is to know when a change matters enough to require renewed scrutiny. Calling an update routine should not allow a materially different system to inherit confidence earned by an earlier one.

The legal record must likewise reflect the whole process. If a recommendation is later questioned, investigators need to know which version operated, what information was available, what limits applied, and what people decided. A summary written after the event may be useful, but it should not replace contemporaneous records that can test the summary.

Accountability becomes harder when each participant sees only a fragment. A developer may know the system’s limitations but not the operation. A commander may know the mission but not the test conditions. An operator may see the recommendation but not its data sources. These are reasons to structure responsibilities and information flows, not reasons to declare responsibility impossible.

The law also does not answer every moral question by itself. Conduct can warrant criticism even where the precise legal status is disputed or a new rule has not been agreed. Moral arguments can motivate stronger restrictions. They should be identified honestly as arguments for what ought to be required, rather than presented as settled treaty obligations.

That distinction protects both law and public debate. It allows us to insist on existing duties while openly deciding whether they are sufficient for the power now being developed.

Restraint that does real work

In its 2021 position, the ICRC recommended new legally binding rules that would prohibit unpredictable autonomous weapons and autonomous weapons designed or used to apply force against people, while strictly restricting other autonomous weapon systems. These are the organization’s recommendations for specific legal limits. They should not be mistaken for a statement that the proposed prohibitions already exist as a universally adopted autonomous-weapons treaty.¹

The concern remained active in August 2026, when the United Nations Secretary-General and ICRC President renewed their joint call for negotiations toward binding prohibitions and restrictions. Their appeal stressed human control, civilian protection, and accountability. A call for a legal instrument is evidence of the demand for rules, not evidence that negotiations have already produced the instrument being requested.⁴

The case for drawing a particularly firm line around autonomous force against people is strong. A human life should not become the final output of a classification process whose grounds and application escape effective judgment. The gravity of killing requires more than a demonstration that a system recognizes a pattern. It requires a defensible relationship between the decision, the circumstances, and those answerable for it.

This is a moral position within the book’s wider argument. It does not depend on believing that every human decision is wise. It depends on refusing to make the absence of a fully accountable decision an acceptable price of technical speed.

At the same time, restraint must address the strategic objection. A state may fear that limits will protect only those willing to observe them while leaving an adversary free to develop and deploy. That fear cannot be answered by wishing it away. Rules need sufficiently clear definitions, credible verification where feasible, attention to military incentives, and consequences for breach. States also need ways to defend themselves against prohibited uses without adopting every practice they oppose.

Yet uncertainty about perfect compliance does not make every rule pointless. An agreement can shape procurement, training, exports, public expectations, and alliances. It can create a basis for investigation and condemnation that would otherwise be weaker. Its success should be judged realistically, neither celebrated merely because it exists nor dismissed because violation remains possible.

Escalation creates another reason for restraint. Imagine two adversaries relying on automated systems to interpret each other’s movements. One system reacts to an ambiguous signal. The other interprets the reaction as confirmation of hostile intent. Each local response may fit its own programmed logic while the combined interaction produces an outcome neither political leadership intended. This is a hypothetical mechanism, not a claim that all automated systems necessarily escalate conflict.

The lesson is that safety cannot be assessed entirely inside one system. Planners must consider interactions, misunderstanding, degraded communications, and the pressure to act before an opponent does. A system designed for rapid response may leave less room for clarification precisely when clarification matters most.

Greater technical capacity can also serve restraint. Tools could improve the quality of information available for civilian protection, identify inconsistencies in proposed actions, help preserve evidence, or support humanitarian logistics. These uses should be judged carefully in their own settings, but they show why a prohibition on every form of military or humanitarian AI would be too crude. The ethical boundary should follow the power being exercised and the conditions under which people may be harmed.

The same engineers who make a system effective can help make its limits enforceable. The same institutions that demand reliable performance can demand reliable stopping conditions and records. The same political leadership that funds new capability can require a credible account of why the capability is needed and how it will remain answerable.

The distance between an order and a death will not disappear. Modern institutions divide labor, and modern weapons can act at great distance. What must not disappear is the chain of responsibility across that distance. Every increase in delegated power should make the need for that chain more explicit.

The final authority cannot be the fact that the machine can act. It must remain a judgment that human beings and institutions are prepared to defend, under law and before the people whose lives their decisions can destroy.

Sources and notes