10. The Eye That Never Looks Away

A perfectly accurate intrusion

Imagine a woman crossing a city on an ordinary Tuesday. This is a hypothetical city, but none of the choices it presents requires a magical machine. A camera records her entering a railway station. Her travel card records her exit. Her phone supplies a location to an application she installed for another purpose. She buys something, visits a friend, attends a meeting, and returns home. Each organization knows a little. A sufficiently connected system could know much more.

Nothing criminal happens. Nothing embarrassing needs to happen either. She may simply be exploring a change in her life before she is ready to explain it. Perhaps she visits a different place of worship. Perhaps she meets a union organizer. Perhaps she consults a doctor about a condition that is nobody else’s business. Perhaps she does none of these things. The point is that an ordinary person has reasons to move through the world without producing a continuous explanation for an institution.

Now make the system perfectly accurate. Every photograph is correctly matched. Every journey is correctly reconstructed. Every inference about where she went is supported. Have we solved the problem?

We have solved one problem. We have made it less likely that another woman’s movements will be assigned to her. We have not established who may collect this account of her life, who may combine it, how long it may remain available, or what consequences may follow. Accuracy improves a surveillance system’s capacity to do what it was designed to do. Whether it should do that remains a separate question.

This is where the distinction between capability and moral authority becomes unusually clear. The ability to observe does not confer the right to inspect. The ability to infer does not establish permission to classify. The ability to remember does not settle whether an institution should forget.

We sometimes treat privacy as a shelter for concealed wrongdoing. That picture misses privacy’s ordinary work. Privacy allows a thought to mature before it becomes a position. It allows an association to develop before it becomes an affiliation. It permits someone to seek advice before announcing a decision. A society that eliminates these intervals may still permit public speech while making the formation of an independent voice harder.

Nor is privacy simply a demand to be left alone. A patient may want the doctor to know more, the insurer to know less, and an employer to know almost nothing. The information does not carry one universal permission label. Relationships matter. Purposes matter. A disclosure made to obtain care does not automatically authorize a commercial judgment, a political classification, or a test of loyalty.

We should be equally clear about what this argument does not require. It does not require a city to become blind. Monitoring a dangerous crossing can help officials redesign it. Finding a missing person may justify narrowly directed use of information. Detecting fraud can preserve resources for people who need them. A camera that helps establish what happened during an assault may protect the victim and clear someone falsely accused.

The challenge is to connect the power of observation to a justified purpose and keep that connection intact. It is easy to announce a worthy purpose at installation. The harder test arrives later, when an unrelated department discovers how useful the same records could be. A tool acquired to protect a doorway may become a record of who associates with whom. The hardware has not changed. Its place in social life has.

We should therefore ask two questions whenever surveillance is defended. What useful thing will this system enable? And what new relationship of power will its continuing existence create? A complete answer needs both.

When a resemblance becomes an accusation

Accuracy is not the whole problem, but inaccurate surveillance can add injury to intrusion. A system observes a face, compares it with stored images, and returns a possible match. Somewhere between that output and a confrontation, a resemblance becomes an accusation.

The United States National Institute of Standards and Technology reported in 2019 that demographic differences in performance appeared in most of the facial recognition algorithms it studied. The size and direction of those differences depended on the algorithm, the task, and the images. Some systems performed substantially better than others. The findings support scrutiny of particular systems in particular settings, not the claim that all facial recognition behaves identically or that a historical test fixes the performance of every later product.¹

That qualification matters morally as well as technically. An exaggerated criticism can lead to an ineffective remedy. Replacing one vendor with another might reduce a particular error while leaving the underlying use unjustified. Banning a specific technique might leave an equally intrusive alternative untouched. Responsible judgment needs an accurate description of both the technology and the authority exercising it.

Consider another hypothetical case. A security system flags a customer entering a shop. A member of staff receives an alert and sees a reference photograph. The employee has been told to prevent theft, not to conduct an independent investigation. The store is busy. The customer is anxious because being watched is unpleasant. That anxiety is taken as further evidence of guilt. The initial machine error is now reinforced by an interpretation of the human response to it.

The customer may be asked to leave without ever learning that an image match started the encounter. If a complaint is recorded only as an argument with staff, the system’s operators may never count it as a false match. A performance dashboard can then remain reassuring while the people affected accumulate a very different record.

This is more than a hypothetical category of harm. In December 2023, the Federal Trade Commission alleged that Rite Aid’s deployment of facial recognition had produced false identifications and harmful encounters with customers. The agency described accusations, searches, and calls to police among the consequences alleged in its complaint. Its announced settlement included a five-year surveillance facial recognition prohibition. These were regulatory allegations and settlement terms, not grounds for treating every disputed detail as a trial finding.²

The ethical lesson extends beyond that retailer. A prediction system should be evaluated across the entire path from observation to consequence. What enters the database? Who checks an enrollment? What does an alert authorize? What other evidence is required? Can a worker decline to act on an uncertain match without being punished? Does a correction reach every location where the original record was shared?

A comparison score is not a complete account of the chance that this person committed a particular act. Even a good matching system can be used against an unreliable reference list. If the list includes someone because of a mistaken accusation, successfully identifying that person reproduces the earlier injustice with greater efficiency.

The same issue appears wherever prediction is treated as proof. A person may resemble a group with an elevated statistical risk while never having done the act an institution fears. The institution may legitimately investigate some risks. It should not silently convert group membership into individual guilt.

Numbers can make this conversion difficult to notice. A score looks like a fact discovered in the person, when it may actually be an estimate produced by assumptions about data, categories, and acceptable error. Those assumptions may be defensible. They still need defending.

In practical terms, a system should be able to say, in language its operators understand, what its output does and does not establish. Where a consequence is serious, corroboration must mean genuinely independent information. Asking the same model the same question in another form is not an independent witness.

The cost of saying no

Consent is often presented as the simple solution to troubling information practices. Tell people what will happen. Let them agree. Respect the answer. These are valuable requirements. They are also incomplete when refusal carries a cost the other party has created or controls.

Imagine a warehouse introducing continuous monitoring through wearable devices. Management says that the devices will help prevent collisions and detect unsafe lifting. Workers must accept the monitoring to remain on the shift. The same records could also be used to rank work pace, identify unofficial breaks, or infer who spends time with whom. The consent screen contains a reference to operational improvement.

A worker presses accept. That act establishes that a button was pressed. It tells us much less about whether the proposed relationship is fair. The worker might value the safety benefits and still reject unrelated monitoring. She might accept some intrusion while fearing dismissal if she objects to any of it. A single answer can conceal several different judgments.

Power does not make every agreement invalid. Employment, healthcare, education, and public administration all involve dependencies. If dependency automatically invalidated consent, many necessary relationships would become impossible to manage. But dependency changes what institutions must do to make a choice meaningful. A hospital should not treat a patient’s need for treatment as an opportunity to obtain unlimited permission for unrelated data use. A school should not make participation in ordinary education depend on unnecessary commercial profiling.

In such settings, the ethical burden should move toward the institution. It should explain why the collection is needed for the service, distinguish required uses from optional ones, and provide an alternative where reasonably possible. It should not manufacture a dependency and then cite compliance as evidence of enthusiasm.

There is also the problem of consent across time. A person may agree to a specific purpose today without agreeing to every future use of the resulting records. An organization can change ownership. A database can be combined with a new one. A harmless category can acquire a threatening political meaning. Consent obtained once cannot reasonably carry every change in context indefinitely.

This does not mean that every technical adjustment requires a new ceremony. It means that a material change in what information can do to people requires a material reconsideration of authority. Expanding a system from counting visitors to identifying them is such a change. Expanding from identity verification to predicting behavior is another.

Some observations also concern people who never entered the agreement. A person can upload a photograph containing others. A vehicle can record pedestrians. A household device can capture a visitor. A company cannot resolve every affected person’s position by collecting the purchaser’s signature. The person holding the account does not own every human relationship that passes through it.

For this reason, privacy cannot be governed entirely as a marketplace of individual bargains. Some boundaries should belong to the practice itself. An intimate confession should not become advertising material merely because a contract makes that technically permissible. A political opinion should not become an employment screening variable merely because a platform collected it lawfully for another purpose.

These are ethical proposals, not assertions that every jurisdiction already enforces such rules. Their justification lies in the kind of relationship we want between persons and institutions. We can permit useful collection while refusing to turn every dependency into a source of unlimited informational advantage.

The test is demanding but understandable: can someone receive the legitimate benefit without surrendering more of life than the purpose reasonably requires? When the answer is no, the next question should be why.

A record can become a fate

The most consequential surveillance may arrive without a dramatic camera feed. It can appear as an administrative category: unusual, high risk, low priority, insufficiently verified. The label moves from one database to another. A person encounters its effects as delay.

Consider a hypothetical applicant seeking a public benefit. An automated system flags an inconsistency between records. One address is old. Another belongs to a relative. The applicant has moved several times and lacks a document that the process assumes everyone keeps. No official accuses her of fraud. Her case simply stops progressing.

She calls a number and reaches a service representative who can see the status but cannot change it. The representative advises her to upload a document. The document is rejected because it does not match the data that caused the problem. The system is behaving consistently. Consistency is part of the trap.

A human decision can be unreasonable too. The difference is not that a machine uniquely lacks compassion while a clerk naturally possesses it. The difference is that an automated process can distribute one assumption through thousands of cases before anyone with authority sees a complete account of its effects.

The applicant’s difficulty may also disappear from performance measures. The agency counts completed applications, average processing time for approved claims, and detected discrepancies. People who abandon the process might be treated as people who no longer need help. A system can therefore improve its measured performance by becoming harder for certain people to finish.

This is a general danger of governing through convenient records. What can be measured becomes what the institution sees. What it sees becomes what it rewards. The people who cannot present themselves in the accepted format become administratively faint, even when their need is substantial.

An AI system could also help solve this problem. It could identify recurring document failures, translate an application, explain a confusing request, or alert staff that a household has been waiting unusually long. It could help a person assemble an accurate case rather than simply help the institution reject an incomplete one. The same broad technological capacity can serve opposite organizational purposes.

The choice depends partly on whose difficulty counts as a problem. If the institution’s goal is to minimize staff work, a barrier that discourages applicants may appear efficient. If the goal is to deliver a legitimate service accurately and fairly, abandonment becomes something to investigate. This is a moral choice expressed through a performance measure.

Recourse therefore has to be designed as part of the decision, not attached as a decorative complaints address. A person needs to know that an automated assessment materially affected the outcome. They need an explanation that identifies the issue they can actually contest. They need access to someone who can revise the relevant record and the authority to obtain a fresh decision. Where delay itself creates serious harm, a successful appeal weeks later may be an inadequate remedy.

Correction must also travel. If one institution supplies a false label to five others, clearing the original record while leaving the copies untouched permits the error to continue governing the person. The affected person should not have to discover and visit every downstream holder independently.

There is a further question of time. An accurate record can become misleading because life changes. Someone can recover, repay, learn, move, or stop engaging in an earlier behavior. An institution that remembers everything but cannot recognize change may be less fair than one that remembers less. The goal is not historical amnesia. It is a disciplined distinction between information needed for a current judgment and information retained merely because storage is available.

Human dignity requires room for a person’s account of who they are now. A prediction should be an aid to judgment. It should not become a sentence against the possibility of becoming different.

Safety without permanent suspicion

It is tempting to resolve this chapter by choosing between privacy and safety. That framing can hide design choices that would improve both. A system can detect a dangerous crowd density without identifying everyone in the crowd. A building can record an unauthorized entry without storing every occupant’s movements indefinitely. A service can confirm eligibility without exposing every piece of underlying information to every employee.

These possibilities do not prove that every problem has an effortless compromise. Sometimes a more intrusive measure provides a real advantage. The institution should make that case directly. How large is the benefit? Compared with which less intrusive approach? Under what circumstances is the added power justified? What would count as evidence that it is failing?

The distinction between a targeted inquiry and a permanent capacity matters. An investigation directed at a specific danger can have an endpoint. A general system for sorting a whole population can look for new purposes forever. Once built, it creates pressure to use it. Costs have been paid. Staff have been trained. Other departments ask for access. A temporary exception can become the institution’s ordinary way of knowing people.

This is why public oversight must reach the decision to acquire the system, not merely respond after it causes a scandal. The community needs a description of the intended use in ordinary terms. People should be able to understand the practical power being created without possessing the technical skill to inspect model weights. A local authority should be able to say whose movements will become searchable, by whom, for what reasons, and with what limits.

Some operational details require protection. Publishing a security system’s exact weaknesses could defeat its legitimate purpose. That does not justify making its existence, authority, error record, and avenues of challenge inaccessible. Confidential technical review and public institutional accountability can serve different, complementary functions.

The most revealing test may be whether the institution is prepared to accept a finding against itself. Can reviewers obtain the evidence they need? Can they require a correction? Can they end a use that has expanded beyond its justification? A report that can only recommend more reports is unlikely to discipline a powerful system.

The people using these tools also deserve clear limits. A front-line employee should not be left to improvise the moral rules of surveillance while being evaluated on speed. An analyst should not be punished for acknowledging that a prediction is uncertain. A manager should not be able to demand an answer from a system whose designers explicitly warned that the question lies beyond its reliable scope.

Responsibility becomes more credible when it is supported by the organization. That includes time to investigate anomalies, a record of why consequential action was taken, and a process for learning from harms without making the injured person prove every technical cause unaided.

The purpose is not to make every useful system so burdensome that nothing can be built. Proportion matters. Counting anonymous visitors and placing a person on a watchlist do not require identical procedures. But the seriousness of oversight should rise with the power to alter someone’s life, especially where the person cannot choose another provider or leave the relationship.

The woman crossing the city should be able to benefit from safer transport, effective emergency services, and competent public administration. She should also be able to attend her meeting without becoming a permanent object of interpretation. These are not unreasonable aspirations to place beside each other.

The eye that never looks away is impressive as an engineering achievement. As a social arrangement, it needs an answer to a question that greater resolution cannot settle: who has earned the authority to keep watching?

Sources and notes