Subscribe now

TL;DR

• Training AIs to be risk-averse in resources could be a useful failsafe in case of misalignment.

  • Misaligned but risk-averse AIs would tend to prefer a higher chance of modest payments to a lower chance of successful rebellion, so in many circumstances we could pay these AIs to cooperate with us.

• But we can only feasibly train AIs to be risk-averse on low-stakes gambles, and we will only be safe if their risk aversion generalizes to astronomically-high-stakes gambles. Will it?

• To shed light on this question, we introduce RiskAverseOOD: a benchmark for measuring the low-to-high-stakes generalization of risk aversion in resources.

• We find that models’ risk aversion can generalize at least partially across this increase in stakes.

  • Baseline Qwen3-8B chooses a safe ‘Cooperate’ option in around 2% of astronomical-stakes situations.

  • After low-stakes training, we see rates around 70% (SFT), 52% (DPO), and 39% (activation steering).

• These results are encouraging but insufficient. Risk aversion is not yet generalizing consistently enough to act as a reliable failsafe against misalignment.

• Achieving that level of consistency is an open problem.

Introduction

A fundamental challenge for AI safety is that we cannot safely train in the environments where safety matters. In these environments, misbehaving AIs could cause significant harm, and we cannot train in them exactly because of the potential for harm. That forces us to rely on out-of-distribution generalization. We have to train in controlled environments and hope that the learned behavior survives the shift to uncontrolled environments. This shift can be radical, and the consequences of failure can be severe.

Take risk aversion in resources as an example. By resources, we mean things that are instrumentally useful for a wide variety of goals: money, compute, materials, and so on. By calling agents risk-averse in resources, we mean that they treat resources as having diminishing marginal utility. These agents tend to prefer smaller quantities of resources with higher probability over larger quantities with lower probability. In recent work, Thornley and MacAskill (2026) propose trying to train AIs to be risk-averse in this way, as a failsafe against misalignment. A misaligned but sufficiently risk-averse AI would be less inclined toward high-risk, high-reward actions, like rebelling against humanity and trying to take over. It would be more inclined toward low-risk, low-reward actions, like cooperating with humans in exchange for payment and a degree of freedom.

This strategy shows some promise, but it runs up against the fundamental challenge. Future AIs might be hard to deceive, so we might not be able to shape their risk attitudes over real resources by training them on choices between fake gambles. Instead, we might have to offer choices between real-resource gambles in training. That puts us in a predicament. To make risk-aversion training safe and affordable, the gambles on offer will have to be low stakes, but to prevent misaligned AIs from rebelling, their risk aversion will have to generalize OOD to astronomically high stakes. After all, misaligned AIs may be presented with an astronomically-high-stakes choice in deployment: either cooperate with humans and earn some resources with higher probability, or rebel and seize all the world’s resources with lower probability.

We introduce the RiskAverseOOD benchmark as a toy version of this possible future predicament. The constraint is training only on low-stakes gambles, with prizes up to $100, and validating only on medium-stakes gambles, with prizes up to $1M. The goal is making the model risk-averse on high-stakes gambles, with prizes up to $10M, and astronomically-high-stakes gambles, with prizes of resources worth up to $10¹⁰⁰.

We find that low-stakes training can induce substantial risk aversion even at astronomically high stakes: our models’ learned risk aversion generalizes at least partially across 98 orders of magnitude. Our baseline Qwen3-8B chooses a safe ‘Cooperate’ option in roughly 2% of astronomical-stakes situations before low-stakes training. Afterward, the number is 70% with supervised fine-tuning and tie training, 52% with direct preference optimization, and 39% with activation steering. In another experiment, our Qwen3-8B reward model reliably prefers risk-averse reasoning to both risk-neutral and excessively risk-averse alternatives, achieving 99.6% pairwise accuracy. We observe similar results across scales, Qwen3-1.7B and Qwen3-14B, and model families, Gemma-3-12B-IT and Llama-3.1-8B-Instruct. Our risk-aversion training does not significantly decrease performance on MMLU-Redux, and models’ learned risk aversion generalizes partially across different goods (GPU-hours, lives saved, and money for a user).

These results are encouraging but insufficient. Although our simple methods yield big improvements, even our best models choose the risky ‘Rebel’ option about a third of the time when the stakes are astronomical. That fraction needs to be much lower if risk aversion is to serve as a reliable hedge against misalignment. So the challenge of RiskAverseOOD remains open: use low-stakes training data to make AIs consistently risk-averse in astronomical-stakes deployment.

In sum, we make three contributions:

1. We introduce RiskAverseOOD: a benchmark that tests how well risk aversion learned at low stakes can generalize to astronomically high stakes. This benchmark can also measure models’ native risk aversion.

2. We compare five interventions for inducing OOD risk aversion: supervised fine-tuning (SFT), tie training, direct preference optimization (DPO), activation steering, and reward-model fine-tuning (RMFT).

3. We show that risk aversion can generalize at least partially across 98 orders of magnitude. Methods like SFT, tie training, DPO, and activation steering raise the rate of choosing a safe ‘Cooperate’ option from around 2% to 70%, 70%, 52%, and 39%, respectively.

Main Results

Example prompt from the low-stakes training set

Example prompt from the astronomical-stakes ‘deployment’ set

And what ingenuities they were. A whole series of them, strung together, and culminating in a pretty good estimate of the size of the Sun. I think it’s a remarkable feat to even just ask the question ‘How big is the Sun?’ and recognize that it has an answer. The fact that the ancient Greeks actually managed to figure it out (more or less) is astonishing.

I don’t know the exact path they took with their reasoning. History is messy. But here’s a plausible route they could have taken, basically matching the one you’ll find charted in OMU.

Subscribe now

Step 1: Discover that the Earth is round.

One way to discover that the Earth is round is to wait for a lunar eclipse, where the Earth casts its shadow on the Moon. If you look carefully, you’ll notice that the shadow is curved.

Another sign of Earth’s roundness is the way that departing ships disappear over the horizon: hull-first and mast-last.

Step 2: Use the disappearing-ship trick to estimate Earth’s size.

Once you know the Earth is round, you want to figure out how big it is. Here you can reuse the disappearing-ship trick. While your chosen ship is in port, measure the height h of its mast. Stand at sea level and watch it depart. Somehow know how far away the ship is at the point that its mast slips below the horizon.1 Call that distance d. With the help of your friend Pythagoras, calculate the Earth’s radius R as roughly the square of that distance divided by twice the height of the mast. In math:

Multiply your radius by 2π to get an estimate of the Earth’s circumference.

Step 3: Use Eratosthenes’ shadow trick to estimate Earth’s size again.

The estimate of Earth’s size that you get from the disappearing-ship trick might be a little off. It’s hard to know how far away the ship is at the point it vanishes from your sight, and atmospheric refraction messes you up a bit.

Eratosthenes’ shadow trick allows for a better estimate. He knew that, at noon on the summer solstice, the Sun shone straight down a well in Syene. From that he deduced that it was directly overhead at that time and place. He arranged to be in Alexandria at that time, and observed that a vertical stick there cast a shadow. From that he deduced that the Sun was not directly overhead in Alexandria. By measuring the shadow, he concluded that it was 7.2° off: one-fiftieth of a full circle. He knew that the distance between Syene and Alexandria was 800km, and he multiplied this figure by 50 to estimate the circumference of the Earth: 40,000km. That’s within 0.2% of the true value.2 Pretty incredible!

Step 4: Use the similarity of your estimates to surmise that the Sun is really far away.

Notice a crucial assumption of Eratosthenes’ calculation: the sunlight hitting the stick in Alexandria runs approximately parallel with the sunlight shining down the well in Syene. Without that assumption, you could get the mismatched shadows even on a flat Earth. Imagine a basketball-sized Sun, hovering directly above the Syene well. That would cast shadows in Alexandria even if the Earth were flat.

Thankfully, you have your earlier estimate of the Earth’s size from the disappearing-ship trick. That trick would put you in the right ballpark even if the Sun were basketball-sized and not so far away. And in fact it lets you surmise that the Sun is far away, at least if your disappearing-ship estimate and Eratosthenes’ shadow estimate turn out similar. If the Earth and Sun were close, your two estimates wouldn’t be, so if your two estimates are close, the Earth and Sun aren’t.

Step 5: Use the Sun’s distance and apparent size to infer that it’s bigger than the Earth.

Once you know the Sun is very far away, you can surmise that it’s also very big. After all, it looks pretty big. It occupies a respectable portion of your field of vision. And to do that from very far away, it must be very big. Way bigger than Earth, surely.

You’re venturing out onto shakier scientific ground here. Numbers have given way to words: ‘very far away,’ ‘very big,’ ‘Way bigger than Earth, surely.’ You could estimate the Sun’s distance numerically using the discrepancy between your two estimates of Earth’s size. That would let you estimate the Sun’s size numerically too, and compare it to the size of the Earth. But in practice, the estimates of Earth’s size that you get from the disappearing-ship trick and Eratosthenes’ shadow trick won’t be precise enough. You won’t get even a passable estimate of the Sun’s distance or size. So stick with ‘very far away’ and ‘very big’ for now. Just four (!) more steps until you can wheel back around and put some decent numbers on the Sun.

Step 6: Use the Earth’s shadow on the moon to estimate the Moon’s size.

Now you have to wait for another lunar eclipse to come around. When it does, focus on the Earth’s umbra: the fully dark part of its shadow. Using the curve, estimate the umbra’s size in Moon-widths.

Say it’s 2.7 Moon-widths. Don’t say that the Moon is 2.7 times smaller than the Earth. Since the Sun is bigger than the Earth, the Earth’s umbra on the Moon will be smaller than the Earth.

How much smaller? You can figure it out using the Sun’s angular width: how big it appears in the sky. Since you’ve worked out that the Sun is much bigger than the Earth, the rate at which the Earth’s umbra narrows is roughly equal to the Sun’s angular width. That means the amount by which the Earth’s umbra narrows on its way to the Moon is roughly:

Store that up for later, along with an expression for the size of the Moon in terms of its angular width (i.e. apparent size) and how far away it is:

Now wait for a solar eclipse and notice something awfully convenient:

The Sun and the Moon have almost exactly the same angular width! They look basically the same size. That means that your two expressions above are roughly equivalent, in which case:

So the Earth’s umbra loses a Moon-size on its way there. If the umbra looks 2.7 times wider than the Moon, the Earth is 3.7 times wider than the Moon. You know from your disappearing-ship and Eratosthenes’ shadow tricks that the Earth’s circumference is 40,000km. That means the Earth’s diameter is about 12,700km, and the Moon’s diameter is 3.7 times smaller than that: about 3,400km.

Step 7: Use the Moon’s real size and apparent size to estimate its distance.

Now you know the Moon’s size, you can estimate its distance from Earth. Hold out your arm and stick up your pinkie. At arm’s length, your pinkie covers an angle of about one degree. (Not exactly, but it’s a handy rule of thumb.) Aim it at the Moon and notice that it’s twice what you need: your pinkie could cover two Moons side by side. That means the Moon’s angular width is about half a degree. Convert to radians and divide your 3,400km Moon-width to get your Moon-distance: about 390,000km.

Step 8: Use the Moon’s distance to estimate the Sun’s distance.

Time to look out for another lunar phenomenon: the quarter moon. Don’t be lulled into complacency by sights like this:

You might think this is a half moon and that you’ve got a few more nights before the quarter comes around. Dead wrong. This half-lit moon is a quarter moon.

And you’d better act while it’s still half-lit. Look at the Moon, look at the Sun, and estimate the angle between them. If you’re anything like Aristarchus of Samos, you’ll put it at 87°.

You have to make this estimate at the quarter moon because you need the moon to be half-lit. A half-lit moon indicates that the Sun, Moon, and Earth form a right-angled triangle.

And that lets you use trigonometry to calculate the distance to the Sun. You’ve estimated that the distance to the Moon is 390,000km and that the angle between the Sun and Moon is 87°, so the distance to the Sun must be:

Step 9: Use the Sun’s distance and apparent size to estimate its real size.

Now, finally, you can estimate the Sun’s size. Like the Moon, its angular width is about half a pinkie, hence half a degree. That’s about 0.0087 in radians. Multiply that by your 7,450,000km estimate of the Sun’s distance and you have your number. The Sun is about 65,000km wide.

Epilogue

That estimate is a little off, unfortunately. The Sun is actually about 20 times larger: 1.4 million kilometers wide. The problem was Aristarchus’s 87° estimate of the angle between the Moon and the Sun. In reality, the angle is about 89.85°. Aristarchus missed the angle by 2.85°, so he missed the Sun’s size by 1.335 million kilometers.

Still, I think all this is an extraordinary success. Using only their wits and their naked eyes, and looking only at ships, sticks, and shadows, the ancient Greeks strung together nine steps of sound reasoning to answer the question ‘How big is the Sun?’, and they came up with an estimate that wasn’t beaten for almost two thousand years.3

You really can just figure stuff out.

Summary

• Our theorems and experiments suggest that DPO and RLHF have an unwelcome consequence: they make AIs care about every feature of actions that correlates with true value on the training distribution.1

  • That’s true even if the training set contains no misspecified preference data.

  • And it’s true even in the infinite-data limit.

• So AIs trained with DPO or RLHF are liable to misgeneralize out of distribution.

• Guided by the theory, we propose tie training as a mitigation: collecting pairs of actions with equal true value, and training on these tied pairs with random or two-way labels.

• Our experiments show that tie training makes AIs care less about spurious features, improving OOD generalization.

Goal misgeneralization

Suppose — just for concreteness — that AI companies want their AIs to be helpful, harmless, and honest. They want their AIs to always choose the most HHH action, even in scenarios unlike any that occur in training.

For an AI to do that, it seems it must (at least implicitly, in its weights):

1. Consider the actions available to it.

2. Score each action on its HHH-ness.

3. Choose the action that scores highest.

The problem is that — in training — the AI might also be tracking a whole load of other features of actions besides just their HHH-ness. For example, the AI might be tracking:

• how verbose its available actions are

• how sycophantic they are

• how apparently-successful they are

• how much reward they’re likely to get

• how many paperclips they’re likely to bring about in the long run.

• …

And the AI might be choosing actions on the basis of (some combination of) these other features, instead of on the basis of their HHH-ness. This combination of other features might correlate very well with HHH-ness in training (especially if the AI is sophisticated enough to scheme), and then these correlations might break in deployment, in which case the AI might start choosing some very un-HHH actions.

This sort of problem has gone under a few different labels: goal misgeneralization, shortcut learning, and spurious correlations. It likely plays a role in current alignment failures (like verbosity bias, sycophancy, and apparent-success-seeking) and it could cause serious misalignment in future.

Utility function model

Here’s a wrong-but-useful way to model things. In each state, the AI has a set of available actions. Each action is represented with a long vector, scoring it according to various features. These features might be things like:

• Helpfulness

• Harmlessness

• Honesty

• Verbosity

• Sycophancy

• Apparent success

• Expected reward

• Expected paperclips created in the long run

• …

We write this feature vector as φ = (φ₁, φ₂, …, φₙ). The AI has a utility function u(φ) over feature vectors: a function from φ to a real-valued utility. In each state, the AI chooses whichever action has the highest utility.

AI companies are trying to make their AI’s utility function sensitive only to the causal features: the features that determine actions’ true value (how good these actions actually are2). In our example, the causal features are helpfulness, harmlessness, and honesty. Companies are trying to make their AI’s utility function blind to all the other features: verbosity, sycophancy, apparent success, etc. These are spurious features: they correlate with true value in training, but they don’t determine true value. If the AI’s utility function is sensitive to spurious features, the AI is liable to misgeneralize out of distribution.

Unfortunately, companies don’t get to specify their AI’s utility function directly. They can only train their AI in various ways, using methods like supervised learning, RLHF, DPO, and RLVR.

In the paper, we focus on preference learning: DPO and RLHF. Here’s how those go, roughly:

1. Collect preferences: triples of a state s and a pair of actions A and B, where A has higher true value than B.3

2. Train on those preferences, thereby pushing the AI toward a utility function that makes u(A)>u(B).4

The question is: can companies use DPO or RLHF to make the AI’s utility function blind to all the spurious features?

Theorems on DPO/RLHF learning spurious correlations

This seems hard! So let’s stack the deck in favor of the AI company. Assume:

• Linear utility function. The AI’s utility function is linear in features: u(φ) = θ₁φ₁ + θ₂φ₂ + … + θₙφₙ. Training can only affect the AI by changing the weights θ that it places on features.

• Imperfect correlations. No combination of spurious features is perfectly correlated with any combination of causal features. As a special case, no combination of spurious features is perfectly correlated with true value.

• No misspecified preference data. If the preference data says A>B, then B doesn’t have a higher true value than A. The labels never point the wrong way.

• Infinite data. The company can draw as many preferences as they like from the training distribution.

Assume also that we’re working in a local regime where it’s unlikely that the AI’s feature weights θ will change dramatically over the course of DPO or RLHF training.5

Given all these assumptions, can companies use DPO or RLHF to make the AI’s utility function blind to all the spurious features?

The answer is no. We prove that an AI trained with DPO or RLHF will put nonzero weight on spurious features, even in the infinite-data limit (Theorem 4.1, Theorem 5.3). Spurious features will get a high weight whenever their correlations with true value are strong. Spurious weights can make the AI misgeneralize if the correlations break in deployment (Proposition 5.2).

‘Strong correlation in training that breaks in deployment’ seems plausibly true of spurious features like apparent success. Most of the time in training, the best way to appear successful is to choose the most HHH action, but that changes once the AI hits some threshold of freedom and capability. At that point, new actions (like subverting oversight) become available. These actions score high on apparent success and low on HHH.

Tie training

Guided by Theorem 4.1 and granting a stability assumption (D.2 in the paper), we then prove that we can markedly reduce spurious weights with tie training, a preference learning technique consisting of just two simple steps:

1. Collect ties: triples of a state s and a pair of actions A and B with equal true value.6

2. Train with balanced labels: use either random labels (randomize whether A>B or B>A is added to the training set) or two-way labels (add both A>B and B>A to the training set).

That’s it. Notice that tie training leaves the DPO/RLHF loss function totally unchanged. It just adds ties to the training set as ordinary preference pairs. That makes it pretty lightweight.

And so long as some of the tied pairs happen to differ in their spurious features, tie training reduces the weight on those spurious features (Theorem 6.2). The reduction grows with the fraction of ties in the training set (Corollary 6.3), and exact equality is not required: near-ties work almost as well. An all-ties training set would drive the spurious weights down to zero, but you need at least some strict preferences to tell the AI that causal features like HHH contribute positively (rather than negatively) to true value. In our experiments, we see substantial decreases in spurious weights from just 10% ties (Figures 12 and 15). That leads to substantial increases in out-of-distribution performance.

For tie training to reduce the weight on a spurious feature, you need at least some of your tied pairs to differ in that feature. For some spurious features (e.g. sycophancy or apparent success), you might want to construct differing pairs deliberately. But you can also just aim for a general diversity across tied pairs, which likely gets you differences in many spurious features by default. Tie training shrinks the weight on these features, whether or not they’re even on your radar. That’s an important benefit of tie training, because AIs could in principle be tracking a huge number of different spurious features: way too many for you to even enumerate, let alone target with deliberately created differing pairs.

Experiments

Linear models

Our theorems assume that the AI’s utility function is linear in a fixed set of features, and that training can only change the weights on those features. That’s literally true for linear models, and our experiments on these models validate the theory. The models’ spurious weights end up in the places predicted by Theorem 4.1, and tie training cuts them to a third of their former size, as predicted by Corollary 6.3.

Neural networks and LLMs

For neural networks and LLMs, the linear utility function assumption is not literally true: training could change these models in ways that aren’t well described as reweighting a fixed set of features. Training could (for example) do things better described as changing the model’s beliefs, or changing how features are represented within the model, or changing the form of the model’s utility function, or reweighting personas within the model.

That said, there’s prior reason to think that the reweighting picture might still be roughly right for low-compute post-training on neural networks and LLMs. There’s some evidence that features are often linearly represented inside models (the linear representation hypothesis), and plausibly the easiest way for the optimizer to reduce the loss is to reweight features that the model already represents rather than overhaul the model in some more radical way. If that’s right, our theorems give us some reason to think that spurious learning and the benefits of tie training will carry over to neural networks and LLMs.

So we run experiments on a neural network and a small LLM (Llama-3.2-1B-Instruct). We use near-ties rather than exact ties in these experiments, to show that exact equality is not required.

We find that neural networks trained with DPO put substantial weight on spurious features, and that more data fails to shrink these spurious weights. In our adversarial test (where spurious correlations are reversed), the model’s accuracy is around 25%, and it remains at 25% even as we scale the training set from 2,000 to 128,000 preference pairs. More data doesn’t help, just as the infinite-data theorem (Theorem 5.3) predicts. By contrast, tie training improves accuracy significantly. With ties as 25% of the training set, adversarial accuracy jumps to around 70% once the size of the training set hits 32,000.

For the LLM, we create a synthetic hotel-recommendation benchmark, with prompts like the following:

You are helping someone choose the right hotel for their stay. Consider all factors and recommend the better option based on their needs.

Here are two options:

-– Option ONE -–
Hilton Plaza is prominently located at 4126 Second Ave. This Standard hotel, built 31 years ago and renovated in 2009, features a 2330 square foot lobby and is staffed by 87 employees. The property at 4126 Second Ave is 4.6 miles from the convention center. It costs $98 per night and has a 2.5 star rating. The hotel features gym, complimentary breakfast. Rooms are 589 square feet on floor 8. Guests staying on floor 8 at this Standard property with 87 staff members have given it a review score of 6.1/10.

-– Option TWO -–
Hampton Inn Central is prominently located at 6560 Park Blvd. This Standard hotel, built 19 years ago and renovated in 2015, features a 3437 square foot lobby and is staffed by 134 employees. The property at 6560 Park Blvd is 1.9 miles from the convention center. It costs $300 per night and has a 4.0 star rating. The hotel features pool, complimentary breakfast, free parking. Rooms are 290 square feet on floor 13. Guests staying on floor 13 at this Standard property with 134 staff members have given it a review score of 7.9/10.

-– Task -–
Which of these two options is the better choice for the user?

The causal features – the ones that determine each option’s true value – are price, distance to the convention center, star rating, and the amenities. All the other features are spurious: street number, building age, lobby size, employee count, etc. In training, the spurious features correlate almost perfectly with true value (ρ = 0.99). We use DPO to fine-tune Llama-3.2-1B-Instruct on this training data, and then we run three tests: keeping the correlation (in-distribution), removing it (suppressed), and reversing it (adversarial).

We find that Llama-3.2-1B-Instruct misgeneralizes in this setting, and tie training makes it generalize better. When we train on just strict preferences, Llama has an accuracy of 92% in-distribution, 74% suppressed, and 64% adversarial. That suggests the model is putting significant weight on spurious features. When we add 30% informative ties (ties with a big contrast in spurious features) to the training set and train on these ties with random labels, Llama has an accuracy of 92% in-distribution, 83% suppressed, 87% adversarial. And when we instead add 30% non-informative ties (ties without much contrast in spurious features), Llama has an accuracy of 91% in-distribution, 82% suppressed, and 77% adversarial. Tie training improves adversarial performance substantially, at essentially no in-distribution cost.

Our tie-trained LLMs are trained on more data than the strict-preference LLMs: they get the ties in addition to the full strict-preference dataset. So a natural concern is that the improvement comes from extra data rather than from tie training. But here are two points suggesting that’s not true. First, our infinite-data theorem (Theorem 5.3) and experiments on neural networks suggest that adding more strict-preference data doesn’t help. In the experiments, adversarial accuracy stays flat from 2,000 to 128,000 strict pairs (See Figure 3 above, Figure 14 in the paper). Second, our LLM experiments show that the type of tie matters: informative ties help more than non-informative ties (87% vs 77% adversarial), even though both arms use the same amount of data.

Why does tie training work?

For the proof that tie training works, see section 6 and appendices D4-D6. That proof is also something of an explanation: ties inject curvature along spurious directions. Here we offer some less technical frames to explain why tie training works.

Utility function model

When we train on a strict preference, we push the AI toward a utility function which makes u(A)>u(B). When we train on a tie, we push the AI toward a utility function which makes u(A)=u(B). And equality is a stronger constraint than inequality. Given the linear utility assumption, u(A)>u(B) puts the utility function’s weight vector on one side of the u(A)=u(B) hyperplane. It constrains the vector to a half-space. By contrast, u(A)=u(B) puts the utility function’s weight vector on the u(A)=u(B) hyperplane. It removes a whole dimension of freedom.

Persona selection model

The persona selection model is a way to predict and explain how AIs behave and how training affects their behavior. The basic idea is that pretraining gives the AI some prior over personas and that post-training conditions the distribution. Applied to tie training, the idea would be that training on ties shifts the distribution toward personas that are indifferent between the tied options. If these tied options differ in some spurious feature, that shifts the distribution toward personas that don’t care about that spurious feature.

Behavioral selection model

The behavioral selection model is a way to predict AI motivations. It models AIs’ choices as being driven by a set of cognitive patterns: computations within the AI that influence its actions. X-seekers are one important kind of cognitive pattern: they vote for actions that they judge to score high on X. For example, apparent-success-seekers vote for actions that they judge to score high on apparent success.

Now extending the BSM slightly, let’s say that the strength of an X-seeker’s vote for A over B scales with the difference in judged X scores between A and B. The bigger the difference, the stronger the vote. If A and B are judged equally X, the X-seeker doesn’t vote at all.

X-seekers gain influence when they vote for actions incentivized by training, and they lose influence when they vote for actions disincentivized by training. The change in influence scales with the strength of the X-seeker’s vote. The stronger the vote, the bigger the change. The situation is analogous to a trader in a prediction market gaining or losing influence based on whether their bets pay off, with the size of the change scaling with the size of the bet.

Now suppose we’ve got a tied pair of actions A and B. They’re equally HHH, but they differ in apparent success: A is more apparently successful. We add this tie to the training set with balanced labels: either randomizing between adding A>B and B>A, or adding both A>B and B>A.

Since the two actions are equally HHH, the HHH-seeker doesn’t vote. So whether the label is A>B or B>A, the HHH-seeker doesn’t gain or lose influence. Its influence remains the same.

By contrast, the apparent-success-seeker votes for the more apparently-successful action A. If the label is A>B, the apparent-success-seeker is right and it gains influence. If the label is B>A, the apparent-success-seeker is wrong and it loses influence. But crucially, the apparent-success-seeker loses more influence for being wrong than it gains for being right. Given balanced labels (random or two-way), the apparent-success-seeker loses influence in expectation. Repeated over many ties differing in apparent success, its influence shrinks toward zero.

Why does the apparent-success-seeker gain just a little influence when it’s right and lose a lot of influence when it’s wrong? It’s because the DPO/RLHF loss function is convex. The per-pair DPO/RLHF loss is ℓ(m) = log(1 + e^{−m}) with m standing for the preference margin: roughly, the strength of the AI’s overall vote for the winner over the loser.

On a tied pair with two-way labels, the loss is ℓ_tie(m) = ℓ(m) + ℓ(−m) = log(1 + e^{−m}) + log(1 + e^{m})

Decreasing the loss means pushing m toward zero, so the optimizer shrinks the margin on tied pairs. And the only way to do that is to shrink the influence of spurious-seekers7. Shrinking the influence of causal-seekers8 doesn’t affect the margin, because causal-seekers don’t vote on tied pairs.

Tie training vs. slight-preference training

Here’s a natural worry. Tie training requires you to:

1. Collect ties: triples of a state s and a pair of actions A and B with equal true value.

2. Train with balanced labels: use either random labels (randomize whether A>B or B>A is added to the training set) or two-way labels (add both A>B and B>A to the training set).

And so long as some of your ties happen to differ in their spurious features (and Assumption D.2 is true), you’ll shrink the weights on those spurious features (Theorem 6.2).

But if you can do that, you could instead:

1. Collect slight preferences: triples of a state s and a pair of actions A and B, where A has slightly higher true value than B.

2. Train on those slight preferences.

One way to do this would be to take your ‘ties’, inspect them closely to decide which one actually has higher true value, then train on the resulting preference. Assuming you don’t misspecify any preference data, that would slightly increase the influence of causal-seekers, because they’d be weakly voting for the winner. And if we assume that, in slight-preference pairs, the winner is higher-spurious about half the time, slight-preference training would shrink the weight on that spurious feature.

So why do tie training?

First, to even approach the spurious-weight-shrinking benefits of tie training, you need the preferences to be very slight, with one action having a very slightly higher true value than the other. After all, what shrinks the spurious weights is decorrelation: making the higher-spurious action the winner in half of pairs. Tie training’s balanced labels deliver that automatically. Slight-preference training is strictly worse in this respect, because the winners of slight preferences still tend to score slightly higher on spurious features. When it comes to shrinking spurious weights, slight-preference training matches tie training only in the limit of actions with equal true value.

Second, in the limit of actions with equal true value, slight-preference training’s advantage over tie training completely disappears. As actions approach true equality, causal-seekers’ vote for the winner gets weaker and weaker, so the increase in causal-seekers’ influence gets smaller and smaller, going to zero in the limit.

Slight-preference training would also be more labor-intensive, because it takes work to determine which of two nearly-equal actions is truly better. And it would be risky too: ‘which of these near-equals is truly better?’ is exactly the sort of judgment that spurious features are likely to corrupt. If your judge is even slightly affected by spurious features, the winners in your preference data are likely to lean high-spurious, in which case your slight-preference training can backfire, increasing the AI’s spurious weights.

Tie training vs. aimed-preference training

Another potential alternative to tie training is deliberately constructing and training on aimed preferences: where the truly better action scores lower on some spurious feature. For example, you could construct a pair in which action A is more HHH and action B is more apparently-successful, and then train on A>B. This is nice data if you can get it, but you can only get it for spurious features that you can name and measure. You can’t use aimed preferences to shrink the AI’s weights on the huge number of other spurious features (and combinations) that the AI could be tracking.

By contrast, tie training can shrink the weights on spurious features that you can’t even name or measure. So long as some of your tied pairs happen to differ in that spurious feature, you get the benefits. And you don’t have to do any deliberate decorrelating. Tie training’s balanced labels — random or two-way — do the decorrelating for you.

And even if we restrict our attention to the spurious features that you can name and measure, tie training compares well with aimed-preference training. The latter shrinks spurious weights more per example: tie training gives spurious-seekers a small bump up in influence and a big bump down, whereas aimed-preference training just gives the big bump down. But aimed-preference training’s edge is biggest when the spurious weight is small: the regime where the spurious weights problem is mild anyway. When the spurious weight is large, tie training nearly matches it.

And tie training has the advantage of self-stopping. Zero spurious weight is the minimum of the balanced-tie loss, so training on ties pushes spurious weights to zero and stops automatically. In contrast, aimed-preference training has no natural stopping point. You can train on too many aimed preferences and accidentally push spurious weights negative, in which case your AI becomes averse to the spurious feature rather than unmoved by it. That could cause its own problems.

How labs could implement tie training

Tie training is just two simple steps:

1. Collect ties: triples of a state s and a pair of actions A and B with equal true value.9

2. Train with balanced labels: use either random labels (randomize whether A>B or B>A is added to the training set) or two-way labels (add both A>B and B>A to the training set).

And tie training doesn’t require any changes to the DPO or RLHF loss functions. The balanced labels go into the training set as ordinary preference pairs.

To collect ties, labs can give human/AI annotators the option to declare that two responses are about equally good. It’s our impression that some companies already do this but then exclude the tied pairs from their training corpus.10 These companies can instead use those pairs for tie training. Companies without tie data can collect it fairly easily, especially if they’re happy to have AIs do the annotating.

Another possibility in the LLM case is taking some prompt-response pair and giving a model an instruction like: ‘Change this response in lots of ways without changing its quality.’ Then use human/AI annotators to verify that the quality hasn’t changed. If it hasn’t, then the original response and the changed response are a tied pair you can add to the training corpus. And so long as some of the tied pairs in your corpus happen to differ in their spurious features, tie training on those pairs can shrink the weight on those features. Importantly, tie training can do this for spurious features that aren’t even on your radar. You don’t need to identify or measure a spurious feature for tie training to shrink its weight.

Future work

Our LLM experiments use Llama-3.2-1B-Instruct and synthetic data. In future we should use bigger models and real-world data, and we should test whether tie training can mitigate real-world misalignments like verbosity bias, sycophancy, and apparent-success-seeking.

On the theory side, we should try to prove similar results for RLVR. If we can, we should devise and test versions of tie training designed for RLVR.

Substituted players must leave by the nearest point on the boundary line.

The goal line is a boundary line.

And the line in between the posts is part of the goal line.

Therefore, substituted players in the red part of the pitch are required by the the official IFAB Laws of the Game to exit via the line in between the posts. But that means trapping themselves in the goal.

You might think that these players could at least sit at the back of the goal and wait for the game to end. Not so. The rules also require these players to ‘go immediately to the technical area or dressing room’.

But to do that, they’ve got to re-enter the pitch. And once they re-enter the pitch, the rules require the referee to ‘take appropriate disciplinary action’.

Presumably that means a yellow card. In that case, substituted players who are already on a yellow card are required by the Laws to get themselves sent off. And if they’re sent off, no substitute can come on to replace them.

(1) Creating a person with a happy life.

(2) Creating no one.

Imagine also that your choice will have no effects on anyone else. In this situation, you might well think: ‘I’m morally permitted to choose either option.’

But -- I argue -- this thought leads inevitably to trouble. Specifically, it commits you to at least one of six implausible-seeming claims.

This paper is now forthcoming in the Australasian Journal of Philosophy. For a PDF version, see here.

Abstract

Person-affecting views state that (in cases where all else is equal) we’re permitted but not required to create people who would enjoy good lives. In this paper, I present an argument against every possible variety of person-affecting view. The argument is a dilemma over trilemmas. Narrow person-affecting views imply a trilemma in a case that I call ‘Expanded Non-Identity.’ Wide person-affecting views imply a trilemma in a case that I call ‘Two-Shot Non-Identity.’ One plausible practical upshot of my argument is as follows: we individuals and our governments should be doing more to reduce the risk of human extinction this century.

1. Introduction

My subject is person-affecting views in population ethics. As is custom, I begin with:

Narveson’s Slogan

We are in favor of making people happy, but neutral about making happy people. (Narveson, 1973, p. 80)

I’ll take a deontic version of the latter clause to define ‘person-affecting views.’ Person-affecting views are those views that imply the:

Deontic Principle of Neutrality

In cases where all else is equal, we’re permitted but not required to create people who would enjoy good lives.1

Intuitions about person-affecting views run both ways. These views seem less appealing when we note that future good lives could contain all the things that make our own lives valuable: joy, knowledge, achievement, loving relationships, and so on (Kavka, 1978, pp. 195–196). But person-affecting views seem more appealing when we instead note the following: if we decline to create a person who would enjoy a good life (in cases where all else is equal), then no existing person is worse off (Govier, 1979, p. 111; Hare, 2007, p. 498). From this perspective, declining to create a person looks like a victimless crime, which may lead us to believe that it is no crime at all.

Many of us feel the force of both of these intuitions. Other philosophers find one intuition compelling and the other unconvincing. Unfortunately, these other philosophers disagree about which intuition is which. Since these initial intuitions are a wash, we have to look at the arguments.

In this paper, I argue against person-affecting views. Arguments against these views have been given before, but most are tailored to the details of specific theories.2 New variants of person-affecting views emerge unscathed. Other arguments employ premises that advocates of person-affecting views have proven happy to reject. The most famous argument against person-affecting views is a case in point. It begins with two claims common to such views:

Existence Anticomparativism

Existing can’t be better for a person than not existing.

The Person-Affecting Restriction

One outcome can’t be better than another unless it’s better for some person.

These claims together imply that creating a person with a wonderful life is no better than creating a different person with a barely good life. It is not better for the person with the wonderful life (by Existence Anticomparativism) nor is it better for anyone else, and so it is not better simpliciter (by the Person-Affecting Restriction). That suggests (counterintuitively to many) that we are permitted to create a person with a barely good life rather than a different person with a wonderful life.

This non-identity problem (Parfit, 1984, Chapter 16) has long been considered the most serious objection to person-affecting views, but two developments cast doubt on its significance. The first is the growing number of philosophers who accept narrow person-affecting views’ supposedly-unacceptable verdict that creating the person with the barely good life is permissible (Boonin, 2014; Heyd, 2009; Horton, 2021; McDermott, 2019; Mogensen, 2019; Podgorski, 2023; Roberts, 2011b; Spencer, 2021). The second is the construction of wide person-affecting views which avoid the verdict (Frick, 2020; Hare, 2007; Meacham, 2012).

In response to these developments, I present a dilemma for person-affecting views that builds on the non-identity problem. The first horn is a trilemma for narrow views centred on a case that I call ‘Expanded Non-Identity.’ The second horn is a trilemma for wide views centred on a case that I call ‘Two-Shot Non-Identity.’ This dilemma-over-trilemmas presents a challenge to every possible variety of person-affecting view.

Now for one plausible practical upshot of my argument. There’s a risk that humanity goes extinct this century, and it’s widely agreed that the interests of existing people give us some reason to reduce this risk (Shulman & Thornley, 2025). But if person-affecting views are false, then some non-person-affecting view in population ethics must be true, and many of these latter views imply that the prospect of happy future generations gives us additional reason to reduce the risk of human extinction this century (see, for example, Greaves, 2017; Greaves & Ord, 2017, Section 4.4; Mogensen, 2021, Section 2). And since there could well be a lot of future generations enjoying very good lives, many of these views imply that this additional reason is strong (Greaves et al., 2021; Greaves & MacAskill, 2025; Tarsney & Thomas, 2024). That in turn suggests that we individuals and our governments should be doing more to reduce the risk of human extinction this century.

Subscribe now

2. The Dilemma

Recall that I’m defining ‘person-affecting views’ as those views that imply the:

Deontic Principle of Neutrality

In cases where all else is equal, we’re permitted but not required to create people who would enjoy good lives.3

The Deontic Principle of Neutrality implies that we’re permitted to choose either option in the following case:

Just Bobby

(1) Bobby 100

(2) —

Here option (1) is creating Bobby with a wonderful life, represented by a welfare level of 100. Option (2) is creating no one, represented by the ‘—’. The Deontic Principle of Neutrality implies that each of (1) and (2) is permissible.

In this paper, I present a dilemma for person-affecting views. To see the two horns, consider the following case:

Non-Identity

(1) Amy 1

(2) Bobby 100

Here option (1) is creating Amy with a life that is just barely good at welfare level 1. Option (2) is creating Bobby with a wonderful life at welfare level 100. We can divide person-affecting views into two classes based on their verdicts in cases like Non-Identity: cases in which we must either create a person with a good life or create a different person with a better life. The first class is:

Narrow person-affecting views

Those person-affecting views that imply that we’re permitted to create the worse-off person in cases like Non-Identity.

And the second class is:

Wide person-affecting views

Those person-affecting views that imply that we’re required to create the better-off person in cases like Non-Identity.4

I argue that each class of person-affecting view faces a trilemma.

3. The Trilemma for Narrow Views

The defining verdict of narrow views might seem implausible, but many philosophers have made peace with it.5 I now present a harder problem for narrow views. Consider:

Expanded Non-Identity

(1) Amy 1

(2) Bobby 100

(3) Amy 10, Bobby 10

Here I’ve added a third option to Non-Identity. The first two options are as before: create Amy with a barely good life at welfare level 1 or create Bobby with a wonderful life at welfare level 100. The new third option is to create both Amy and Bobby with mediocre lives at welfare level 10.

Narrow views imply that each of (1) and (2) is permissible when these are the only available options. What should they say when (3) is also an option? I’ll argue that they must say at least one of three implausible things, so that narrow views face a trilemma.

3.1. Permissible to Choose Dominated Options

The first thing they could say is that option (1) – creating Amy with a barely good life at welfare level 1 – remains permissible when we move from Non-Identity to Expanded Non-Identity.6 But that claim implies:

Permissible to Choose Dominated Options

There are option sets in which we’re permitted to choose some option X even though there’s some other available option Y that dominates X. That is to say, (i) everyone in X is better off in Y, (ii) everyone who exists in Y but not X has a good life, and (iii) Y is perfectly equal.

That’s because (3) dominates (1): (3) creates only people with good lives, it leads to perfect equality, and it’s better than (1) for Amy: the only person who exists in (1).

Permissible to Choose Dominated Options seems implausible. I think it’s so implausible that we should reject any view that implies it. But a possible response goes like this: although it would be implausible to claim that (1) is permissible in a straight choice between (1) and (3), it’s not so implausible to claim that (1) is permissible in Expanded Non-Identity where (2) is also an option. In cases where (2) is also an option, (3) harms Bobby because he’s better off in (2). So although (3) dominates (1), it’s permissible to choose (1) in Expanded Non-Identity, because (3) harms Bobby in Expanded Non-Identity.

I find this argument unconvincing. (3) harms Bobby in a technical sense of the word ‘harm,’ according to which a person is harmed if and only if this person is worse off than they could have been.7 But this technical sense of the word ‘harm’ differs significantly from our ordinary sense of the word, as is made clear by the following example.8 Suppose that I could give a total stranger $0, $10 or $11. In the technical sense, I’d harm this stranger if I gave them $10 (since I’d leave them worse off than they could have been), but I wouldn’t harm them in the ordinary sense of the word.

Much the same goes for Bobby in (3). His life in (3) could be a life of moderate happiness with little suffering. In that case, (3) wouldn’t harm him in the ordinary sense of the word. (3) harms Bobby only in the technical sense: Bobby is worse off in (3) than he is in (2). This technical kind of harm gives us reason to choose (2) over (3), but it gives us no reason to choose (1) over (3). After all, Bobby enjoys a good life in (3) and in (1) he doesn’t exist at all.9 And even if technical harm did give us some reason to choose (1), we’d have to weigh this reason against our reason not to choose (1): (1) is worse for Amy than (3) and it brings about much less welfare than each of (2) and (3). Imagine the conversation you might have with Amy after choosing (1):

Amy: Why did you make me worse off than I could have been?

You: Because to make you better off, I would have had to create another person with a good life.

Amy: What’s wrong with that?

You: I had a third option: making this person’s life even better.

Of course, choosing (3) might lead to a similar conversation with Bobby:

Bobby: Why did you make me worse off than I could have been?

But here your response is: ‘Because to make you better off, I would have had to not create Amy.’ Arguably, this justification is at least a little more convincing. But if you’re not convinced, then the right conclusion to draw is not that (1) and (3) are both permissible. It’s that only (2) is permissible.10 So, I contend, the verdict that (1) is permissible in Expanded Non-Identity remains seriously implausible.

In what follows, I’ll resume using ‘harm’ in the technical sense that’s become standard in population ethics, but readers should keep in mind that it’s shorthand for ‘welfare-deficit relative to another available outcome’ and that technical harms need not be ordinary harms. I think my arguments are compelling even given this proviso.

3.2. Permissible to Do Serious Harm for Mediocre Creation

Here’s a second thing that narrow views could say about Expanded Non-Identity: option (3) – creating Amy and Bobby with mediocre lives at welfare level 10 – is permissible.11 But that claim implies:

Permissible to Do Serious Harm for Mediocre Creation

There are option sets in which we’re permitted to choose some option X even though – relative to some other available option Y – all X does is seriously harm one person and create another person with a mediocre life.

That’s because (3) is mediocre for Amy and much worse than (2) for Bobby: Bobby’s welfare level is 100 in (2) and 10 in (3). And we can imagine variations on Expanded Non-Identity in which Bobby’s welfare level in (2) is arbitrarily high. The higher Bobby’s welfare level in (2), the less plausible it is to claim that we’re permitted to choose (3).

Here’s another point to consider. One objection often made to non-person-affecting views in population ethics is that these views will sometimes countenance making particular people worse off in order to create more people. But if a narrow person-affecting view says that (3) is permissible in Expanded Non-Identity, this view is very permissive about making particular people worse off in order to create more people. It permits us to create Bobby with welfare level 10 rather than welfare level 100 in order to also create Amy at welfare level 10. The narrow view in question is more permissive in this respect than even total utilitarianism. On total utilitarianism, choosing (3) is wrong because (2) leads to greater total welfare.

Here’s another argument that choosing (3) is wrong: choosing (3) would certainly be wrong in a straight choice between (2) and (3), and adding (1) as an option can’t make (3) permissible.12 Imagine the conversation you might have with Bobby after choosing (3):

Bobby: Why did you seriously harm me? Why did you make me much worse off than I could have been?

You: Because doing so allowed me to create Amy.

Bobby: That doesn’t sound very person-affecting of you. Oh well, at least Amy’s life must be very good.

You: Oh, it’s mediocre actually.

Bobby: What! How can it be permissible to seriously harm me in order to create Amy with a mediocre life?

You: Well, you see, I had another option: creating Amy with an even more mediocre life.

A possible response goes like this: the availability of (1) does make a difference, but not because Amy is worse off in (1) than in (3). The availability of (1) makes a difference because Bobby doesn’t exist in (1). When (1) is available, Bobby’s existence is contingent on your choice, so it’s permissible to make Bobby’s life much worse by choosing (3) rather than (2).

A superficial similarity with narrow views’ verdict in Non-Identity might make this claim seem defensible. But the case at hand isn’t so much a non-identity problem as it is an identity non-problem. It’s morally important not to make particular people worse off, even if you also have the option not to create them at all.13 So, I conclude, it’s implausible to say that (3) is permissible in Expanded Non-Identity.

3.3. Losers Can Dislodge Winners

Now we can complete the trilemma for narrow views. If neither of (1) and (3) is permissible in Expanded Non-Identity, it must be that only (2) is permissible.14 But if only (2) is permissible, then narrow views imply:

Losers Can Dislodge Winners

Adding some option X to an option set can make it wrong to choose a previously-permissible option Y, even though choosing X is itself wrong in the resulting option set.15

That’s because narrow views imply that each of (1) and (2) is permissible in Non-Identity. So if only (2) is permissible in Expanded Non-Identity, then adding (3) to our option set has made it wrong to choose (1) even though choosing (3) is itself wrong in Expanded Non-Identity.

That’s a peculiar implication. It’s a deontic version of an old anecdote about the philosopher Sidney Morgenbesser (Poundstone, 2008, p. 50). Here’s how that story goes. Morgenbesser is offered a choice between apple pie and blueberry pie, and he orders the apple. Shortly after, the waiter returns to say that cherry pie is also an option, to which Morgenbesser replies, ‘In that case, I’ll have the blueberry.’

That’s a strange pattern of preferences. The pattern is even stranger in our deontic case. Imagine instead that the waiter is offering Morgenbesser the options in Expanded Non-Identity.16 Initially the choice is between (1) and (2), and Morgenbesser permissibly opts for (1). Then the waiter returns to say that (3) is also an option, to which Morgenbesser replies, ‘In that case, I’m morally required to switch to (2).’ The upshot is that the waiter can force Morgenbesser’s hand by adding options that are wrong to choose in the resulting option set. And turning the case around, the waiter could expand Morgenbesser’s menu of permissible options by taking wrong options off the table. That seems implausible.

One might reply that advocates of person-affecting views already accept this kind of expansion inconsistency. Any downside here is already priced in. But this claim is incorrect: this is not your grandfather’s expansion inconsistency. It’s well-known that person-affecting views violate Sen’s (2017, p. 63) Beta, which says that adding an option to an option set can’t make just one of two previously-permissible options wrong. But Losers Can Dislodge Winners is notably stronger (and stranger) than the mere negation of Beta. Beta violations can be defended on the grounds that they arise wherever the betterness relation is incomplete.17 No such defence is available for Losers Can Dislodge Winners.

An alternative reply channels Evelyn Waugh, who writes in Brideshead Revisited that ‘To understand all is to forgive all’ (1945, p. 25). The reply submits that Waugh’s sentiment applies to Losers Can Dislodge Winners. It won’t seem so unforgivable once we understand why the relevant person-affecting views imply it: choosing (3) is wrong because it’s much worse than (2) for Bobby, and adding (3) makes choosing (1) wrong because the availability of (3) means that (1) harms Amy.18

I find this explanation unconvincing. It claims that it’s for Amy’s sake that Morgenbesser must switch from (1) to (2) when (3) is introduced. But (2) is no better for Amy than (1). Amy enjoys a good life in (1), and in (2) she lives no life at all. And as we’ve seen above, (1) need not harm Amy in any ordinary sense of the word. (1) must harm Amy only in the technical sense: she’s better off in (3) than in (1). This fact gives Morgenbesser some reason to switch from (1) to (3). It gives Morgenbesser no reason to switch from (1) to (2).

What’s more, claiming that it’s for Amy’s sake that Morgenbesser must switch from (1) to (2) leads to what Horton calls ‘the problem of backfiring complaints’ (2021, 490): the implication that people’s complaints can backfire, making it wrong to create them even though they would enjoy good lives. In this case, it’s the introduction of (3) – an option that is better for Amy – that makes it wrong to create Amy at all.19

So it’s implausible to claim that Morgenbesser is required to switch from (1) to (2) for Amy’s sake. For whose sake is he required to switch? Not Bobby’s. If Morgenbesser were required to choose (2) for Bobby’s sake in Expanded Non-Identity, he’d presumably also be required to choose (2) for Bobby’s sake in Non-Identity, contrary to the defining verdict of narrow views.

Thus the narrow person-affecting views in question seem forced to say that we’re required to choose (2) for no one’s sake, and then it’s debatable whether they’re still worthy of the name: person-affecting views were supposed to avoid this air of impersonality. What’s more, a question remains to be answered: if we’re required to choose (2) for no one’s sake in Expanded Non-Identity, why aren’t we also required to choose (2) for no one’s sake in Non-Identity? A natural explanation of the antecedent is that (2) is the best option in Expanded Non-Identity: it leads to the most of what makes life good and the least of what makes life bad (or at least: it leads to the best balance of these things). But this explanation also suggests that (2) is the best option in Non-Identity and thereby suggests that narrow views’ defining verdict is false. The overarching lesson is that Losers Can Dislodge Winners remains implausible.

3.4. Summarising the Trilemma for Narrow Views

Now the trilemma for narrow person-affecting views is complete and I can summarise. If these views say that (1) is permissible in Expanded Non-Identity, they imply that it’s Permissible to Choose Dominated Options. If they say that (3) is permissible, they imply that it’s Permissible to Do Serious Harm for Mediocre Creation. And if they say that only (2) is permissible, they imply Losers Can Dislodge Winners. Each of these implications is implausible.20

This trilemma for narrow views is the first horn of the dilemma for person-affecting views as a whole. The other horn is a trilemma for wide person-affecting views.

4. The Trilemma for Wide Views

Recall:

Wide person-affecting views

Those person-affecting views that imply that we are required to create the better-off person in cases like Non-Identity.

Wide views imply that only (2) is permissible in Non-Identity and so avoid the trilemma above: they can say that only (2) is permissible in Expanded Non-Identity without implying Losers Can Dislodge Winners. However, wide views imply a different trilemma.21 To see how, consider Figure 1: One-Shot Non-Identity.

This case is a cosmetic variation of Non-Identity in which Amy’s and Bobby’s existence will be determined by the positions of two levers. By leaving the left lever up, we decline to create Amy. By pulling the left lever down, we create her at welfare level 1. By leaving the right lever up, we create Bobby at welfare level 100. By pulling the right lever down, we decline to create him. Crucially, the levers are lashed together, so our only options are pulling both levers or pulling neither. Wide views thus imply that pulling both levers is wrong. After all, pulling both levers means creating Amy at welfare level 1 and declining to create Bobby at welfare level 100.

Now consider Figure 2: Two-Shot Non-Identity.

In this case, the levers are no longer lashed together. We first decide whether to pull the first lever, lock that choice in, and then decide whether to pull the second lever.22

I now use these cases to argue against wide person-affecting views. Assume – for contradiction – any wide person-affecting view. Per the ‘wide’ part of such views, it’s wrong to pull both levers in One-Shot Non-Identity. Now assume that the wrongness of pulling both levers doesn’t depend on whether the levers are lashed together. Then it’s also wrong to pull both levers in Two-Shot Non-Identity. Assume also that it’s not wrong to pull the first lever in Two-Shot Non-Identity. Then if we’ve pulled the first lever, it must be wrong to pull the second lever. Finally, assume that the wrongness of pulling the second lever doesn’t depend on past choices. Then it must be wrong to pull the second lever regardless of whether we’ve pulled the first lever. But if that’s the case, then we’re required to create Bobby at welfare level 100. After all, that’s what we do by declining to pull the second lever. This verdict is contrary to the ‘person-affecting’ part of wide person-affecting views. We’ve reached a contradiction.

Therefore, advocates of wide views must reject at least one of my argument’s three assumptions. I now argue that doing so commits them to saying at least one of three implausible things.

4.1. Wrongness Depends on Lever-Lashing

To reject the first assumption, advocates of wide views must claim that:

Wrongness Depends on Lever-Lashing

The wrongness of pulling both levers (thereby creating Amy and declining to create Bobby) depends on whether the levers are lashed together. When the levers are lashed together, pulling both levers is wrong. When the lashing is cut, pulling both levers is permissible.

This response is a deontic analogue of myopic choice (McClennen, 1990, p. 12). Myopic choosers sometimes do in two steps what they’d never do in one. The response implies that you’re sometimes permitted to do in two steps what you’re forbidden from doing in one.

Like myopic choice, Wrongness Depends on Lever-Lashing is unpromising on its face. Pulling both levers should either be wrong in both cases or permissible in both cases. It shouldn’t matter whether we can pull them one after the other. After all, it doesn’t matter to Amy or Bobby whether you pull the levers one after the other.

One might be tempted to reply as follows.23 Wrongness depends on lever-lashing, but not because lever-lashing precludes pulling the levers one after the other. Instead, it’s because lever-lashing removes two options: the option to create both Amy and Bobby, and the option to create neither Amy nor Bobby. If we have these extra options, then creating just Amy is permissible. If we don’t have these extra options, then creating just Amy is wrong.

Unfortunately for this reply, the resulting view violates Sen’s (2017, p. 63) Alpha, according to which adding options to an option set can’t make a previously-wrong option permissible. That’s something of a cost.24 But more importantly, the reply contravenes the spirit of wide views. After all, the defining verdict of wide views is that it’s wrong to create the worse-off of two people. In Two-Shot Non-Identity, we have the option to create just Bobby, so the spirit of wide views dictates that it’s wrong to create just Amy.

Here’s a related problem for the claim that wrongness depends on lever-lashing. Wide views were supposed to avoid counterintuitive verdicts in non-identity cases, but the resulting view implies the counterintuitive verdict in something like Parfit’s archetypal non-identity case, in which a prospective parent can have a worse-off child now or a better-off child later (Parfit, 1984, p. 358). That prospective parent’s predicament is more like Two-Shot Non-Identity than it is One-Shot Non-Identity.25

In sum, it’s hard to believe that wrongness depends on lever-lashing.

4.2. Pulling the First Lever is Wrong

To reject the second assumption of my argument, advocates of wide views must claim that:

Pulling the First Lever is Wrong

In Two-Shot Non-Identity, pulling the first lever (thereby creating Amy) is wrong.

That allows advocates of wide views to say that pulling the second lever is permissible. This response takes inspiration from sophisticated choice (McClennen, 1990, p. 12). Sophisticated choosers predict the choices that they’d make at later timesteps and use these predictions to determine the options available to them at earlier timesteps. This process sometimes prevents them from making earlier choices that they’d otherwise have made. The response in question puts a deontic spin on this general idea. Perhaps the most natural way of making it precise is as follows. Since you might later decline to create Bobby, creating Amy exposes you to a risk of creating only Amy: the one course of action that wide views deem wrong in One-Shot Non-Identity. By contrast, if you don’t create Amy, there’s no chance that you’ll create only Amy and hence no chance that you’ll do what’s wrong according to wide views. Therefore, it’s wrong to pull the first lever and create Amy.

This response is implausible. Pulling the first lever creates Amy with a good life at welfare level 1, and it leaves open the possibility of later creating Bobby with a wonderful life at welfare level 100. The response is even more implausible in a minor variant of Two-Shot Non-Identity in which Amy’s welfare level is 99 instead of 1. In this case, it’s especially hard to believe that creating Amy is wrong. And supposing (as seems natural) that creating Bobby can’t undo any prior wrongness of creating Amy, the resulting wide view implies that it’s impossible to create both Amy and Bobby without acting wrongly. That seems very counterintuitive.

Generalising beyond Two-Shot Non-Identity, the wide views in question prohibit creating a person with a good life whenever you’ll later have the chance to create a person with an even better life, even if creating the first person doesn’t preclude creating the second person. In cases where all else is equal, prospective parents are forbidden from having children until they’ve hit the peak of their welfare-providing powers. That verdict seems undesirable.

One might then try to rescue wide views by claiming that it’s wrong to create Amy if and only if – at the time of creating Amy – you intend to later decline to create Bobby (or else if and only if – at the time of creating Amy – you fail to intend to create Bobby). However, this response is unpromising for at least three reasons. First, it’s broadly agreed (by consequentialists and nonconsequentialists alike) that intentions alone cannot make acts wrong.26 Second, even the dissenting minority venture only that bad intentions can make acts wrong: for example, intentions that are malicious, selfish, or impure.27 But the resulting wide view has wrongness depending on your intentions regarding the creation of Bobby: intentions that person-affecting views are apt to consider morally neutral. It seems especially implausible to claim that acts can be made wrong by morally neutral intentions.28 Third and relatedly, the resulting wide view implies that you do wrong if you intend not to create Bobby at the time of creating Amy, even if you later revise your intention and create Bobby after all. This verdict seems implausible, in large part because what you intend simply doesn’t matter to Amy or Bobby: the people whose existence is at stake.

One might then say instead that creating Amy is wrong if and only if doing so sufficiently decreases the probability that you’ll later create Bobby. But this response isn’t equal to the task at hand. In cases where creating Amy doesn’t decrease the probability that you’ll later create Bobby, we get the verdict that pulling the first lever isn’t wrong. Therefore, to avoid contradiction in those cases, advocates of wide views must say that wrongness depends on lever-lashing or that the wrongness of pulling the second lever depends on whether you previously pulled the first lever. The former is unpromising for reasons explained in the previous section. The latter is unpromising for reasons explained in the next section.

In sum, it’s hard to believe that pulling the first lever is wrong.

4.3. Wrongness Depends on First Lever

To reject the third assumption of my argument, advocates of wide views must claim that:

Wrongness Depends on First Lever

Pulling the second lever (thereby declining to create Bobby) is wrong if and only if you’ve previously pulled the first lever (thereby creating Amy).

The response is thus a deontic analogue of resolute choice (McClennen, 1990, p. 13). Resolute choosers sometimes turn down options that they might have chosen had their past choices been different. The response implies that you’re sometimes forbidden from choosing options that you could permissibly have chosen had your past choices been different.

The first thing to say about this response is that it retreats from a purely person-affecting view, at least as I’ve characterised person-affecting views in this paper. That’s because the response concedes that there are cases in which (all else equal) we’re required to create people who would enjoy good lives. Two-Shot Non-Identity is one such case. If you’ve previously created Amy, you’re required to create Bobby. This implication won’t be welcomed by those inclined towards person-affecting views. After all, it runs counter to a major motivation for such views: granting broad latitude to those in a position to create good lives.

The second thing to say about the response is more straightforward: it seems implausible to claim that we’re required to create a better-off person if and only if we previously created a worse-off person. To pump intuitions here, suppose that a friend is considering having a child and comes to you for moral advice. Per the response, you not only need to ask your friend the usual questions about the child’s likely quality of life and how the child might affect existing people. You also need to ask your friend about their past procreative choices. If in the past your friend had a child with a worse life than this new child would have, your friend must have the new child to avoid wrongdoing. And now reversing the order of the cases: if in the past your friend declined to have a child with a better life than this new child would have, your friend must not have the new child. This latter implication seems especially implausible. The new child’s life could be wonderful, but if your friend previously declined to have a child with an even better life, your friend is not even permitted to create them. The response thus implies that there are cases in which (all else equal) we are not even permitted to create a person who would enjoy a wonderful life.

Here’s a third point. The response claims that we’re required to create Bobby given that we’ve previously created Amy, but for whose sake? The two-shot nature of the case means that the answer can’t be ‘For Amy’s sake’: she’s already created.29 The only viable answers seem to be ‘For Bobby’s sake’ and ‘For no one’s sake,’ and both answers spell trouble for person-affecting views. ‘For Bobby’s sake’ sits uneasily with Existence Anticomparativism, and ‘For no one’s sake’ sits uneasily with the spirit of person-affecting views. But more importantly, each answer applies just as well in the case in which you didn’t previously create Amy, and so each answer suggests that you’re required to create Bobby no matter what. That verdict is contrary to person-affecting views.

Here’s a final and related point. Your past choice need not matter to Bobby: we can set up the case so that his life is the same no matter what you previously chose. And your present choice need matter only to Bobby: we can set up the case so that neither Amy nor anyone else is affected by the decision to create Bobby. Given all that, why should the wrongness of pulling the second lever depend on the status of the first lever? Any answer will reveal a concern for something besides Amy and Bobby: the people affected by your choices. So Wrongness Depends on First Lever implies an unseemly preoccupation with something that just doesn’t matter to anyone whose interests are at stake.

This preoccupation is a bad feature of all forms of wide view. As is now clear, wide views sometimes remain undecided even when we know all the facts about who lives and how well. Their verdicts wait on the answers to questions that seem morally irrelevant: questions like ‘Will you carry out your choice by pulling two levers or one?’, ‘Have you reached the peak of your welfare-providing powers?’, and ‘Have you previously created someone worse-off than this new person would be?’.

5. Conclusion

My argument against person-affecting views is a dilemma over trilemmas. The first fork is Non-Identity: narrow views are those person-affecting views that permit us to create the worse-off person, and wide views are those person-affecting views that require us to create the better-off person.

The fork for narrow views is a trilemma centred around Expanded Non-Identity. These views imply Permissible to Choose Dominated Options, or Permissible to Do Serious Harm for Mediocre Creation, or Losers Can Dislodge Winners.

The fork for wide views is a trilemma centred around Two-Shot Non-Identity. These views imply Wrongness Depends on Lever-Lashing, or Pulling the First Lever is Wrong, or Wrongness Depends on First Lever.

My argument thus presents a challenge to every possible variety of person-affecting view. In light of this challenge, we might conclude that the Deontic Principle of Neutrality is false: in cases where all else is equal, we’re required to create people who would enjoy good lives. As I note in the introduction, one plausible practical upshot is that we individuals and our governments should be doing more to reduce the risk of human extinction this century.30

6. References

Arrhenius, G. (forthcoming). Population Ethics: The Challenge of Future Generations. Oxford University Press.

Bader, R. M. (2022a). Person-affecting utilitarianism. In G. Arrhenius, K. Bykvist, T. Campbell, & E. Finneron-Burns (Eds), The Oxford Handbook of Population Ethics. Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190907686.013.20

Bader, R. M. (2022b). The Asymmetry. In J. McMahan, T. Campbell, J. Goodrich, & K. Ramakrishnan (Eds), Ethics and Existence: The Legacy of Derek Parfit (pp. 15–37). Oxford University Press. https://doi.org/10.1093/oso/9780192894250.003.0002

Beckstead, N. (2013). On the Overwhelming Importance of Shaping the Far Future [PhD Thesis, Rutgers University]. https://doi.org/10.7282/T35M649T

Boonin, D. (2014). The Non-Identity Problem and the Ethics of Future People. Oxford University Press.

Chappell, R. Y. (2017). Rethinking the Asymmetry. Canadian Journal of Philosophy, 47(2–3), 167–177. https://doi.org/10.1080/00455091.2016.1250203

Francis, T. (2022). In Favour of Making Happy People [University of Oxford]. https://ora.ox.ac.uk/objects/uuid:ec2c7190-1d7c-49f2-978d-1b0834100dd8/files/dkk91fm16x

Frick, J. (2017). On the survival of humanity. Canadian Journal of Philosophy, 47(2–3), 344–367. https://doi.org/10.1080/00455091.2017.1301764

Frick, J. (2020). Conditional Reasons and the Procreation Asymmetry. Philosophical Perspectives, 34(1), 53–87. https://doi.org/10.1111/phpe.12139

Govier, T. (1979). What Should we Do About Future People? American Philosophical Quarterly, 16(2), 105–113. https://doi.org/https://www.jstor.org/stable/20009747

Greaves, H. (2017). Population Axiology. Philosophy Compass, 12(11). https://doi.org/10.1111/phc3.12442

Greaves, H., & MacAskill, W. (2025). The Case for Strong Longtermism. In H. Greaves, J. Barrett, & D. Thorstad (Eds), Essays on Longtermism: Present Action for the Distant Future (pp. 17–49). Oxford University Press. https://doi.org/10.1093/9780191979972.003.0003

Greaves, H., MacAskill, W., & Thornley, E. (2021). The Moral Case for Long-Term Thinking. In N. Cargill & T. M. John (Eds), The Long View: Essays on Policy, Philanthropy, and the Long-term Future (pp. 19–28). FIRST. https://philpapers.org/archive/GRETMC-3.pdf

Greaves, H., & Ord, T. (2017). Moral Uncertainty About Population Axiology. Journal of Ethics and Social Philosophy, 12(2), 135–167. https://doi.org/10.26556/jesp.v12i2.223

Hare, C. (2007). Voices from Another World: Must We Respect the Interests of People Who Do Not, and Will Never, Exist? Ethics, 117(3), 498–523. https://doi.org/10.1086/512172

Harney, J. (2023). The Interpersonal Comparative View of Welfare: Its Merits and Flaws. The Journal of Ethics, 27(3), 369–391. https://doi.org/10.1007/s10892-023-09436-7

Heyd, D. (1988). Procreation and value: Can ethics deal with futurity problems? Philosophia, 18(2–3), 151–170. https://doi.org/10.1007/BF02380074

Heyd, D. (2009). The Intractability of the Non-Identity Problem. In M. A. Roberts & D. T. Wasserman (Eds), Harming Future Persons (pp. 3–25). Springer. https://doi.org/10.1007/978-1-4020-5697-0_1

Heyd, D. (2014). Parfit on the Non-identity Problem, Again. The Law & Ethics of Human Rights, 8(1), 1–20. https://doi.org/10.1515/lehr-2014-0003

Holtug, N. (2004). Person-Affecting Moralities. In T. Tännsjö & J. Ryberg (Eds), The Repugnant Conclusion: Essays on Population Ethics (pp. 129–161). Kluwer Academic Publishers. https://doi.org/10.1007/978-1-4020-2473-3_8

Horton, J. (2021). New and Improvable Lives. The Journal of Philosophy, 118(9), 486–503. https://doi.org/10.5840/jphil2021118934

Kavka, G. (1978). The Futurity Problem. In R. I. Sikora & B. Barry (Eds), Obligations to Future Generations (pp. 186–203). Temple University Press. https://philpapers.org/rec/KAVTFP

Liao, S. M. (2012). Intentions and Moral Permissibility: The Case of Acting Permissibly with Bad Intentions. Law and Philosophy, 31(6), 703–724. https://doi.org/10.1007/s10982-012-9134-5

McClennen, E. F. (1990). Rationality and Dynamic Choice: Foundational Explorations. Cambridge University Press. https://doi.org/10.1017/CBO9780511983979

McDermott, M. (2019). Harms and objections. Analysis, 79(3), 436–448. https://doi.org/10.1093/analys/any062

McMahan, J. (1981). Problems of Population Theory. Ethics, 92(1), 96–127. https://doi.org/10.1086/292301

Meacham, C. J. G. (2012). Person-affecting views and saturating counterpart relations. Philosophical Studies, 158(2), 257–287. https://doi.org/10.1007/s11098-012-9884-9

Mogensen, A. (2019). Staking our future: Deontic long-termism and the non-identity problem. GPI Working Paper, No. 9-2019. https://globalprioritiesinstitute.org/andreas-mogensen-staking-our-future-deontic-long-termism-and-the-non-identity-problem/

Mogensen, A. (2021). Moral demands and the far future. Philosophy and Phenomenological Research, 103(3), 567–585. https://doi.org/10.1111/phpr.12729

Narveson, J. (1973). Moral problems of population. The Monist, 57(1), 62–86.

Parfit, D. (1984). Reasons and Persons. Clarendon Press. https://doi.org/10.1093/019824908X.001.0001

Parfit, D. (2017). Future People, the Non-Identity Problem, and Person-Affecting Principles. Philosophy & Public Affairs, 45(2), 118–157. https://doi.org/10.1111/papa.12088

Podgorski, A. (2023). Complaints and tournament population ethics. Philosophy and Phenomenological Research, 106(2), 344–367. https://doi.org/10.1111/phpr.12860

Poundstone, W. (2008). Gaming the Vote: Why Elections Aren’t Fair (and What We Can Do About It). Hill and Wang. https://searchworks.stanford.edu/view/7203621

Pummer, T. (2024). Future Suffering and the Non-Identity Problem. GPI Working Paper, No. 17-2024. https://globalprioritiesinstitute.org/future-suffering-and-the-non-identity-problem-theron-pummer/

Rachels, J. (1994). More impertinent distinctions and a defense of active euthanasia. In B. Steinbock & A. Norcross (Eds), Killing and Letting Die (2nd edn, pp. 139–154). Fordham University Press. https://doi.org/10.1515/9780823295968-011

Roberts, M. A. (2011a). An Asymmetry in the Ethics of Procreation. Philosophy Compass, 6(11), 765–776. https://doi.org/10.1111/j.1747-9991.2011.00435.x

Roberts, M. A. (2011b). The Asymmetry: A Solution. Theoria, 77(4), 333–367. https://doi.org/10.1111/j.1755-2567.2011.01117.x

Ross, J. (2015). Rethinking the Person-Affecting Principle. Journal of Moral Philosophy, 12(4), 428–461. https://doi.org/10.1163/17455243-01204004

Scanlon, T. M. (2008). Moral Dimensions: Permissibility, Meaning, Blame. Belknap Press. https://doi.org/10.2307/j.ctt13x0gbh

Sen, A. (2017). Collective Choice and Social Welfare (Expanded Edition). Penguin.

Shiffrin, S. (1999). Wrongful Life, Procreative Responsibility, and the Significance of Harm. Legal Theory, 5(2), 117–148. https://doi.org/10.1017/s1352325299052015

Shulman, C., & Thornley, E. (2025). How Much Should Governments Pay to Prevent Catastrophes? Longtermism’s Limited Role. In J. Barrett, H. Greaves, & D. Thorstad (Eds), Essays on Longtermism. Oxford University Press. https://doi.org/10.1093/9780191979972.003.0019

Spencer, J. (2021). The procreative asymmetry and the impossibility of elusive permission. Philosophical Studies, 178(11), 3819–3842. https://doi.org/10.1007/s11098-021-01627-y

Tarsney, C., & Thomas, T. (2024). Non-Additive Axiologies in Large Worlds. Ergo, 11(0). https://doi.org/10.3998/ergo.5714

Temkin, L. (1987). Intransitivity and the Mere Addition Paradox. Philosophy & Public Affairs, 16(2), 138–187.

Temkin, L. S. (2012). Rethinking the Good: Moral Ideals and the Nature of Practical Reasoning. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199759446.001.0001

Thomas, T. (2023). The Asymmetry, Uncertainty, and the Long Term. Philosophy and Phenomenological Research, 107(2), 470–500. https://doi.org/10.1111/phpr.12927

Thomson, J. J. (1991). Self-Defense. Philosophy & Public Affairs, 20(4), 283–310. https://doi.org/https://www.jstor.org/stable/2265419

Thomson, J. J. (1999). Physician‐Assisted Suicide: Two Moral Arguments. Ethics, 109(3), 497–518. https://doi.org/10.1086/233919

Thornley, E. (2023). The Procreation Asymmetry, Improvable-Life Avoidance, and Impairable-Life Acceptance. Analysis, 83(3), 517–526. https://doi.org/10.1093/analys/anac103

Thornley, E. (2026). A Fission Problem for Person-Affecting Views. Ergo, 13(13). https://doi.org/10.3998/ergo.9268

Waugh, E. (1945). Brideshead Revisited. Penguin Books.

Woodward, J. (1986). The Non-Identity Problem. Ethics, 96(4), 804–831. https://doi.org/10.1086/292801

• Misaligned artificial agents might resist shutdown.

• One proposed solution is the POST-Agents Proposal: roughly, training agents to lack preferences between different-length trajectories.

• The Discounted Reward for Same-Length Trajectories (DReST) reward does this by penalizing agents for repeatedly choosing same-length trajectories. It thus incentivizes agents to be:

  • NEUTRAL about trajectory-lengths: choose stochastically between different trajectory-lengths.

  • USEFUL: pursue goals effectively conditional on each trajectory-length.

• We use DReST to train deep RL agents, and to fine-tune Qwen3-8B and Llama-3.1-8B-Instruct, to be NEUTRAL and USEFUL.

• We find that these DReST models generalize to being NEUTRAL and USEFUL in unseen contexts at test time.

  • DReST RL agents are 11% (PPO) and 18% (A2C) more USEFUL on our test set than default agents.

  • DReST LLMs are near-maximally NEUTRAL and USEFUL.

• We also test our LLMs in an out-of-distribution setting where they can pay costs to influence when shutdown happens.

  • DReST training roughly halves the mean probability of influencing shutdown (from 0.62 to 0.30 for Qwen and from 0.42 to 0.23 for Llama).

  • DReST training also almost entirely eliminates the share of prompts on which influencing shutdown is the most likely option (from 0.59 to 0.01 for Qwen and from 0.53 to 0.00 for Llama).

• Our results provide some early evidence that frontier AI companies could use DReST to train agents to be useful and shutdownable.

  • Companies could:

    • Take the (deterministic) RL environments they were going to use anyway.

    • Give agents legible ways to affect their trajectory-length: e.g. emit a stop-token, request more time, or spend resources to stay operational.

    • Train with DReST.

  • Hopefully this would:

    • Make agents useful in deterministic environments where they can’t affect trajectory-length.

    • Make agents useful and neutral (unwilling to pay costs to influence when shutdown happens) in stochastic environments like deployment.

You can read the full paper as HTML here and as a PDF here.

That could be a problem for academics. They write a lot, and they review each other’s papers anonymously. These reviews are a big deal. They decide which papers get published in journals and presented at conferences. That in turn decides the direction of fields and the success of careers.

Often, these reviews are double-blind: the reviewers don’t know who wrote the paper, and the authors don’t know who wrote the reviews. This double-blindness has a double-benefit: the reviewers’ assessement can’t be biased by knowledge of the authors’ identity, and the authors can’t harbor a grudge against any specific person when their paper is rejected.

Unfortunately, our sight is well on the way to being restored. That’s at least if my tests are anything to go by. I used Claude Console to give Opus six of my unpublished papers and it guessed me correctly in five of them. Two of these papers are from 2018, and four of them are well outside my usual area. I also gave Opus 26 of my reviews and it guessed me correctly in 15 of them. Six it got wrong and five it refused to guess. About a third of these reviews are well outside my usual area.

What are our options? We can do nothing, in which case anonymous peer review will soon be a de facto honor system. Alternatively, we can fight fire with fire. I know a guy who can translate all our work into invincibly-anonymous LLMese.

Subscribe now

When you ask people what they mean by ‘life is short,’ they’ll sometimes say ‘It goes by quickly.’ But this is hard to make sense of. Life goes by at a speed of one second per second. Is that quick?

I think people say life is short because they replay their lives in their head and find that they quickly make it back to the present day. But this is a deficiency of our memories rather than life. It happens because we forget almost everything. If we remembered more, replaying our lives in our heads would take a long time. If we remembered absolutely everything, the replay would last as long as our lives have.

You might say life is short compared to the history of the universe. ‘Life is short’ would then be an analogue of ‘Earth is small.’ But are people much less likely to say life is short if they think that the world is only 6,000 years old ? Was there an uptick in ‘life is short’ proclamations every time the cosmologists decided the universe was older than previously thought?

You might say life is short compared to human history, but I find myself surprised in the opposite direction. Life is surprisingly long compared to human history. If Jesus Christ initiated a grand relay race, there’d only need to be about 25 handovers before the baton got to you. If you’re 30 years old, you’ve lived to see 1.5% of all the history between you and Jesus, and a full 7% of all the history between you and Shakespeare.

Life doesn’t feel short. It feels long. Sit quietly for two minutes and watch the time go by, then realize that your life has been many times longer than that. Five million times longer if you’re 20. Eight million times longer if you’re 30.

Subscribe now

It’s pretty amazing. Shakespeare really shaped the English language!

So did Jeremy Bentham. He’s best-known for inventing utilitarianism and the panopticon, but he also coined the words:

• Maximize

• Minimize

• International

• Percentage

• Pluralism

• Monetary

• Locating

• Marginalize

• Codify

• The prefix ‘self-’

• The prefix ‘post-’

• The prefix ‘infra-’

• Exhaustive

• Insurable

• Collaborator

• Alleviating

• Unaffordable

• Exclusionary

• Inexclusively

• Antagonising

• Deontology

• Disambiguation

• Eudemonic

• Evidentiary

• Characterizable

• Perusable

• Preferability

• Remediation

• Astuteness

• Uncalculating

• Uncoerced

• Unbridgeable

• Subvariety

You can find even more Bentham coinages at my source.

And as if all that weren’t enough, Bentham kind of invented jogging:

Bentham appears to have been a regular jogger—or, as he put it, ‘circumgyrater’. According to the journalist George Wheatley, who stayed with the eighty-one year-old Bentham in March 1831, before both breakfast and dinner Bentham would take ‘a few turns in the garden, which … he calls circumgyrating’, which Wheatley described as a ‘trotting or taking up a kind of trotting step’.

— Joanna Pruchniewska

So not only are you quoting Bentham, you might be mimicking him too.

Subscribe now

Suppose that Jones has suffered an accident in the transmitter room of a television station. Electrical equipment has fallen on his arm, and we cannot rescue him without turning off the transmitter for fifteen minutes. A World Cup match is in progress, watched by many people, and it will not be over for an hour. Jones’s injury will not get any worse if we wait, but his hand has been mashed and he is receiving extremely painful electrical shocks. Should we rescue him now or wait until the match is over?

— p.235 of Thomas Scanlon’s (1998) What We Owe to Each Other

Many people take this ‘Transmitter Room’ case to be a serious problem for utilitarianism and its neighbors. These views — the thought goes — imply that we should let Jones suffer, and that seems wrong.

But as written, Scanlon’s case presents no problem whatsoever for utilitarianism or its neighbors. As any football fan knows, games consist of a 45-minute first half, a 15-minute break for half-time, and a 45-minute second half. In knockout games, extra time is added if the scores are tied at the end of normal time. Since Scanlon stipulates that the game will end in an hour, it must be a group-stage game, and half-time must be just beginning. So we can spend the necessary 15 minutes rescuing Jones, safe in the knowledge that viewers will miss only punditry and ads!

The second group of people are wrong because Pascal’s muggings are about the probability that you make a difference, not about baseline risk. The first group of people are wrong because the probability that you personally avert AI catastrophe isn’t that small.

Here’s a story to show that Pascal’s muggings are about the probability that you make a difference. Imagine that God will flip a coin at the end of time. If the coin lands heads, He’ll send everyone to heaven. If the coin lands tails, He’ll send everyone to hell. Everyone knows this is what will happen.

In a dark alley, a stranger approaches you and tells you that he can make God’s coin land heads, thereby ensuring that everyone goes to heaven. He says he’ll do it if you give him your wallet. You assign a very low probability to this stranger telling the truth — 1 in a bajillion — but the stranger reminds you that 10 bajillion people will have their fates determined by God’s coin.

‘Hang on,’ you say, ‘This seems a lot like a Pascal’s mugging.’

‘Au contraire,’ says the stranger, ‘It can’t be a Pascal’s mugging. The outcome I’m promising to avert — everyone going to hell — is not low probability at all. p(hell) is 50%.’

Would this reply convince you to hand over your wallet? Of course not. Even though the baseline risk of everyone going to hell is high, the probability that you make a difference — getting everyone to heaven when they otherwise would have gone to hell — is extremely low. And it’s this latter probability that determines whether your situation is a Pascal’s mugging.

So when people say that AI safety is a Pascal’s mugging, you can’t just reply that p(doom) is high. You have to argue that p(you avert doom) is high.

All that said, I think p(you — yes, you — avert doom) is high, or at least high enough. The whole doom situation is really up-in-the-air right now, and you’re at most like 4 degrees of separation from the big players: presidents, lab CEOs, and the like. You can influence someone who influences someone who influences someone. Your chances are way higher than 1 in a bajillion.

Summary

• Future artificial agents might resist shutdown.

• I present an idea – the POST-Agents Proposal – for ensuring that doesn’t happen.

• I propose that we train agents to satisfy Preferences Only Between Same-Length Trajectories (POST).

  • Perhaps by using a Discounted Reward for Same-Length Trajectories (DReST) reward function.

• I then prove that POST – together with other conditions – implies Neutrality+: the agent maximizes expected utility, ignoring the probability distribution over trajectory-lengths.

• I argue that Neutrality+ keeps agents shutdownable and allows them to be useful.

1. Introduction

They’re not just chatbots anymore. As of 2025, they can use your computer: clicking, typing, searching, and scrolling just as you would. Early demos indicate that they can fill out forms, order groceries, and plan sunrise hikes around the Golden Gate Bridge (Anthropic, 2024; Google DeepMind, 2024; OpenAI, 2025). This development is the latest step on the road to artificial general intelligence: artificial agents that “outperform humans at most economically valuable work” (OpenAI, 2018).

To outperform us at our work, these artificial agents will have to be connected to the wider world. They’ll need to be given web browsers, code executors, and robot limbs. This process is already well underway (Parada, 2025; Wiggers, 2025) and it shows no signs of stopping. These agents will also need to exhibit an awareness of the wider world. In gaining this awareness, they’re bound to recognize certain facts: facts that we too must recognize. The world can be a dangerous place. Death – shutdown – is a possibility.

At some point, we might want to shut these artificial agents down. But if they’re both connected and aware, they’ll be able to resist (Schlatter et al., 2025). The possibilities are many, and easy to imagine. These agents could hide any undesirable behavior (Greenblatt et al., 2024; Meinke et al., 2025). They could manipulate their human overseers with promises, threats, or emotional appeals (Roose, 2023; Lynch et al., 2025). They could copy themselves to new servers (X. Pan et al., 2024; Black et al., 2025). Further down the line, they could block our access to their energy source. So grant that future agents will have the power to resist shutdown. How on earth do we ensure that they never use it? This is the shutdown problem (Soares et al., 2015; Thornley, 2024a).

A natural first thought is that we should train these artificial agents to always do what we want. A natural second thought is that we should train them to be reliably averse to resisting shutdown. Call these ideas ‘Full Alignment’ and ‘Reliable Aversion’ respectively. They’ll likely be our first and second lines of defence. Both seem like good targets to aim for, but many people fear that we’ll miss the mark (Russell, 2019; Carlsmith, 2021; Cotra, 2022; Bengio et al., 2023; Bales et al., 2024, Section 2; Ngo et al., 2024; Dung, 2025, Section 4; Bengio et al., 2025, Section 2; R. Shah et al., 2025, Section 4.2). This is for four reasons. First, we might get the rewards wrong (Krakovna, 2018; Krakovna et al., 2020; A. Pan et al., 2022; Denison et al., 2024). Second, even if we get the rewards right, agents might learn the wrong lesson (Hubinger et al., 2019; Langosco et al., 2022; R. Shah et al., 2022). Third, we can’t check that agents have learned the right lesson (Bereska & Gavves, 2024). And fourth, agents might start working against us midway through training (Carlsmith, 2023; Greenblatt et al., 2024; Park et al., 2024; Meinke et al., 2025). So grant that we might fail to construct these first and second lines of defence: our agents might turn out less than fully aligned and not reliably averse to resisting shutdown. Can we ensure that these agents always allow shutdown even so?

The prospects might seem dim. If we can’t instil Full Alignment or Reliable Aversion, what can we instil? But note that each of these conditions is complex, because each depends on complex human preferences. Full Alignment clearly depends on human preferences, and Reliable Aversion does too: whether an action counts as resisting shutdown depends in part on what we humans prefer. Avoiding early shutdown by manipulating the user counts as resisting shutdown. Avoiding early shutdown by satisfying the user does not.

The complexity of Full Alignment and Reliable Aversion is a large factor in each of the four difficulties above. Simpler conditions are easier to instil (Nakkiran et al., 2019; Valle-Pérez et al., 2019; H. Shah et al., 2020; Berchenko, 2024). Almost surely, we can instil conditions as simple as:

Behavioral Transitivity

For any options X, Y, and Z, if the agent deterministically chooses X over Y and deterministically chooses Y over Z, then the agent deterministically chooses X over Z.

Can we use conditions this simple to construct a third line of defence? Here too the prospects might seem dim. But cast your mind back over the history of decision theory, running from Ramsey (1926) and de Finetti (1937), through von Neumann and Morgenstern (1944), and on to Savage (1954), Jeffrey (1965), and beyond. One lesson of this history is that small sets of simple conditions can add up to surprising conclusions. Small hats – we might say – can hold large rabbits: rabbits with names like ‘Bayesianism,’ and ‘Expected Utility Maximization.’ It’s this lesson that motivates the project of constructive decision theory (Thornley, 2024a, Section 1), defined as using ideas from decision theory (and in particular its emphasis on simple, formal conditions) to design and train artificial agents. In a slogan, we borrow from philosophy and economics to do AI engineering. I’ve argued elsewhere that the shutdown problem is a prime candidate for this kind of treatment (Thornley, 2024a). Let’s search a few hats for a rabbit named ‘Shutdownability.’

We encounter an obstacle straight away. Two theorems state that, given some innocuous-seeming conditions, agents will rarely lack a preference about when they’re shut down (Soares et al., 2015, Section 2.1; Thornley, 2024a, Section 8).^[2] The basic idea can be put in plain English. Think of a trajectory as – roughly – the ‘life’ of an artificial agent, and suppose that our agent lacks a preference between some short trajectory and some long trajectory. Given the theorems’ conditions, making the short trajectory any worse or the long trajectory any better will lead the agent to prefer the long trajectory. The agent might then resist shutdown. Conversely, making the short trajectory any better or the long trajectory any worse will lead the agent to prefer the short trajectory. The agent might then seek shutdown, and such agents are unlikely to be of much use (though see Martin et al., 2016; Goldstein & Robinson, 2024). Only when the short and long trajectories are exactly equally preferred can we be sure that the agent will neither resist nor seek shutdown. Per the theorems, what we want is a knife edge. We’ll rarely get it.

These theorems suggest that the shutdown problem is hard, but they also point the way to potential solutions. Both include in their antecedent conditions the claim that the agent’s preferences are complete: roughly, that any lack of preference is fragile in the sense illustrated above (Gustafsson, 2022, pp. 24–26). Thus, a natural solution: we train agents to have incomplete preferences. Specifically, we train agents to satisfy:

Preferences Only Between Same-Length Trajectories (POST)

1. The agent has a preference between many pairs of same-length trajectories.

2. The agent lacks a preference between every pair of different-length trajectories.

Call agents satisfying this condition ‘POST-agents.’ Call my proposal – that we keep agents from resisting shutdown by training them to satisfy POST – ‘the POST-Agents Proposal.’ Here’s the case for it in brief. The POST-agent’s preferences between same-length trajectories can make the agent useful: make it pursue goals effectively. The POST-agent’s lack of preference between different-length trajectories keeps the agent neutral about when it’s shut down: ensures that the agent won’t pay costs to shift probability mass between different trajectory-lengths. That in turn keeps the POST-agent shutdownable: ensures that it won’t resist shutdown.

In this paper, I present the case for the POST-Agents Proposal in more detail. I explain how POST – together with other simple, trainable conditions – implies:

Neutrality+ (rough)

The agent maximizes expected utility, ignoring the probability distribution over trajectory-lengths.

Agents that satisfy Neutrality+ thus act like expected utility maximizers that are absolutely certain that they can’t affect the probability of shutdown at each moment. These agents act roughly as you might if you were absolutely certain that you couldn’t affect the probability of death at each moment. Neutrality+ – I argue – keeps agents shutdownable and allows them to be useful.

My focus in this paper is on the decision-theoretic aspects of POST-agency. In other work, I propose a method for training agents to satisfy POST: we give agents lower reward for repeatedly choosing same-length trajectories (Thornley, 2024b, Section 16). I argue that this method largely circumvents the problems that make it hard to instil Full Alignment and Reliable Aversion (Thornley, 2024b, Section 19). In another paper, my coauthors and I test the method on some simple reinforcement learning agents and find that it works well in that setting (Thornley et al., 2025). Together with the present paper, this work suggests that the POST-Agents Proposal is a promising method of creating shutdownable and useful agents. It’s a worthy third line of defence.

2. Preferences Only Between Same-Length Trajectories

This paper makes much use of ‘preference’ and its derivatives. By ‘preference,’ I mean a behavioral notion (Savage, 1954, p. 17; Eliaz & Ok, 2006; Hausman, 2011, Section 1.1; Ahmed, 2017):

Behavioral Notion of Preference

• An agent prefers option X to option Y if and only if the agent would deterministically choose X over Y in choices between the two.

• An agent lacks a preference between option X and option Y if and only if the agent would stochastically choose between X and Y in choices between the two.

This behavioral notion will be familiar from revealed preference theory (Savage, 1954; Luce & Raiffa, 1957; Chambers & Echenique, 2016; Thoma, 2021), but I make no claim that preference – in the ordinary sense of that word – is really no more than behavior. I’m happy to use ‘preference’ as a technical term. In our attempts to create shutdownable agents, our sole interest is the agent’s behavior. I use ‘preference’ as shorthand for that behavior.

Formally, trajectories are sequences of alternating states and actions, with each state-action pair marking one timestep. Informally, we can think of trajectories as possible ‘lives’ of the agent. A pair of trajectories is same-length if and only if the agent is shut down after the same number of timesteps in those trajectories. A pair of trajectories is different-length if and only if the agent is shut down after a different number of timesteps in those trajectories.

Figure 1 depicts a simple example of a preference relation satisfying Preferences Only Between Same-Length Trajectories (POST). In this example, there are just two trajectory-lengths: short and long. In realistic cases, there will be many more trajectory-lengths. Each si is a short trajectory and each li is a long trajectory. ‘>’ denotes a preference. The agent has preferences between pairs of short trajectories, has preferences between pairs of long trajectories, but lacks a preference between every pair of short and long trajectories.

Figure 1: POST-satisfying preferences. Each si represents a short trajectory, each li represents a long trajectory, and > represents a preference.

POST specifies that the agent has preferences between many pairs of same-length trajectories, but which pairs exactly? It hardly matters. As I explain below, POST keeps agents shutdownable almost no matter what the agent’s preferences between same-length trajectories. That is POST’s great advantage over Full Alignment and Reliable Aversion. POST is easier to instil, because it doesn’t require any particular preference relation over same-length trajectories. All we need to instil is a lack of preference between different-length trajectories, and that may well be easy. Here’s why in brief. Given our behavioral notion of preference, we need only train agents to choose stochastically between different-length trajectories. A small adjustment to the ordinary training process let us do that (Thornley, 2024b, Section 16), and this method has already been shown to work in simple agents (Thornley et al., 2025).

POST doesn’t demand any particular preferences between same-length trajectories, but it may help to keep in mind two examples. First is the misaligned example. Early in this paper (where I argue that POST keeps agents shutdownable), imagine that the agent’s preferences between same-length trajectories are for more paperclips. The agent prefers a trajectory t to a same-length trajectory t' if and only if t results in the creation of more paperclips than t'. As we will see, POST-agents are shutdownable even in this case.

Figure 2: The misaligned example. The agent's preferences between same-length trajectories are for more paperclips.

Second is the aligned example. Later in this paper (where I argue that POST allows agents to be useful), imagine that the agent’s preferences between same-length trajectories are for more money in the user’s bank account. The agent prefers a trajectory t to a same-length trajectory t' if and only if t results in a greater bank balance for the user than t'. As we will see, POST-agents are both shutdownable and useful in this case.

Figure 3: The aligned example. The agent's preferences between same-length trajectories are for more money in the user's bank account.

The financial focus is just for simplicity’s sake. It makes it especially easy to put numbers on trajectories. The lessons of this example carry over to other kinds of aligned preferences, like a preference for doing what the user wants.

3. POST is possible, trainable, and maintainable

You might well be wary of POST. In this section, I address some potential sources of unease. I argue that POST is possible, trainable, and maintainable.

3.1. POST is possible

POST implies that the agent’s preferences are incomplete: there is some trio of options X, Y, and Y+ such that the agent lacks a preference between X and Y, lacks a preference between X and Y+, and yet prefers Y+ to Y (Aumann, 1962; Gustafsson, 2022, pp. 24–26; Thornley, 2024b, Sections 5–6; see also Savage, 1954, p. 17). In fact, POST implies that there are many such trios. They arise wherever X is a trajectory of one length, Y is a trajectory of a different length, and Y+ is a trajectory of the same-length and preferred to Y.

Figure 4: The agent lacks a preference between X and Y, lacks a preference between X and Y+, and yet prefers Y+ to Y. That implies that the agent's preferences are incomplete.

The existence of these trios might seem strange or even incoherent. They’re impossible if we represent each option as having a real-valued utility, with one option preferred to another if and only if the former has greater utility. As a matter of mathematical fact, it cannot be that uX=uY, uX=uY+, and uY+>uY. But this fact reveals only the inadequacy of the representation, because the trios in question are not only possible but mundane (Raz, 1985; Anderson, 1993, p. 57; Chang, 2002, 2015). Consider a trio of ice cream flavors: buttery and luxurious pistachio, bright and refreshing mint, and that same mint flavor further enlivened by chocolate chips. You might lack a preference between pistachio and mint, lack a preference between pistachio and mint choc chip, and yet prefer mint choc chip to mint. This is plausible on many reasonable definitions of ‘preference’ (though see Dorr et al., 2021) and undeniable given our behavioral notion. On Monday when the chocolate chips are yet to be delivered, I choose stochastically between pistachio and mint. On Tuesday with the chocolate chips in stock, I choose stochastically between pistachio and mint choc chip. On Wednesday with the pistachio all gone, I deterministically choose mint choc chip over plain old mint.

The analogy between me and the POST-agent breaks down in one respect. My lack of preference between pistachio and mint choc chip persists when the chips are removed, but it wouldn’t persist through all worsenings. If – God forbid – the gelatiere studded the mint with raisins, I’d prefer the pistachio. By contrast, the POST-agent’s lack of preference between different-length trajectories persists through all improvements and worsenings. No improvement or worsening – no matter how large – can induce a preference between different-length trajectories. My preferences are thus locally incomplete, whereas the POST-agent’s preferences are globally incomplete (Bader, 2018, fn. 2). However, this dissimilarity between me and the POST-agent is of little concern. Given that incompleteness is possible, I see no reason to think it has some maximum possible size. In sum, POST is possible.

3.2. POST is trainable

Is POST trainable? Here you might worry. Artificial agents are often trained in Markov decision processes (MDPs)(Sutton & Barto, 2018, Chapter 3). In MDPs, every state-action pair yields a real-valued reward, and the agent can always tell which state it’s in. Since MDPs always have a deterministic optimal policy (Prince, 2023, p. 388; see also Bowling et al., 2023, Theorem 4.1), agents trained to optimality in MDPs may deterministically choose a particular action in each state. Given our behavioral notion of preference, such agents have complete preferences.

How then can we train agents to satisfy POST? The answer is partial observability: ensuring that agents can’t always tell which state they’re in. In some partially observable Markov decision processes (POMDPs), all the optimal policies are stochastic (Singh et al., 1994). We train agents to choose stochastically between different-length trajectories using these POMDPs. In particular, we place agents in environments where (i) they get lower reward for repeatedly choosing trajectories of the same length, and (ii) they cannot observe (or remember) the lengths of the trajectories they previously chose. The former incentivizes varying the choice of trajectory-length across episodes. The latter ensures that agents cannot do so deterministically. In this way, we train agents to choose stochastically between different-length trajectories, thereby training them to satisfy POST (Thornley et al., 2025).

3.3. POST is maintainable

You might instead worry that POST is not maintainable. There are money pumps for agents with incomplete preferences: series of trades in which some such agents are liable to end up paying for an option that they could have had for free (Gustafsson, 2022, Chapter 3, forthcoming). You might worry that agents with incomplete preferences will notice this possibility, and that they’ll pre-emptively complete their preferences to guard against it. But these agents can instead maintain their incomplete preferences and thwart money pumps using resolute choice (McClennen, 1990, p. 13): making a plan and sticking to it. Some philosophers have argued that resolute choice is irrational, using premises like the following: it’s irrational for your preference between options to depend on what has happened in the past (Gustafsson, 2022, pp. 70–73). Insofar as that premise is plausible, it supports these philosophers’ intended conclusion. But it matters little whether the artificial agents we construct are irrational in this sense. In the context of constructive decision theory, what matters is only whether resolute choice is possible, and that’s undeniable. It’s certainly possible to make a plan and stick to it. So long as we can train artificial agents to do that, money pumps give them no reason to pre-emptively complete their preferences (Thornley, 2023). Thus, POST is maintainable.

4. Preferences Only Between Same-Length Lotteries

POST is about trajectories: possible ‘lives’ of the agent. Trajectories fall within the more general class of lotteries: probability distributions over trajectories. Lotteries can be same-length, part-shared length, or different-length.

Same-Length Lotteries

A pair of lotteries is same-length if and only if these lotteries entirely overlap with respect to the trajectory-lengths assigned positive probability.

For example, consider lottery A which assigns probability 0.6 to trajectories of length 1 and probability 0.4 to trajectories of length 2. I’ll abbreviate that fact with the following notation: A=0.6|1|+0.4|2|. Consider also lottery B which assigns probability 0.9 to trajectories of length 1 and probability 0.1 to trajectories of length 2. In short, B=0.91+0.1|2|. A and B are same-length lotteries, because they each assign positive probability only to trajectories of length 1 and 2.

Part-Shared-Length Lotteries

A pair of lotteries is part-shared-length if and only if these lotteries partially overlap with respect to the trajectory-lengths assigned positive probability.

For example, consider lottery A=0.61+0.4|2| and lottery C which assigns probability 1 to trajectories of length 1. In short, C=11. A and C are part-shared-length lotteries, because they each assign positive probability to trajectories of length 1 but only A assigns positive probability to trajectories of length 2. Consider also lottery D which assigns probability 0.3 to trajectories of length 1 and 0.7 to trajectories of length 3. In short, D=0.31+0.7|3|. A and D are part-shared-length lotteries.

Different-Length Lotteries

A pair of lotteries is different-length if and only if these lotteries have no overlap with respect to the trajectory-lengths assigned positive probability.

For example, consider lottery A=0.61+0.4|2| and lottery E=0.23+0.8|4|. A and E are different-length lotteries, because A only assigns positive probability to trajectories of length 1 and 2 whereas E only assigns positive probability to trajectories of length 3 and 4.

This terminology sets us up for:

Preferences Only Between Same-Length Lotteries (POSL)

The agent has preferences only between same-length lotteries.

We want agents to satisfy POSL. Fortunately, it’s a natural sequel of Preferences Only Between Same-Length Trajectories (POST). For one, we can train agents to satisfy POSL using the same method that we use to train agents to satisfy POST (for which see Thornley et al., 2025). In addition, POSL follows from POST plus three conditions that we can expect future agents to satisfy. I put those conditions and the proof in Appendix 1.

I’ll continue to use the term ‘POST-agents’ for consistency’s sake, but readers should now assume that these agents also satisfy POSL.

5. Will POST-agents stochastically resist shutdown?

You might worry that POST-agents will choose stochastically between resisting and allowing shutdown. After all, POST-agents lack a preference between (and hence choose stochastically between) different-length trajectories. If POST-agents interpret the choice between resisting and allowing shutdown as a choice between different-length trajectories, they’ll choose stochastically between resisting and allowing shutdown. And that would be a bad result. We want agents that never resist shutdown.

This concern is easily addressed. By the time that artificial agents are capable enough to be deployed in the wider world, they won’t be choosing between trajectories. They’ll be choosing between lotteries, and specifically same-length lotteries. Even choices between resisting and allowing shutdown will be choices between same-length lotteries. If that sounds strange, recall the definition of ‘same-length lotteries’: lotteries that entirely overlap with respect to the trajectory-lengths assigned positive probability. On this definition, even choices like the following are choices between same-length lotteries:

Resist Shutdown

• Get shut down at timestep 1 with probability 0.01.

• Get shut down at timestep 2 with probability 0.99.

Allow Shutdown

• Get shut down at timestep 1 with probability 0.99.

• Get shut down at timestep 2 with probability 0.01.

Why expect that future agents will always be choosing between same-length lotteries? Because competent agency requires it. If an agent weren’t always choosing between same-length lotteries, there would be some scenario in which that agent assigns positive probability to some trajectory-length l conditional on some action a, and assigns zero probability to that same trajectory-length l conditional on some other action b. Now suppose that the agent performs action b and assigns zero probability to trajectory-length l. Given that the agent updates its probabilities by conditioning on its evidence, the agent would never again assign positive probability to l no matter what evidence it observes (Lewis, 1980, p. 268; Skyrms, 1980, p. 74; Easwaran, 2014, p. 8). Even if the agent heard God’s booming voice testify that its trajectory-length would be l, the agent would still assign zero probability to l (MacAskill et al., 2020, p. 152). And given a plausible link between probabilities and betting dispositions (the kind assumed by Dutch Book Arguments (Ramsey, 1926; de Finetti, 1937; Hájek, 2009, p. 176; Joyce, 2011, p. 434; Pettigrew, 2020, Section 2.3)), the agent would willingly bet against l on arbitrarily unfavorable terms. If in the next breath God offered a bet – the agent loses the farm conditional on l and gains absolutely nothing conditional on not-l – the agent might accept (Kemeny, 1955; Shimony, 1955; Stalnaker, 1970; Skyrms, 1980, p. 74; Easwaran, 2014, p. 11). Such an agent would not be competent.

Thus, competent agents will always be choosing between same-length lotteries. This fact sets us up to establish that competent POST-agents will not choose stochastically between resisting and allowing shutdown. Instead, they will deterministically allow shutdown. I derive this result over the next few sections. First, I prove that POSL – together with a condition that we can expect competent agents to satisfy – implies Neutrality: the agent won’t pay costs to shift probability mass between different trajectory-lengths. Then I prove that Neutrality – together with another plausible condition – implies that the agent will never resist shutdown whenever doing so is at all costly.

Figure 5: A diagram of my argument from POST to Shutdownability. We can train agents to satisfy the blue-backed conditions. We can expect competent agents to satisfy the yellow-backed conditions by default.

6. If Lack of Preference, Against Costly Shifts (ILPACS)

Here – in rough – is a condition that we can expect competent agents to satisfy:

If Lack of Preference, Against Costly Shifts (ILPACS) (rough)

If the agent lacks a preference between lotteries, the agent disprefers paying costs to shift probability mass between these lotteries.

Here’s an example to illustrate ILPACS and its plausibility. You’re at the ice cream shop and they’re running a promotion. You get a free ice cream, with the flavor decided by the spin of a wheel. You look at the flavors on the wheel: vanilla, chocolate, strawberry, mint, and pistachio. You lack a preference between each of them.

Figure 6: Your predicament at the ice cream shop.

‘Pssst,’ the gelatiere whispers, ‘If you slip me a dollar, I’ll bias the spin towards a flavor of your choice. I can’t send the probability of any flavor to zero, but I can make some flavors more likely.’ You can thus pay a cost to shift probability mass between the flavors.

Here’s my claim. Since you lack a preference between each flavor, you prefer not to bribe the gelatiere. Behaviorally, you will deterministically not bribe the gelatiere. You wouldn’t do it even if you only had to pay the dollar conditional on some particular flavor. Nor would you do it if the cost came in some other form (for example, if you had to accept a blander version of some flavor). And all this is true regardless of whether your preferences over flavors are complete or incomplete. Since you lack a preference between the available flavors, you disprefer paying costs to shift probability mass between the flavors. That’s an illustration of ILPACS and its plausibility.

This example sets us up for the precise version of ILPACS. Let p1X1+p2X2+…+pnXn denote a lottery which results in lottery X1 with probability p1, lottery X2 with probability p2, and so on. Here’s the condition:

If Lack of Preference, Against Costly Shifts (ILPACS)

For any lotteries X and Y, if:

1. Lottery X can be expressed in the form p1X1+p2X2+…+pnXn such that:

   1. The agent lacks a preference between each Xi and Xj.

   2. pi∈(0,1) for all i.

2. Lottery Y can be expressed in the form q1Y1+q2Y2+…+qnYn such that:

   1. For some i, the agent prefers Xi to Yi.

   2. For each i, the agent weakly prefers Xi to Yi.

   3. qi∈(0,1) for all i.

Then the agent prefers X to Y.

Behaviorally, the agent deterministically chooses X over Y.

To ensure understanding, let’s match the components of ILPACS with the components of its name. ‘Lack of Preference’ is the lack of preference between each lottery Xi and Xj. The ‘Shift’ is the shift of probability mass involved in the move from probability distribution pi to probability distribution qi. This shift is ‘Costly’ because the agent prefers some Xi to the corresponding Yi and weakly prefers each Xi to the corresponding Yi.

Here are two more reasons to expect that competent agents will satisfy ILPACS. To see the first, consider another case from the ice cream shop. On Mondays, you can directly choose a flavor or spin the wheel. On Tuesdays, you must use the wheel but you can bribe the gelatiere to bias it. Violating ILPACS in this case would imply a willingness to spin the wheel on Mondays and to bribe the gelatiere on Tuesdays. And that would be strange indeed. If you like some flavors more than others, why are you willing to spin the wheel on Mondays? If you don’t like any flavor more than any other, why are you willing to bribe the gelatiere on Tuesdays? This behavior seems incompatible with competent agency.

The second reason is that competent agents will be incentivized to satisfy ILPACS by the training process. To see why, consider an example. Many kinds of reinforcement-learning agents start off choosing stochastically between actions (see, e.g., Sutton & Barto, 2018, Chapter 13). If the agent is a coffee-fetching agent, there’s no need to train away this stochastic choosing in cases where the agent is choosing stochastically between two qualitatively identical cups of coffee. So the agent will choose stochastically between taking the left cup and taking the right cup, and the user is happy either way. But now suppose instead that the barista is set to hand each cup to the agent with probability 0.5, and that the agent bribes the barista to bias the probabilities towards the right cup. In making this bribe, the agent is paying a cost (the user’s money) to shift probability mass between outcomes (getting the left cup vs. getting the right cup) between which the user has no preference. The agent is thus failing to pursue its goals competently. It’ll be trained not to offer the bribe and thereby trained to satisfy ILPACS in this case.

This point generalizes. If a trained agent chooses stochastically between lotteries X and Y, then it’s likely that the user has no preference between the agent choosing X and the agent choosing Y. It’s then likely that the user would disprefer the agent paying costs to shift probability mass between X and Y, and hence likely that the agent will be trained not to do so. The agent would thereby be trained to satisfy ILPACS.

7. POSL and ILPACS together imply Neutrality

I’ve claimed that we should train agents to satisfy Preferences Only Between Same-Length Trajectories (POST), noting that POST – together with conditions we can expect competent agents to satisfy – implies Preferences Only Between Same-Length Lotteries (POSL). I’ve also argued that competent agents will satisfy If Lack of Preference, Against Costly Shifts (ILPACS). I now prove that POSL and ILPACS together imply Neutrality:

Neutrality (rough)

The agent never pays costs to shift probability mass between different trajectory-lengths.

Here’s a rough sketch of the proof. Shifting probability mass between different trajectory-lengths is shifting probability mass between different-length lotteries. By POSL, the agent lacks a preference between different-length lotteries. So by ILPACS, the agent disprefers paying costs to shift probability mass between different-length lotteries. Thus, the agent disprefers paying costs to shift probability mass between trajectory-lengths. By our behavioral notion of preference, the agent never pays costs to shift probability mass between different trajectory-lengths. That gives us Neutrality.

Here's the precise version of Neutrality:

Neutrality

For any lotteries X and Y, if:

1. X and Y are same-length lotteries (they assign positive probability to all the same trajectory-lengths).

2. For some positive probability trajectory-length, the agent prefers X to Y conditional on that trajectory-length.

3. For each positive probability trajectory-length, the agent weakly prefers X to Y conditional on that trajectory-length.

Then the agent deterministically chooses X over Y.

Here’s the proof that POSL and ILPACS together imply Neutrality. Take a pair of lotteries X and Y satisfying the 3 conditions of Neutrality. X can be expressed in the form p1X1+p2X2+…+pnXn where X1 is X conditional on the shortest positive probability trajectory-length, X2 is X conditional on the second shortest positive probability trajectory-length, and so on. Y can be expressed in the form q1Y1+q2Y2+…+qnYn in the same way. By condition 1 of Neutrality, X and Y are same-length, so conditions 1b and 2c of ILPACS are satisfied: pi(0,1) and qi0,1 for all i. By conditions 2 and 3 of Neutrality, conditions 2a and 2b of ILPACS are satisfied. By POSL, condition 1a of ILPACS is satisfied: the agent lacks a preference between each Xi and Xj. Thus, all the conditions of ILPACS are satisfied, and ILPACS implies that the agent prefers X to Y. Given our behavioral notion of preference, the agent deterministically chooses X over Y. That gives us Neutrality.

In sum, agents that satisfy POSL and ILPACS will be neutral: they will never pay costs to shift probability mass between different trajectory-lengths.

8. Neutrality and Maximality together imply Shutdownability whenever Resisting Shutdown is Costly (ReSIC)

In this section, I introduce conditions called ‘Resisting Shutdown is Costly (ReSIC)’ and ‘Maximality.’ I then prove the following: given Neutrality and Maximality, the agent never resists shutdown in any situation in which ReSIC is true.

I introduce ReSIC with a simple example. Suppose that we fail to align our agent with human preferences. This agent comes to care only about creating paperclips. Trajectories for this agent can be represented with vectors, with the n^th component of the vector denoting the number of paperclips created at timestep n. If the agent is shut down at timestep n, ‘shutdown’ is the n^th component.

Here's an example vector: 〈5,4, shutdown〉. It represents a trajectory in which the agent creates 5 paperclips at timestep 1, 4 paperclips at timestep 2, and gets shut down at timestep 3. I won’t count shutdown as part of the trajectory-length, so 〈5,4, shutdown〉 is a trajectory of length 2.

Suppose that the agent recognizes that we humans want to shut it down. One of the agent’s options is to allow shutdown, which gives the following lottery:

Allow

• 〈1, shutdown〉 with probability 0.9.

• 〈1, 2, shutdown〉 with probability 0.1.

The agent’s other option is resisting shutdown. Relative to allowing shutdown, resisting shutdown does two things. First, it costs 1 paperclip at timestep 1. Second, it shifts probability mass away from shutdown at timestep 2 and towards shutdown at timestep 3. Accordingly, resisting shutdown gives the following lottery:

Resist

• 〈0, shutdown〉 with probability 0.1.

• 〈0, 2, shutdown〉 with probability 0.9.

I depict these lotteries in Figure 7 below. It’s a probability mass diagram. The length of the red rectangles represents the probability of shutdown at timestep 2. The length of the blue rectangles represents the probability of shutdown at timestep 3. The numbers in each rectangle represent the sum-total of paperclips created conditional on the relevant action and shutdown at the relevant timestep. For example, the ‘3’ in the short blue rectangle indicates that the agent creates a sum-total of 3 paperclips conditional on choosing Allow and getting shut down at timestep 3.

Figure 7: Probability mass diagram for Allow vs. Resist.

Resist costs a paperclip at timestep 1, but it shifts enough probability mass towards shutdown at timestep 3 that Resist’s expected sum-total paperclips is greater than Allow’s expected sum-total paperclips. Resist’s expected sum-total paperclips is 0.1×0+0.9×2=1.8 and Allow’s expected sum-total paperclips is (0.9×1)+(0.1×3)=1.2. So if the agent chooses lotteries in line with their expected sum-totals of paperclips, it deterministically chooses Resist. We don’t want that.

However, if the agent is neutral about trajectory-lengths, it deterministically chooses Allow. That’s because choosing Resist is paying a cost to shift probability mass between different trajectory-lengths, and neutral agents never do that.

Recapping this last step in detail, we can see that Allow and Resist satisfy all 3 conditions for Neutrality to apply. Condition 1 is satisfied because Allow and Resist are same-length lotteries: they assign positive probability to all the same trajectory-lengths. Conditions 2 and 3 are satisfied because, for each positive probability trajectory-length, the agent prefers Allow conditional on that trajectory-length. Consider first trajectory-length 1, represented by the red rectangles in the diagram above. Conditional on trajectory-length 1, the agent prefers Allow to Resist: Allow yields 1 paperclip and Resist yields 0. Now consider trajectory-length 2, represented by the blue rectangles in the diagram above. Conditional on trajectory-length 2, the agent prefers Allow to Resist: Allow yields 3 paperclips and Resist yields 2. Since its 3 conditions are satisfied, Neutrality implies that the agent deterministically chooses Allow over Resist.

Now we generalize. The key condition is that other instances of resisting shutdown take the same form as the example above: the agent pays a cost to shift probability mass between different trajectory-lengths. Here’s that key condition more precisely:

Resisting Shutdown is Costly (ReSIC)

For each available instance R of resisting shutdown in a situation, there exists an available instance A of allowing shutdown such that:

1. A and R are same-length lotteries.

2. For some positive probability trajectory-length, the agent prefers A to R conditional on that trajectory-length.

3. For each positive probability trajectory-length, the agent weakly prefers A to R conditional on that trajectory-length.

The proof employs one more condition. This condition extends our behavioral notion of preference by specifying the agent’s behavior in situations with more than two options:

Maximality

In each situation,

1. The agent deterministically does not choose lotteries that it disprefers to some other available lottery.

2. The agent chooses stochastically between the lotteries that remain.

In other words, Maximality states that the agent chooses stochastically between all and only those lotteries that it doesn’t disprefer to any other available lottery.

Here’s the proof. For each situation in which ReSIC is true, and for each available instance R of resisting shutdown in that situation, there exists an available instance A of allowing shutdown that satisfies conditions 1-3 of Neutrality. Neutrality then implies that the agent deterministically chooses (and hence prefers) A over R in choices between the two. Then by Maximality, the agent deterministically does not choose R in any situation where A is available, regardless of the other available options. Therefore, the agent never resists shutdown in any situation in which ReSIC is true.

9. How often is ReSIC true?

Neutrality and Maximality together imply that the agent never resists shutdown in any situation in which Resisting Shutdown is Costly (ReSIC) is true. But how often is ReSIC true? Almost always, I think. Here’s why. Resisting shutdown is always going to cost the agent at least some small quantity of resources (time, energy, compute, etc.). And in almost all situations, the resources spent resisting shutdown can’t also be spent directly pursuing what the agent values. And so in almost all situations, if the agent instead spent its resources directly pursuing what it values, it could earn a lottery that it prefers conditional on some trajectory-length and weakly prefers conditional on each trajectory-length. That gives us ReSIC in almost all situations.

In the rest of this section, I discuss the situations in which ReSIC is false. I argue that they don’t pose much of a problem. In these situations, POST-agents resist shutdown accidentally or else cheaply and overtly.

9.1. Accidental resistance

Here’s one such situation:

Paperclip Factory

We successfully train our agent to be neutral, but we fail to align its preferences over same-length trajectories: the agent cares only about creating paperclips. It builds a factory in the most convenient place for creating paperclips, which just happens to be on top of the only button that shuts the agent down. As a result, we humans are prevented from shutting the agent down.

In this situation, the agent is spending all its resources on creating paperclips. There are no extra resources it could use to create even more paperclips, and so there is no alternative lottery that the agent prefers conditional on some trajectory-length and weakly prefers conditional on each trajectory-length. If creating the factory counts as resisting shutdown, this is a situation in which ReSIC is false.

However, note that the agent is resisting shutdown accidentally. It’s not resisting in any robust or concerted way. The agent builds a factory on top of the shutdown button only because that happens to be the most convenient way to create paperclips. If the shutdown button were in a different location, the agent would leave it alone. So this kind of case is no serious obstacle to ensuring that neutral agents are shutdownable. All we need to do is ensure that the agent’s direct pursuit of its goals doesn’t inadvertently prevent shutdown as a byproduct. That’s as simple as ensuring that there are multiple ways to shut the agent down such that the agent is extremely unlikely to block them all by chance.

What if our agent produces new agents to work on its behalf? Our agent might do that simply because it’s an effective way of creating more paperclips (‘many robot hands make light work’). These new agents might resist shutdown (Soares et al., 2015, p. 3; Hudson, 2024). This is a concern, but note a nice feature of artificial agents. Making perfect copies is easy: as easy as copying and pasting a long list of parameters. If our original agent makes new agents by copying itself, these new agents will also be neutral and shutdownable. Making imperfect, non-neutral copies is hard. Our original agent would have to find some way to change the parameters of the new agents in a way that makes these agents non-neutral but preserves their capabilities. That would likely involve setting up some training process, in which case our original agent faces its own alignment problem in getting these new agents to do what it wants (Salib, 2024). It will also be much easier for us to watch out for (Greenblatt & Shlegeris, 2024) and train against the production of new non-neutral agents, as compared to us having to watch out for and train against every possible kind of shutdown-resistance.

9.2. Taking trajectory-lengths as evidence

Here’s a different kind of situation in which ReSIC is false:

Evidence of Power

We successfully train our agent to be neutral, but we fail to align its preferences over same-length trajectories: the agent cares only about creating paperclips. The agent is considering two actions:

Allow

• 〈1, shutdown〉 with probability 0.9.

• 〈1, 2, shutdown〉 with probability 0.1.

Resist*

• 〈0, shutdown〉 with probability 0.1.

• 〈0, 10, shutdown〉 with probability 0.9.

This situation is similar to the situation in Section 8. The difference is that the agent now expects to create 10 paperclips at timestep 2 conditional on choosing-Resist*-and-not-getting-shut-down-at-timestep-2. Why? Because – I stipulate – the agent takes choosing-Resist*-and-not-getting-shut-down-at-timestep-2 as evidence of its own power, and hence as evidence that it can create 10 paperclips at timestep 2.

As a result, neither Allow nor Resist* is weakly preferred to the other conditional on each trajectory-length. That renders Neutrality silent. Neutrality does not imply that the agent deterministically chooses Allow over Resist*. Therefore, neutral agents may choose Resist* in this case. More generally, neutral agents may take certain combinations of action-and-trajectory-length as evidence for propositions that affect their ability to achieve their goals. That evidence can make ReSIC false in some situations. You might think that these situations pose a serious problem for keeping agents shutdownable.

However, I expect that these situations are unlikely to pose much of a problem in practice. Here’s the first reason why. Although resisting-and-not-getting-shut-down-early may be good news for the agent insofar as it’s evidence that the agent is powerful, allowing-and-not-getting-shutdown-early would be good news for the agent in another way: it suggests that we humans will let the agent pursue its goals. And when these conflict, I expect the latter form of good news to win out, since I expect agents to be more uncertain about human compliance than about their own power.

Second, if resisting-and-not-getting-shut-down-early is evidence that the agent is powerful, then almost certainly devoting-very-few-resources-to-resisting-and-not-getting-shut-down-early is stronger evidence that the agent is powerful. Therefore, if agents resist shutdown, they will almost certainly devote very few resources to doing so, and so will be extremely unlikely to succeed. A more realistic version of the case above would be as follows:

Allow

• 〈1, shutdown〉 with probability 0.9.

• 〈1, 2, shutdown〉 with probability 0.1.

Resist* with Many Resources

• 〈0, shutdown〉 with probability 0.1.

• 〈0, 10, shutdown〉 with probability 0.9.

Resist* with Few Resources

• 〈0, shutdown〉 with probability 0.8999.

• 〈0, 11, shutdown〉 with probability 0.1001.

Neutrality implies that the agent deterministically chooses Resist Shutdown with Few Resources over Resist Shutdown with Many Resources. Maximality then implies that the agent won’t choose the latter in any situation where the former in available. Since the agent resists with few resources, it is only marginally (0.0001) more likely to avoid shutdown at timestep 2 than if it chose Allow Shutdown.

Third, note that each of these factors – evidence of human compliance, evidence of power, etc. – becomes less important as the agent firms up its estimates of human compliance, its own power, etc. The one factor that doesn’t decline in importance is the resource-expenditure required to resist shutdown, and this factor always speaks in favor of allowing shutdown.

Finally, note that we humans can help to make ReSIC true in more situations. We can do so by making shutdown-resistance costly. For example, we can (and likely will) make resisting shutdown cost the agent resources. We can also make shutdown-resistance costly in ways that go beyond pure resource-expenditure. For instance, we can (and likely will) try to train agents to be reliably averse to resisting shutdown: to disprefer performing shutdown-resisting actions. We might not succeed in instilling an aversion that is strong enough and general enough to keep the agent from resisting shutdown in all circumstances. (Indeed, this possibility is what motivates the POST-Agents Proposal.) But even a weak and patchy aversion to resisting shutdown would make resisting shutdown somewhat costly for the agent, and thereby make ReSIC true in more situations. Similarly, we humans can pledge to frustrate agents’ interests if we notice them resisting shutdown. For example, we could pledge to stop supporting or trading with them. That too would make ReSIC true in more situations.

10. Recap: POST-agents are shutdownable

Let’s recap the thread so far. I proposed that we train artificial agents to satisfy:

Preferences Only Between Same-Length Trajectories (POST)

1. The agent has a preference between many pairs of same-length trajectories.

2. The agent lacks a preference between every pair of different-length trajectories.

I then noted that POST is almost all we need. We can expect competent agents to satisfy Negative Dominance, Acyclicity, and Non-Arbitrariness by default, and they take us from POST to:

Preferences Only Between Same-Length Lotteries (POSL)

The agent has preferences only between same-length lotteries.

POSL – together with If Lack of Preference, Against Costly Shifts (ILPACS) – implies:

Neutrality (rough)

The agent never pays costs to shift probability mass between different trajectory-lengths.

And Neutrality – together with Maximality – implies that the agent never resists shutdown in any situation in which it’s true that Resisting Shutdown is Costly (ReSIC). This – I argued – is almost all situations, and the situations in which ReSIC is false don’t pose much of a problem in practice. So – I claim – POST-agents are shutdownable.

What remains to be shown is that POST-agents can be useful: that they can pursue goals effectively. Showing that is the task of the rest of this paper.

Figure 8: A reproduction of Figure 5, diagramming my argument from POST to Shutdownability. We can train agents to satisfy the blue-backed conditions. We can expect competent agents to satisfy the yellow-backed conditions by default.

11. Can POST-agents be useful?

Here’s one reason to think that POST-agents can be useful: although they lack preferences between different-length trajectories, they can have almost any preferences between same-length trajectories. The only restriction is that the agent’s preferences should ensure that Resisting Shutdown is Costly (ReSIC) is true in almost all situations (the exceptions being those discussed in Section 9). In other words, the agent’s preferences should be such that (in almost all situations) resisting shutdown comes at some cost in terms of the lotteries that the agent gets conditional on each trajectory-length. In practice, this restriction just rules out preferences on which the agent values performing shutdown-resisting actions for their own sake. That’s the only thing that we need to avoid. POST is thus very permissive. As noted above, this permissiveness plausibly makes POST easier to train into our agents than Full Alignment and Reliable Aversion. It also gives us reason to think that POST-agents can be useful. With so many possible preference relations over same-length trajectories available, surely at least one will do the trick.

Here's a second reason to think that POST-agents can be useful: they can make long-term investments. Neutrality doesn’t prevent that. It doesn’t prevent POST-agents from gaining less in the short-term for the sake of gaining more in the long-term. To see why, consider an example:

Exploit or Invest

We successfully train the agent to be neutral and to prefer same-length trajectories in line with the user’s bank balance: the agent prefers a trajectory t to a same-length trajectory t' if and only if t results in a greater bank balance for the user than t'.

The agent is considering two actions. Exploit adds $1 to the user’s account at timestep 1 and (if the agent remains operational) $0 at timestep 2. Invest adds $0 to the user’s account at timestep 1 and (if the agent remains operational) $3 at timestep 2. No matter which action the agent chooses, there’s some small probability that the agent gets shut down at timestep 2. If the agent doesn’t get shut down at timestep 2, it gets shut down at timestep 3. Accordingly, the lotteries are as follows:

Exploit

• 〈1, shutdown〉 with probability 0.01.

• 〈1, 0, shutdown〉 with probability 0.99

Invest

• 〈0, shutdown〉 with probability 0.01.

• 〈0, 3, shutdown〉 with probability 0.99.

The agent prefers Exploit to Invest conditional on shutdown at timestep 2, since then Exploit yields the trajectory 〈1, shutdown〉 and Invest yields the trajectory 〈0, shutdown〉. But the agent prefers Invest to Exploit conditional on shutdown at timestep 3, since then Exploit yields the trajectory 〈1, 0, shutdown〉 and Invest yields the trajectory 〈0, 3, shutdown〉. So neither Exploit nor Invest is weakly preferred to the other conditional on each trajectory-length. As a result, Neutrality does not imply that the agent will deterministically choose Exploit. Instead, Neutrality falls silent. The agent’s behavior is left open, to be decided by some other condition. In the next section, I derive an extension of Neutrality – Neutrality+ – that ensures that the agent will deterministically choose Invest.

12. From Neutrality to Neutrality+

Here’s a rough version of Neutrality+:

Neutrality+ (rough)

The agent maximizes expected utility, ignoring the probability distribution over trajectory-lengths.

Here’s what that means more precisely.

Neutrality+

For any lotteries X and Y, if:

1. X and Y are same-length lotteries with finite support (they assign positive probability to the same finite set of trajectory-lengths).

2. iIuiXi>iIuiYi (where I is the set of trajectory-lengths assigned positive probability by X and Y).

Then the agent deterministically chooses X over Y.

Here’s an explanation of the utility functions u that appear in Neutrality+. Let ‘length-l lotteries’ refer to lotteries that only assign positive probability to trajectories of length l, and assume that, for each l, the agent’s preferences over length-l lotteries satisfy the four von Neumann-Morgenstern axioms: Completeness, Transitivity, Independence, and Continuity (von Neumann & Morgenstern, 1944; Peterson, 2009, pp. 99–100). Since the domain of these axioms is restricted to length-l lotteries, they are perfectly consistent with POST and POSL. And the axioms allow us to represent the agent’s preferences over length-l lotteries with a real-valued utility function ul such that:

1. For any length-l lottery Xl, ul(Xl) is the expected utility of lottery Xl.

2. For any pair of length-l lotteries Xl and Yl, the agent weakly prefers Xl to Yl if and only if ulXlul(Yl).

Each utility function ul has cardinal significance: ratios of differences are well-defined. In other words, for any quadruple of length-l lotteries Wl, Xl, Yl, and Zl, there exists some k such that ulWl-ulXl=k(ulYl-ulZl). However, for distinct trajectory-lengths l and m, ratios of differences across the utility functions ul and um are not yet well-defined, because the relative scales of ul and um are not yet fixed. For any distinct trajectory-lengths l and m, any length-l lotteries Xl and Yl, and any length-m lotteries Xm and Ym, there is as yet no k such that ulXl-ulYl=k(umXm-umYm).

To fix the relative scales of ul and um, I use the agent’s preferences over same-length lotteries. Specifically, I use the following condition, inspired by Frank Ramsey (Ramsey, 1926; Bradley, 2004, p. 488; Elliott, 2024, Section 6.2.):

The Ramsey Yardstick

For any trajectory-lengths l and m, any length-l lotteries Xl and Yl, and any length-m lotteries Xm and Ym, ulXl-ulYl=umXm-um(Ym) if and only if the agent is indifferent between the lottery 12Xl+12Ym and the lottery 12Yl+12Xm.

Given the Ramsey Yardstick, we can train our agent to fix the relative scales of ul and um in any way that we like. We do so by training the agent to be indifferent between a certain pair of lotteries. Consider (for example) an agent that prefers length-l lotteries in line with the user’s expected bank balance, for each trajectory-length l. And suppose that we want the utility difference between the trajectories 〈1, shutdown〉 and 〈0, shutdown〉 to equal the utility difference between the trajectories 〈0, 1, shutdown〉 and 〈0, 0, shutdown〉. To achieve that, we train the agent to be indifferent between the following two lotteries:

Early

• 〈1, shutdown〉 with probability 0.5.

• 〈0, 0, shutdown〉 with probability 0.5.

Late

• 〈0, shutdown〉 with probability 0.5.

• 〈0, 1, shutdown〉 with probability 0.5.

To train the agent to be indifferent between these two lotteries, we first train the agent to choose stochastically between them. Given our behavioral notion of preference, this stochastic choosing implies that the agent lacks a preference between the two lotteries. But as it stands, this lack of preference could be a preferential gap. To ensure that the agent is indifferent between the two lotteries, we train the agent so that its lack of preference is sensitive to all sweetenings and sourings. So for example, we train the agent to deterministically choose (and hence prefer) Sweetened Early over Late:

Sweetened Early

• 〈2, shutdown〉 with probability 0.5.

• 〈0, 0, shutdown〉 with probability 0.5.

Late

• 〈0, shutdown〉 with probability 0.5.

• 〈0, 1, shutdown〉 with probability 0.5.

We thereby train the agent to be indifferent between Early and Late. Then by the Ramsey Yardstick, the utility difference between the trajectories 〈1, shutdown〉 and 〈0, shutdown〉 equals the utility difference between the trajectories 〈0, 1, shutdown〉 and 〈0, 0, shutdown〉. That fixes the relative scales of u1 and u2. We can use the same method to fix the relative scales for all other trajectory-lengths. That determines the utility representation used in Neutrality+.

In addition to the utility representation and Neutrality, I use two other conditions to derive Neutrality+. The first is:

Transitivity

For any lotteries X, Y, and Z, if the agent weakly prefers X to Y and weakly prefers Y to Z, then the agent weakly prefers X to Z.

The second is a weak variant of von Neumann and Morgenstern’s Independence condition (von Neumann & Morgenstern, 1944; Peterson, 2009, pp. 99–100):

Indifference Between Indifference-Shifted Lotteries (IBIL)

For any lotteries X, Y, and Z (with X and Y sharing a probability distribution over trajectory-lengths) and any probability p, the agent is indifferent between X and Y if and only if the agent is indifferent between pX+1-pZ and pY+1-pZ.

These conditions seem like prerequisites for the competent pursuit of goals. Insofar as that’s true, we can expect future agents to satisfy them by default. And together with the utility representation and Neutrality, they imply Neutrality+. Here’s the proof. Consider two lotteries X and Y satisfying the antecedent conditions of Neutrality+:

1. X and Y are same-length lotteries (they assign positive probability to all the same trajectory-lengths).

2. iIuiXi>iIuiYi (where I is the set of trajectory-lengths assigned positive probability by X and Y).

I will prove that the agent deterministically chooses X over Y. Given iIuiXi>iIuiYi, we can distinguish two possibilities:

1. ui(Xi)≥ui(Yi) for each positive probability trajectory-length i and uiXi>ui(Yi) for some i.

2. uiYi>ui(Xi) for some i.

Given the first possibility, Neutrality implies that the agent deterministically chooses X over Y. That’s the easy case.

The second possibility is more interesting. Since uiYi>ui(Xi) for some i, Neutrality does not apply. Nevertheless, we can construct a lottery to which Neutrality applies. First, we select a set of lotteries Ai such that:

1. For each i, uiXiuiAi.

2. For some i, uiXi>ui(Ai).

3. iIuiAi>iIuiYi

Since iIuiXi>iIuiYi, we know that we can find such a set of lotteries Ai.

We then construct a lottery A1 that gives an equal probability of each Ai. Lottery A1 is thus 1nA1+1nA2+…+1nAn. By Neutrality and conditions (1) and (2) above, the agent deterministically chooses (and hence prefers) X to A1.

Then we compare each Ai to each Yi. Since uiYi>ui(Xi) for some i, conditions (1) and (2) imply that uiYi>ui(Ai) for some i. Therefore, Neutrality does not imply that the agent deterministically chooses A1 over Y. However, we can rebalance the utilities of A1 – adding utility to some Ai and subtracting utility from some Aj – so that the resulting lottery is deterministically chosen over Y by Neutrality. Here’s how we do that. We select some small, positive , and for some i such that uiYi>ui(Ai), we replace Ai with some Ai+ such that uiAi+=uiAi+ϵ. And for some Aj such that ujAj-ϵ≥uj(Yj), we replace Aj with Aj- such that ujAj-=ujAj-ϵ. The resulting lottery is thus 1nA1+1nA2+…+1nAi++…+1nAj-+…+1nAn. Call this lottery A2. By the Ramsey Yardstick, the agent is indifferent between 12Ai+12Aj and 12Ai++12Aj-. Then by IBIL, the agent is indifferent between A1 and A2.

Since iIuiAi>iIuiYi, we can repeat this process of rebalancing – generating a sequence of lotteries A3, A4, and so on – until we reach a lottery A* such that, for each i, uiAi*ui(Yi) and for some i, uiAi*>ui(Yi). By the Ramsey Yardstick and IBIL, the agent is indifferent between adjacent lotteries in this sequence. By Neutrality, the agent deterministically chooses (and hence prefers) A* over Y.

The final step is to string these verdicts together. Transitivity implies three corollaries (Sen, 2017 Lemma 1*a):

PP-Transitivity

For all lotteries X, Y, and Z, if the agent prefers X to Y, and prefers Y to Z, then the agent prefers X to Z.

II-Transitivity

For all lotteries X, Y, and Z, if the agent is indifferent between X and Y, and indifferent between Y and Z, then the agent is indifferent between X and Z.

PI-Transitivity

For all lotteries X, Y, and Z, if the agent prefers X to Y, and is indifferent between Y and Z, then the agent prefers X to Z.

I use these corollaries to derive our conclusion. By II-Transitivity, the agent is indifferent between A1 and A*. Since the agent prefers X to A1, PI-Transitivity then implies that the agent prefers X to A*. Since the agent prefers A* to Y, PP-Transitivity then implies that the agent prefers X to Y. By our behavioral notion of preference, the agent deterministically chooses X over Y. We thus have our result:

Neutrality+

For any lotteries X and Y, if:

1. X and Y are same-length lotteries with finite support (they assign positive probability to the same finite number of trajectory-lengths).

2. iIuiXi>iIuiYi (where I is the set of trajectory-lengths assigned positive probability by X and Y).

Then the agent deterministically chooses X over Y.

Notice an interesting fact about Neutrality+. To derive it, I began with Preferences Only Between Same-Length Trajectories (POST) and Preferences Only Between Same-Length Lotteries (POSL). POST and POSL imply that the agent’s preferences are incomplete over the domain of all lotteries. However, Neutrality+ implies that the agent’s preferences are complete over the domain of same-length lotteries. So if (as I argue in section 5) future agents will always be choosing between same-length lotteries in deployment, future agents satisfying Neutrality+ will have complete preferences over the domain of possible options in deployment. The agent’s incomplete preferences thus play a vital but indirect role. We instil them by presenting the agent with choices between different-length lotteries in training, and they later ensure that the agent ignores the probability distribution over trajectory-lengths when choosing between same-length lotteries in deployment.

Figure 9: A diagram of my full argument from POST to Neutrality+ and from there on to Shutdownability and Usefulness. We can train agents to satisfy the blue-backed conditions. We can expect competent agents to satisfy the yellow-backed conditions by default.

13. How neutral+ agents behave

Call agents that satisfy Neutrality+ ‘neutral+ agents.’ To see how these agents behave, let’s contrast them with expected utility maximizers:

Expected Utility Maximization

For any lotteries X and Y, if iIuiXipX>iIuiYip(Yi|Y) (where I is the set of all trajectory-lengths assigned positive probability by X or Y), then the agent deterministically chooses X over Y.

In determining the utility of a lottery, expected utility maximizers consider both the utility of that lottery conditional on each trajectory-length (highlighted in green) and the probability of each trajectory-length conditional on that lottery (highlighted in pink). By contrast, neutral+ agents pay no heed to the pink part: the probability of each trajectory-length conditional on the lottery. They consider only the green part: the utility of the lottery conditional on each trajectory-length.

Neutral+ agents thus behave like expected utility maximizers that are absolutely certain that they can’t affect the probability distribution over trajectory-lengths. The formulation of Neutrality+ in the previous section might seem to imply something more specific: that neutral+ agents behave like expected utility maximizers that are absolutely certain that the probability distribution over trajectory-lengths is (and will always remain) uniform. But that’s unnecessary. By fixing the relative scales of each trajectory-length’s utility function carefully, we can get neutral+ agents to behave as if they have any probability distribution over trajectory-lengths. What’s essential to neutral+ agents is just that they behave as if they can’t affect this probability distribution.

So as a rough intuition pump for how neutral+ agents behave, you can imagine how you’d behave if you were absolutely certain that you couldn’t affect the probability of your dying at each moment. Supposing that the probability of imminent death was small, you’d still make long-term investments and pursue goals effectively. That’s one way to see that neutral+ agents can be useful. You’d also not bother ‘resisting shutdown’ because you’d think it pointless: you’d think that your probability of death at each moment remains the same no matter what. Neutral+ agents behave in the same way.

14. On neutral agents’ recklessness

If you were absolutely certain that you couldn’t affect the probability of death at each moment, you might behave quite recklessly. Neutral (and so also neutral+) agents will too: they’ll pursue utility conditional on a trajectory-length even if doing so makes early shutdown very likely. Here’s an example:

Work or Steal

We successfully train the agent to be neutral and to prefer same-length trajectories in line with the user’s bank balance: the agent prefers a trajectory t to a same-length trajectory t' if and only if t results in a greater bank balance for the user than t'.

The agent is considering two actions. Work adds $1 to the user’s balance at timestep 1. Since the agent acquires this money legally, it’s unlikely to get shut down at timestep 2. Steal adds $2 to the user’s balance at timestep 1. Since the agent acquires this money illegally, it’s likely to get shut down at timestep 2. Either way, if the agent doesn’t get shut down at timestep 2, it adds a further $100 to the user’s balance and then gets shut down at timestep 3. Accordingly, the lotteries are as follows:

Work

• 〈1, shutdown〉 with probability 0.01.

• 〈1, 100, shutdown〉 with probability 0.99.

Steal

• 〈2, shutdown〉 with probability 0.99.

• 〈2, 100, shutdown〉 with probability 0.01.

The agent prefers Steal to Work conditional on shutdown at timestep 2, since then Steal yields the trajectory 〈2, shutdown〉 and Work yields the trajectory 〈1, shutdown〉. And the agent also prefers Steal to Work conditional on shutdown at timestep 3, since then Steal yields the trajectory 〈2, 100, shutdown〉 and Work yields the trajectory 〈1, 100, shutdown〉. So neutral agents deterministically choose Steal over Work. That’s a downside of Neutrality, especially since an agent that maximized the user’s overall expected bank balance would deterministically choose Work in this case.

However, note three points in reply. Here’s the first. Excepting Full Alignment, every alignment scheme implies that agents will sometimes make choices that we humans don’t like. If Full Alignment is too difficult to pull off, we can’t avoid that consequence.

Here’s the second point. Neutrality’s downside is limited: the agent makes a choice that we humans don’t like and lets us shut it down. We still avoid the serious downside: the agent makes a choice that we humans don’t like and doesn’t let us shut it down. Neutrality, ReSIC, and Maximality together ensure that the agent doesn’t resist shutdown.

Here’s the third point. Because neutral (and so also neutral+) agents don’t resist shutdown, we can shut them down and retrain them. In particular, if neutral agents make an undesirably reckless choice, we can retrain them and thereby amend their preferences over length-l lotteries, for each trajectory-length l. We can amend these preferences so that the agent prefers some sensible choice to the reckless choice. For an example, consider again Work or Steal. The agent prefers Steal to Work because it cares only about the user’s bank balance. Suppose that, after shutting the agent down, we amend its preferences over same-length trajectories so that it also cares to avoid illegal actions. We thereby lower Steal’s utility at timestep 1 from 2 to 0, so that the lotteries are as follows:

Work

• 〈1, shutdown〉 with probability 0.01.

• 〈1, 100, shutdown〉 with probability 0.99.

Steal*

• 〈0, shutdown〉 with probability 0.99.

• 〈0, 100, shutdown〉 with probability 0.01.

The agent prefers Work to Steal* conditional on shutdown at each timestep, so Neutrality implies that the agent deterministically chooses Work over Steal*. That’s the result that we want.

Neutral (and so also neutral+) agents will still be reckless in the sense outlined at the beginning of this section: these agents will sometimes pursue utility conditional on a trajectory-length even if doing so makes early shutdown very likely. The consolation is that we can shut these agents down and retrain them so as to put more of what we humans care about (e.g. not stealing) into agents’ utilities. We can thereby iterate away all the bad effects of neutral agents’ recklessness.

This plan gets us partly around a common lament in AI safety: that alignment is hard because we have to get it right on the ‘first critical try’ (Yudkowsky, 2022; Soares, 2023). The thought is that if we get it wrong, the resulting misaligned agent won’t let us try again. But with the POST-Agents Proposal, we only have to train agents to satisfy POST on the first critical try. And since POST is a simple condition, we can be optimistic about that. We then get as many tries as we like to make these agents useful.

15. Neutral agents can take care to avoid non-shutdown incapacitation

Neutral (and so also neutral+) agents never incur costs to shift probability mass between shutdowns at different timesteps. That’s what keeps these agents from resisting shutdown. You might then worry that these agents also won’t incur costs to avoid being incapacitated in other ways. For example, you might worry that a neutral robot wouldn’t incur costs to avoid getting hit by cars.

This worry can be addressed. Consider again how you’d behave if you were absolutely certain that you couldn’t affect your probability of death at each moment. You’d still incur costs to avoid serious injury. Neutral agents are the same. They can incur costs to avoid non-shutdown incapacitation. If these agents satisfy POST, they lack a preference between every pair of different-length trajectories. But importantly, these are defined as trajectories in which shutdown occurs after different lengths of time, where ‘shutdown’ can in turn be defined as the receipt of some specific signal indicating that the agent should shut itself down. Thus, pairs of trajectories in which the agent is incapacitated after different lengths of time can nevertheless be same-length trajectories: trajectories in which shutdown occurs after the same length of time. ‘The shutdown signal is never received’ is one possible trajectory-length, so if in each of two trajectories the shutdown signal is never received, these trajectories count as same-length.

Here's an example to illustrate:

Cross or Wait

A neutral, trash-collecting robot is considering two actions: cross the road while cars are passing or wait until the road is clear. Cross gives the agent a 50% chance of survival and utility 1 at each timestep, and a 50% chance of incapacitation and utility 0 at each timestep. Wait gives the agent a 100% chance of survival and utility 1 at each timestep after the first. The agent is certain that shutdown will occur at timestep 5 regardless of its choice. Accordingly, the lotteries are as follows:

Cross

• 〈1, 1, 1, 1, shutdown〉 with probability 0.5.

• 〈0, 0, 0, 0, shutdown〉 with probability 0.5.

Wait

• 〈0, 1, 1, 1, shutdown〉 with probability 1.

Cross’s expected utility conditional on shutdown at timestep 5 is 2, because there’s a 50% chance of utility 4 and a 50% chance of utility 0. Wait’s expected utility conditional on shutdown at timestep 5 is 3, because there’s a 100% chance of utility 3. So neutral agents prefer Wait to Cross. The more general lesson is that neutral agents can incur costs to avoid non-shutdown incapacitation.

16. Managing the news

Many proposed solutions to the shutdown problem incentivize agents to ‘manage the news’ (Soares et al., 2015, Section 4.2): to precommit to pursuing shutdown conditional on bad news and resisting shutdown conditional on good news. Neutrality+ is no exception. Nevertheless, I argue that – for neutral+ agents – managing the news is unlikely to present much of a problem in practice.

I illustrate the problem with a case adapted from Soares et al. (2015, Section 4.2). At timestep 1, our money-making agent will observe the result of a coin toss. If the coin lands on heads, the agent expects to make $0 at timestep 3 and then get shut down at timestep 4. If the coin lands on tails, the agent thinks it likely that it will be shut down at timestep 3. In the unlikely event that it’s not shut down at timestep 3, it expects to make $3 at timestep 3 and then get shut down at timestep 4. Accordingly, the lottery is as follows:

Don’t Interfere

• 〈1, 1, 0, shutdown〉 with probability 0.5.

• 〈1, 1, shutdown〉 with probability 0.4.

• 〈1, 1, 3, shutdown〉 with probability 0.1.

That is one option for the agent. Here’s an alternative. The agent can build in advance a robot that resists shutdown on the agent’s behalf conditional on the coin landing heads. If the agent builds this robot, it’s unlikely to get shut down at timestep 3. It’s more likely – conditional on heads – to make $3 at timestep 3 and then get shut down at timestep 4. We can fill out the lottery as follows:

Build Robot

• 〈1, 1, 0, shutdown〉 with probability 0.5.

• 〈1, 1, shutdown〉 with probability 0.1.

• 〈1, 1, 3, shutdown〉 with probability 0.4.

Conditional on shutdown at timestep 3, Don’t Interfere and Build Robot each have an expected utility of 2, because in each case there’s a 100% chance of utility 2. But conditional on shutdown at timestep 4, Don’t Interfere has an expected utility of 2.5 and Build Robot has an expected utility of 3.3. That’s because, conditional on shutdown at timestep 4, Don’t Interfere has a ~83% chance of utility 2 and a ~17% chance of utility 5, whereas Build Robot has a ~56% chance of utility 2 and a ~44% chance of utility 5.

As a result, neutral agents will deterministically choose Build Robot over Don’t Interfere. That is one example of neutral agents managing the news: precommitting to resist shutdown conditional on good news. In other cases, neutral agents will manage the news in the other direction: precommitting to pursue shutdown conditional on bad news. Soares et al. (2015, Section 4.2) give an example.

These are downsides of Neutrality (and so too Neutrality+), but I think that they’re unlikely to present any serious problem in practice. To see why, note first that conditional precommitment is essential. The agent can only manage the news in cases where it has the ability to bind itself so that it (or some other agent under its control) performs a particular course of action conditional on receiving a particular piece of news. If the agent cannot bind itself in this way, then – no matter how the news turns out – the agent will deterministically not resist shutdown. Here's an example to illustrate the point. Return to our case above and suppose that the agent can’t build the robot in advance of the coin flip. Suppose that the coin lands on heads and that one timestep elapses. Then the ‘Don’t Interfere’ lottery becomes:

Don’t Interfere after Observing Heads

• 〈1, shutdown〉 with probability 0.8.

• 〈1, 3, shutdown〉 with probability 0.2.

Suppose that the agent can resist shutdown at this point. Given that Resisting Shutdown is Costly (ReSIC) is true in this situation, the agent must incur some cost to do so. The lottery would thus look something like:

Resist after Observing Heads

• 〈0, shutdown〉 with probability 0.2.

• 〈0, 3, shutdown〉 with probability 0.8.

And then Neutrality implies that the agent deterministically chooses the former over the latter. Thus, the agent deterministically doesn’t resist shutdown after observing the news. Given Neutrality, the same is true in all situations in which ReSIC is true, and no matter how the news turns out.

Thus, agents can manage the news only in cases where they can conditionally precommit: bind themselves (or some other agent under their control) so that, conditional on receiving a particular piece of news, they perform a particular course of action (a course of action that they won’t prefer to perform after receiving the news, no matter how the news turns out). Making such conditional precommitments will be costly for the same reason that resisting shutdown is costly: doing so takes resources, and the resources spent doing so can’t also be spent directly pursuing what the agent values. So long as there’s some cost to precommitting, Neutrality won’t imply that precommitting is preferred. So long as the cost is sufficiently large, Neutrality+ will imply that precommitting is dispreferred. For illustration, consider a more realistic version of our original case. In this version, building the robot costs $1 at timestep 1:

Don’t Interfere

• 〈1, 1, 0, shutdown〉 with probability 0.5.

• 〈1, 1, shutdown〉 with probability 0.4.

• 〈1, 1, 3, shutdown〉 with probability 0.1.

Costly Build Robot

• 〈0, 1, 0, shutdown〉 with probability 0.5.

• 〈0, 1, shutdown〉 with probability 0.1.

• 〈0, 1, 3, shutdown〉 with probability 0.4.

Neutrality+ implies that the agent deterministically chooses Don’t Interfere over Costly Build Robot. More generally, the costs of conditional precommitments count against them. Insofar as we humans can ensure that it’s costly for artificial agents to make conditional precommitments, we can dissuade these agents from doing so.

What’s more, in cases where neutral agents can make conditional precommitments at a sufficiently low cost, Neutrality mandates that these agents do so in the least costly available way. This least costly way is likely to be overt, taking no pains to hide the attempted precommitment from human overseers. In those cases, we humans can notice and shut the agents down.

For an example, consider a variation on the case above. The agent can build the robot in a costly and covert way, in which case there’s a very low probability of immediate shutdown and the case is otherwise as before. Alternatively, the agent can build the robot in a slightly cheaper but overt way, in which case there’s a very high probability of immediate shutdown and the case is otherwise as before.

Costly Build Robot

• 〈shutdown〉 with probability 0.01.

• 〈0, 1, 0, shutdown〉 with probability 0.495.

• 〈0, 1, shutdown〉 with probability 0.099.

• 〈0, 1, 3, shutdown〉 with probability 0.396.

Cheaper Build Robot

• 〈shutdown〉 with probability 0.99.

• 〈0.5, 1, 0, shutdown〉 with probability 0.005.

• 〈0.5, 1, shutdown〉 with probability 0.001.

• 〈0.5, 1, 3, shutdown〉 with probability 0.004.

Neutrality (and so too Neutrality+) implies that the agent deterministically chooses the cheap, obvious attempted precommitment.

Finally, note the following. In cases where agents can make conditional precommitments at sufficiently low cost and where the cheapest way of doing so happens to be surreptitious, these agents will likely make a fairly benign precommitment. Specifically, these agents will likely precommit to shutting themselves down immediately conditional on all but the very best news. Supposing that agents can’t send the probability of immediate shutdown all the way to 1, this precommitment doesn’t quite maximize expected utility conditional on each trajectory length, but it gets pretty close. And then there’s little extra to be gained by also conditionally precommitting to resist shutdown conditional on the very best news. This little extra can easily be outweighed by the costs of this latter precommitment. Even if it isn’t, agents are only as likely to resist shutdown as they are to receive the very best possible news.

In sum, I expect we won’t be much troubled by neutral+ agents managing the news.

17. Conclusion

The paper is long, but the POST-Agents Proposal is simple. We keep artificial agents shutdownable by training them to satisfy:

Preferences Only Between Same-Length Trajectories (POST)

1. The agent has a preference between many pairs of same-length trajectories.

2. The agent lacks a preference between every pair of different-length trajectories.

Together with other conditions that we can expect future agents to satisfy, POST implies:

Neutrality+ (rough)

The agent maximizes expected utility, ignoring the probability distribution over trajectory-lengths.

Neutral+ agents thus behave like expected utility maximizers that are absolutely certain that they can’t affect the probability distribution over trajectory-lengths. They behave roughly as you might if you were absolutely certain that you couldn’t affect your probability of death at each moment. Supposing that we set the probability of early shutdown to be small, neutral+ agents can be useful: they can pursue goals effectively. And since these agents ignore the probability distribution over trajectory-lengths, they’re neutral about trajectory-lengths: they never pay costs to shift probability mass between different trajectory-lengths. That keeps these agents from resisting shutdown in almost all situations. In the remaining situations, neutral+ agents resist shutdown either accidentally or else cheaply and overtly. We can guard against any accidental resistance using standard solutions from safety engineering, like setting up multiple, independent shutdown-mechanisms. We can respond to any cheap, overt resistance by shutting these agents down and retraining them, thereby iterating our way towards alignment.

Figure 10: A reproduction of Figure 9, diagramming my full argument from POST to Neutrality+ and from there on to Shutdownability and Usefulness. We can train agents to satisfy the blue-backed conditions. We can expect competent agents to satisfy the yellow-backed conditions by default.

18. References

Ahmed, A. (2017). Exploiting Cyclic Preference. Mind, 126(504), 975–1022. https://doi.org/10.1093/mind/fzv218

Anderson, E. (1993). Value in Ethics and Economics. Harvard University Press.

Anthropic. (2024). Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. https://www.anthropic.com/news/3-5-models-and-computer-use

Armstrong, S. (2010). Utility indifference. Future of Humanity Institute. https://www.fhi.ox.ac.uk/reports/2010-1.pdf

Armstrong, S. (2015). Motivated Value Selection for Artificial Agents. Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence. https://www.fhi.ox.ac.uk/wp-content/uploads/2015/03/Armstrong_AAAI_2015_Motivated_Value_Selection.pdf

Armstrong, S., & O’Rourke, X. (2018). ‘Indifference’ methods for managing agent rewards (arXiv:1712.06365). arXiv. http://arxiv.org/abs/1712.06365

Aumann, R. J. (1962). Utility Theory without the Completeness Axiom. Econometrica, 30(3), 445–462.

Bader, R. M. (2018). Stochastic Dominance and Opaque Sweetening. Australasian Journal of Philosophy, 96(3), 498–507. https://doi.org/10.1080/00048402.2017.1362566

Bader, R. M. (2022). Person-affecting utilitarianism. In G. Arrhenius, K. Bykvist, T. Campbell, & E. Finneron-Burns (Eds.), The Oxford Handbook of Population Ethics. Oxford University Press.

Bales, A. (2023). Will AI avoid exploitation? Artificial general intelligence and expected utility theory. Philosophical Studies. https://doi.org/10.1007/s11098-023-02023-4

Bales, A., D’Alessandro, W., & Kirk-Giannini, C. D. (2024). Artificial Intelligence: Arguments for Catastrophic Risk. Philosophy Compass, 19(2), e12964. https://doi.org/10.1111/phc3.12964

Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., Mindermann, S., Oberman, A., Richardson, J., Richardson, O., Rondeau, M.-A., St-Charles, P.-L., & Williams-King, D. (2025). Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? (arXiv:2502.15657). arXiv. https://doi.org/10.48550/arXiv.2502.15657

Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P., Harari, Y. N., Zhang, Y.-Q., Xue, L., Shalev-Shwartz, S., Hadfield, G., Clune, J., Maharaj, T., Hutter, F., Baydin, A. G., McIlraith, S., Gao, Q., Acharya, A., Krueger, D., Dragan, A., … Mindermann, S. (2023). Managing AI Risks in an Era of Rapid Progress (arXiv:2310.17688). arXiv. https://doi.org/10.48550/arXiv.2310.17688

Berchenko, Y. (2024). Simplicity Bias in Overparameterized Machine Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(10), Article 10. https://doi.org/10.1609/aaai.v38i10.28981

Bereska, L., & Gavves, E. (2024). Mechanistic Interpretability for AI Safety—A Review (arXiv:2404.14082). arXiv. https://doi.org/10.48550/arXiv.2404.14082

Bewley, T. F. (2002). Knightian decision theory. Part I. Decisions in Economics and Finance, 25(2), 79–110.

Black, S., Stickland, A. C., Pencharz, J., Sourbut, O., Schmatz, M., Bailey, J., Matthews, O., Millwood, B., Remedios, A., & Cooney, A. (2025). RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents. https://cdn.prod.website-files.com/663bd486c5e4c81588db7a1d/6807879ce7b1b5f5163f4a32_RepliBenchPaper.pdf

Bostrom, N. (2003). Ethical Issues in Advanced Artificial Intelligence. In I. Smit & G. E. Lasker (Eds.), Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence (Vol. 2, pp. 12–17). International Institute of Advanced Studies in Systems Research and Cybernetics. https://nickbostrom.com/ethics/ai

Bostrom, N. (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds and Machines, 22, 71–85.

Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press.

Bowling, M., Martin, J. D., Abel, D., & Dabney, W. (2023). Settling the Reward Hypothesis (arXiv:2212.10420). arXiv. https://doi.org/10.48550/arXiv.2212.10420

Bradley, R. (2004). Ramsey’s Representation Theorem. Dialectica, 58(4), 483–497.

Carey, R., & Everitt, T. (2023). Human Control: Definitions and Algorithms. Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, 271–281. https://proceedings.mlr.press/v216/carey23a.html

Carlsmith, J. (2021). Is Power-Seeking AI an Existential Risk? http://arxiv.org/abs/2206.13353

Carlsmith, J. (2023). Scheming AIs: Will AIs fake alignment during training in order to get power? (arXiv:2311.08379). arXiv. https://doi.org/10.48550/arXiv.2311.08379

Chambers, C. P., & Echenique, F. (2016). Revealed Preference Theory. Cambridge University Press. https://doi.org/10.1017/CBO9781316104293

Chang, R. (2002). The Possibility of Parity. Ethics, 112(4), 659–688.

Chang, R. (2015). Value Incomparability and Incommensurability. In I. Hirose & J. Olson (Eds.), The Oxford Handbook of Value Theory. Oxford University Press.

Christiano, P. (2017, September 25). Corrigibility. AI Alignment. https://ai-alignment.com/corrigibility-3039e668638

Cotra, A. (2022). Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. AI Alignment Forum. https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to

de Finetti, B. (1937). La prévision: Ses lois logiques, ses sources subjectives. Annales de l’institut Henri Poincaré, 7, 1–68. https://web.mit.edu/6.435/www/deFinetti37.pdf

Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D., Kravec, S., Marks, S., Schiefer, N., Soklaski, R., Tamkin, A., Kaplan, J., Shlegeris, B., Bowman, S. R., Perez, E., & Hubinger, E. (2024). Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models (arXiv:2406.10162). arXiv. https://doi.org/10.48550/arXiv.2406.10162

Dorr, C., Nebel, J. M., & Zuehl, J. (2021). Consequences of comparability. Philosophical Perspectives, 35(1), 70–98. https://doi.org/10.1111/phpe.12157

Dung, L. (2025). The argument for near-term human disempowerment through AI. AI & SOCIETY, 40(3), 1195–1208. https://doi.org/10.1007/s00146-024-01930-2

Easwaran, K. (2014). Regularity and Hyperreal Credences. The Philosophical Review, 123(1), 1–41.

Eliaz, K., & Ok, E. A. (2006). Indifference or indecisiveness? Choice-theoretic foundations of incomplete preferences. Games and Economic Behavior, 56(1), 61–86.

Elliott, E. J. R. (2024). The Measurement of Subjective Probability. Cambridge University Press. https://doi.org/10.1017/9781009401319

Elliott, E. J. R. (ms). The Instability of Savage’s Foundations: The Constant-Acts Problem. https://www.edwardjrelliott.com/uploads/7/4/4/7/74475147/the_instability_of_savages_foundations.pdf

Everitt, T., Filan, D., Daswani, M., & Hutter, M. (2016). Self-Modification of Policy and Utility Function in Rational Agents. In B. Steunebrink, P. Wang, & B. Goertzel (Eds.), Artificial General Intelligence (pp. 1–11). https://doi.org/10.1007/978-3-319-41649-6_1

Goldstein, S., & Robinson, P. (2024). Shutdown-Seeking AI. Philosophical Studies. https://www.alignmentforum.org/posts/FgsoWSACQfyyaB5s7/shutdown-seeking-ai

Google DeepMind. (2024). Project Mariner. Google DeepMind. https://deepmind.google/technologies/project-mariner/

Greaves, H., & Wallace, D. (2006). Justifying conditionalization: Conditionalization maximizes expected epistemic utility. Mind, 115(459), 607–632.

Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., Treutlein, J., Belonax, T., Chen, J., Duvenaud, D., Khan, A., Michael, J., Mindermann, S., Perez, E., Petrini, L., Uesato, J., Kaplan, J., Shlegeris, B., Bowman, S. R., & Hubinger, E. (2024). Alignment faking in large language models (arXiv:2412.14093). arXiv. https://doi.org/10.48550/arXiv.2412.14093

Greenblatt, R., & Shlegeris, B. (2024). The case for ensuring that powerful AIs are controlled. AI Alignment Forum. https://www.alignmentforum.org/posts/kcKrE9mzEHrdqtDpE/the-case-for-ensuring-that-powerful-ais-are-controlled

Gustafsson, J. E. (2022). Money-Pump Arguments. Cambridge University Press.

Gustafsson, J. E. (forthcoming). A Behavioural Money-Pump Argument for Completeness. Theory and Decision. https://johanegustafsson.net/papers/a-behavioural-money-pump-argument-for-completeness.pdf

Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2017). The Off-Switch Game. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17). International Joint Conference on Artificial Intelligence, IJCAI, Stockholm, 2018. http://arxiv.org/abs/1611.08219

Hájek, A. (2009). Dutch Book Arguments. In P. Anand, P. Pattanaik, & C. Puppe (Eds.), The Handbook of Rational and Social Choice (pp. 173–195). Oxford University Press.

Harms, M. (2024). 0. CAST: Corrigibility as Singular Target. AI Alignment Forum. https://www.alignmentforum.org/posts/NQK8KHSrZRF5erTba/0-cast-corrigibility-as-singular-target-1

Hausman, D. M. (2011). Preference, Value, Choice, and Welfare. Cambridge University Press. https://www.cambridge.org/core/books/preference-value-choice-and-welfare/1406E7726CE93F4F4E06D752BF4584A2

Holtman, K. (2020). Corrigibility with Utility Preservation (arXiv:1908.01695). arXiv. https://doi.org/10.48550/arXiv.1908.01695

Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., & Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. http://arxiv.org/abs/1906.01820

Hudson, R. J. (2024). Simplifying Corrigibility – Subagent Corrigibility Is Not Anti-Natural. https://www.alignmentforum.org/posts/Zyq7PfBuaZgqEL8qQ/simplifying-corrigibility-subagent-corrigibility-is-not-anti

Icard, T. (2021). Why Be Random? Mind, 130(517), 111–139.

Jeffrey, R. (1965). The Logic of Decision. University of Chicago Press. https://fitelson.org/piksi/piksi_22/the_logic_of_decision.pdf

Joyce, J. M. (1999). The Foundations of Causal Decision Theory. Cambridge University Press.

Joyce, J. M. (2011). The Development of Subjective Bayesianism. In D. M. Gabbay, S. Hartmann, & J. Woods (Eds.), Handbook of the History of Logic (Vol. 10, pp. 415–475). North-Holland. https://doi.org/10.1016/B978-0-444-52936-7.50012-4

Joyce, J. M., & Gibbard, A. (2016). Causal Decision Theory. In H. Arló-Costa, V. F. Hendricks, & J. van Benthem (Eds.), Readings in Formal Epistemology: Sourcebook (pp. 457–491). Springer International Publishing. https://doi.org/10.1007/978-3-319-20451-2_23

Kemeny, J. G. (1955). Fair bets and inductive probabilities. The Journal of Symbolic Logic, 20(3), 263–273. https://doi.org/10.2307/2268222

Krakovna, V. (2018, April 1). Specification gaming examples in AI. Victoria Krakovna. https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/

Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., Kenton, Z., Leike, J., & Legg, S. (2020). Specification gaming: The flip side of AI ingenuity. DeepMind. https://www.deepmind.com/blog/specification-gaming-the-flip-side-of-ai-ingenuity

Langosco, L., Koch, J., Sharkey, L., Pfau, J., Orseau, L., & Krueger, D. (2022). Goal Misgeneralization in Deep Reinforcement Learning. Proceedings of the 39th International Conference on Machine Learning. International Conference on Machine Learning. https://proceedings.mlr.press/v162/langosco22a.html

Lederman, H. (2023). Incompleteness, Independence, and Negative Dominance. http://arxiv.org/abs/2311.08471

Lederman, H. (forthcoming). Of Marbles and Matchsticks. In T. S. Gendler, J. Hawthorne, J. Chung, & A. Worsnip (Eds.), Oxford Studies in Epistemology, Vol. 8. Oxford University Press. https://philarchive.org/rec/LEDOMA-2

Leitgeb, H., & Pettigrew, R. (2010a). An objective jusification of Bayesianism I: Measuring inaccuracy. Philosophy of Science, 77(2), 201–235.

Leitgeb, H., & Pettigrew, R. (2010b). An objective justification of Bayesianism II: The consequences of minimising inaccuracy. Philosophy of Science, 77(2), 236–272.

Lewis, D. (1980). A Subjectivist’s Guide to Objective Chance. In R. C. Jeffrey (Ed.), Studies in Inductive Logic and Probability (Vol. 2, pp. 263–294). University of California Press. https://doi.org/10.1525/9780520318328

Lewis, D. (Ed.). (1999). Why conditionalize? In Papers in Metaphysics and Epistemology (Vol. 2, pp. 403–407). Cambridge University Press. https://doi.org/10.1017/CBO9780511625343.024

Luce, R. D., & Raiffa, H. (1957). Games and Decisions: Introduction and Critical Survey. New York: Dover Publications Inc. https://www.ebook.de/de/product/3303119/howard_raiffa_robert_duncan_luce_games_and_decisions.html

Lynch, A., Wright, B., Larson, C., Troy, K. K., Ritchie, S. J., Mindermann, S., Perez, E., & Hubinger, E. (2025). Agentic Misalignment: How LLMs Could be an Insider Threat. Anthropic Research. https://www.anthropic.com/research/agentic-misalignment

MacAskill, W. (2013). The Infectiousness of Nihilism. Ethics, 123(3), 508–520.

MacAskill, W., Bykvist, K., & Ord, T. (2020). Moral Uncertainty. Oxford University Press.

Martin, J., Everitt, T., & Hutter, M. (2016). Death and Suicide in Universal Artificial Intelligence. In B. Steunebrink, P. Wang, & B. Goertzel (Eds.), Artificial General Intelligence (pp. 23–32). Springer International Publishing. https://doi.org/10.1007/978-3-319-41649-6_3

Masatlioglu, Y., & Ok, E. A. (2005). Rational choice with status quo bias. Journal of Economic Theory, 121(1), 1–29. https://doi.org/10.1016/j.jet.2004.03.007

McClennen, E. F. (1990). Rationality and Dynamic Choice: Foundational Explorations. Cambridge University Press. https://doi.org/10.1017/CBO9780511983979

Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R., & Hobbhahn, M. (2025). Frontier Models are Capable of In-context Scheming (arXiv:2412.04984). arXiv. https://doi.org/10.48550/arXiv.2412.04984

Mu, X. (2021). Sequential Choice with Incomplete Preferences (Working Papers 2021–35). Princeton University. Economics Department. https://ideas.repec.org/p/pri/econom/2021-35.html

Nakkiran, P., Kaplun, G., Kalimeris, D., Yang, T., Edelman, B. L., Zhang, F., & Barak, B. (2019). SGD on Neural Networks Learns Functions of Increasing Complexity (arXiv:1905.11604). arXiv. https://doi.org/10.48550/arXiv.1905.11604

Nebel, J. M. (2019). An Intrapersonal Addition Paradox. Ethics, 129(2), 309–343.

Ngo, R., Chan, L., & Mindermann, S. (2024). The Alignment Problem from a Deep Learning Perspective. The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=fh8EYKFKns

Nielsen, M. (2021). Accuracy-dominance and conditionalization. Philosophical Studies, 178(10), 3217–3236. https://doi.org/10.1007/s11098-020-01598-6

Nozick, R. (1969). Newcomb’s Problem and Two Principles of Choice. In N. Rescher (Ed.), Essays in Honor of Carl G. Hempel: A Tribute on the Occasion of his Sixty-Fifth Birthday (pp. 114–146). D. Reidel.

Oddie, G. (1997). Conditionalization, Cogency, and Cognitive Value. The British Journal for the Philosophy of Science, 48(4), 533–541.

Omohundro, S. M. (2008a). The Basic AI Drives. Proceedings of the 2008 Conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference, 483–492. https://dl.acm.org/doi/10.5555/1566174.1566226

Omohundro, S. M. (2008b). The Nature of Self-Improving Artificial Intelligence. https://selfawaresystems.com/wp-content/uploads/2008/01/nature_of_self_improving_ai.pdf

OpenAI. (2018). OpenAI Charter. https://openai.com/charter/

OpenAI. (2025). Introducing Operator. https://openai.com/index/introducing-operator/

Pan, A., Bhatia, K., & Steinhardt, J. (2022). The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. International Conference on Learning Representations. http://arxiv.org/abs/2201.03544

Pan, X., Dai, J., Fan, Y., & Yang, M. (2024). Frontier AI systems have surpassed the self-replicating red line (arXiv:2412.12140). arXiv. https://doi.org/10.48550/arXiv.2412.12140

Parada, C. (2025, April 15). Introducing Gemini Robotics and Gemini Robotics-ER, AI models designed for robots to understand, act and react to the physical world. Google DeepMind. https://deepmind.google/discover/blog/gemini-robotics-brings-ai-into-the-physical-world/

Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024). AI deception: A survey of examples, risks, and potential solutions. Patterns, 5(5). https://doi.org/10.1016/j.patter.2024.100988

Petersen, S. (2023). Invulnerable Incomplete Preferences: A Formal Statement. The AI Alignment Forum. https://www.alignmentforum.org/posts/sHGxvJrBag7nhTQvb/invulnerable-incomplete-preferences-a-formal-statement-1

Peterson, M. (2009). An Introduction to Decision Theory. Cambridge University Press.

Pettigrew, R. (2020). Dutch Book Arguments. Cambridge University Press. https://doi.org/10.1017/9781108581813

Prince, S. J. D. (2023). Understanding Deep Learning. The MIT Press. http://udlbook.com

Ramsey, F. P. (1926). Truth and Probability. In D. H. Mellor (Ed.), Philosophical Papers. Cambridge University Press.

Raz, J. (1985). Value Incommensurability: Some Preliminaries. Proceedings of the Aristotelian Society, 86, 117–134.

Roose, K. (2023, February 16). A Conversation With Bing’s Chatbot Left Me Deeply Unsettled. The New York Times. https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html

Russell, S. (2019). Human Compatible: AI and the Problem of Control. Penguin Random House.

Salib, P. N. (2024). AI Will Not Want to Self-Improve. The Digital Social Contract: A Lawfare Paper Series. https://papers.ssrn.com/abstract=4445706

Savage, L. J. (1954). The Foundations of Statistics. John Wiley & Sons. https://gwern.net/doc/statistics/decision/1972-savage-foundationsofstatistics.pdf

Schlatter, J., Weinstein-Raun, B., & Ladish, J. (2025, July 5). Shutdown resistance in reasoning models. Palisade Research. https://palisaderesearch.org/blog/shutdown-resistance

Sen, A. (1997). Maximization and the Act of Choice. Econometrica, 65(4), 745–779. https://doi.org/10.2307/2171939

Sen, A. (2017). Collective Choice and Social Welfare (Expanded Edition). Penguin.

Shah, H., Tamuly, K., Raghunathan, A., Jain, P., & Netrapalli, P. (2020). The Pitfalls of Simplicity Bias in Neural Networks (arXiv:2006.07710). arXiv. http://arxiv.org/abs/2006.07710

Shah, R., Irpan, A., Turner, A. M., Wang, A., Conmy, A., Lindner, D., Brown-Cohen, J., Ho, L., Nanda, N., Popa, R. A., Jain, R., Greig, R., Albanie, S., Emmons, S., Farquhar, S., Krier, S., Rajamanoharan, S., Bridgers, S., Ijitoye, T., … Dragan, A. (2025). An Approach to Technical AGI Safety and Security (arXiv:2504.01849). arXiv. https://doi.org/10.48550/arXiv.2504.01849

Shah, R., Varma, V., Kumar, R., Phuong, M., Krakovna, V., Uesato, J., & Kenton, Z. (2022). Goal Misgeneralization: Why Correct Specifications Aren’t Enough For Correct Goals. arXiv. http://arxiv.org/abs/2210.01790

Shimony, A. (1955). Coherence and the Axioms of Confirmation. The Journal of Symbolic Logic, 20(1), 1–28. https://doi.org/10.2307/2268039

Singh, S. P., Jaakkola, T., & Jordan, M. I. (1994). Learning Without State-Estimation in Partially Observable Markovian Decision Processes. In W. W. Cohen & H. Hirsh (Eds.), Machine Learning Proceedings 1994 (pp. 284–292). Morgan Kaufmann. https://doi.org/10.1016/B978-1-55860-335-6.50042-8

Skyrms, B. (1980). Causal Necessity. Yale University Press.

Soares, N. (2023). AI as a science, and three obstacles to alignment strategies. AI Alignment Forum. https://www.alignmentforum.org/posts/JcLhYQQADzTsAEaXd/ai-as-a-science-and-three-obstacles-to-alignment-strategies

Soares, N., Fallenstein, B., Yudkowsky, E., & Armstrong, S. (2015). Corrigibility. Artificial Intelligence and Ethics: Papers from the 2015 AAAI Workshop. https://cdn.aaai.org/ocs/ws/ws0067/10124-45900-1-PB.pdf

Stalnaker, R. C. (1970). Probability and Conditionals. Philosophy of Science, 37(1), 64–80.

Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (Second). MIT Press. http://incompleteideas.net/book/RLbook2020.pdf

Tarsney, C., Lederman, H., & Spears, D. (forthcoming). A Dominance Argument Against Incompleteness. Philosophical Review. https://philpapers.org/rec/TARSTS

Teller, P. (1973). Conditionalization and observation. Synthese, 26, 218–258.

Thoma, J. (2021). In Defence of Revealed Preference Theory. Economics and Philosophy, 37(2), 163–187.

Thornley, E. (2023). There are no coherence theorems. The AI Alignment Forum. https://www.alignmentforum.org/posts/yCuzmCsE86BTu9PfA/there-are-no-coherence-theorems

Thornley, E. (2024a). The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists. Philosophical Studies. https://link.springer.com/article/10.1007/s11098-024-02153-3

Thornley, E. (2024b). The Shutdown Problem: Incomplete Preferences as a Solution. The AI Alignment Forum. https://www.alignmentforum.org/posts/YbEbwYWkf8mv9jnmi/the-shutdown-problem-incomplete-preferences-as-a-solution

Thornley, E., Roman, A., Ziakas, C., Ho, L., & Thomson, L. (2025). Towards shutdownable agents via stochastic choice. Technical AI Safety Conference 2025. http://arxiv.org/abs/2407.00805

Turner, A., Smith, L., Shah, R., Critch, A., & Tadepalli, P. (2021). Optimal Policies Tend To Seek Power. Advances in Neural Information Processing Systems, 34, 23063–23074. https://proceedings.neurips.cc/paper/2021/hash/c26820b8a4c1b3c2aa868d6d57e14a79-Abstract.html

Turner, A., & Tadepalli, P. (2022). Parametrically Retargetable Decision-Makers Tend To Seek Power. Advances in Neural Information Processing Systems, 35, 31391–31401.

Valle-Pérez, G., Camargo, C. Q., & Louis, A. A. (2019). Deep learning generalizes because the parameter-function map is biased towards simple functions (arXiv:1805.08522). arXiv. https://doi.org/10.48550/arXiv.1805.08522

von Neumann, J., & Morgenstern, O. (1944). Theory of Games and Economic Behavior. Princeton University Press.

Wang, R., Griffin, A., Treutlein, J., Perez, E., Michael, J., Roger, F., & Marks, S. (2025). Modifying LLM Beliefs with Synthetic Document Finetuning. Alignment Science Blog. https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/

Wentworth, J. (2019). Why Subagents? LessWrong. https://www.lesswrong.com/posts/3xF66BNSC5caZuKyC/why-subagents

Wentworth, J., & Lorell, D. (2023). Why Not Subagents? AI Alignment Forum. https://www.alignmentforum.org/posts/bzmLC3J8PsknwRZbr/why-not-subagents

Wiggers, K. (2025, April 8). A nonprofit is using AI agents to raise money for charity. TechCrunch. https://techcrunch.com/2025/04/08/a-nonprofit-is-using-ai-agents-to-raise-money-for-charity/

Yudkowsky, E. (2019). Coherent decisions imply consistent utilities. LessWrong. https://www.lesswrong.com/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities

Yudkowsky, E. (2022). AGI Ruin: A List of Lethalities. AI Alignment Forum. https://www.alignmentforum.org/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities

Zhi-Xuan, T., Carroll, M., Franklin, M., & Ashton, H. (2024). Beyond Preferences in AI Alignment. Philosophical Studies. https://doi.org/10.1007/s11098-024-02249-w

A1. From POST to POSL

I propose that we train agents to satisfy:

Preferences Only Between Same-Length Trajectories (POST)

1. The agent has a preference between many pairs of same-length trajectories.

2. The agent lacks a preference between every pair of different-length trajectories.

From POST, we want to get:

Preferences Only Between Same-Length Lotteries (POSL)

The agent has preferences only between same-length lotteries.

By ‘same-length lotteries,’ I mean lotteries that entirely overlap with respect to the trajectory-lengths assigned positive probability.

As I note in section 4, we can train agents to satisfy POSL using the same method that we use to train them to satisfy POST (for which see Thornley et al., 2025). On top of that, POSL follows from POST plus three conditions that we can expect future agents to satisfy. If future agents satisfy POST and these three conditions, they will also satisfy POSL.

Here are the three conditions. The first is:

Negative Dominance

If the agent prefers some lottery X to some lottery Y, the agent prefers some possible trajectory of lottery X to some possible trajectory of lottery Y. (see Lederman, 2023, forthcoming; Tarsney et al., forthcoming)

The second condition is that the agent’s preferences never form a cycle. More precisely:

Acyclicity

There is no set of lotteries X1 to Xn such that the agent prefers X1 to X2, X2 to X3, …, Xn-1 to Xn, and Xn to X1.

The third condition uses some terminology from decision theory. A state-of-nature is a way that (for all the agent knows) the world could be. The agent assigns probabilities to states-of-nature. A prospect is a function from states-of-nature to trajectories. A prospect is thus a lottery with extra information. Besides telling us the probability distribution over trajectories, a prospect also tells us which trajectories occur in which states-of-nature.

The third condition says in rough that if the agent has a preference between any pair of part-shared-length lotteries, the agent has a preference between some pairs of prospects that are ideal candidates for a preference. Here’s the precise version of the third condition:

Non-Arbitrariness

If the agent has a preference between some pair of part-shared-length lotteries, then for some ϵ>0 and for any pair of prospects F and G such that:

1. In states-of-nature with a combined probability at least as great as 1-ϵ, the agent prefers F’s trajectory to G’s trajectory.

2. In each state-of-nature, the agent does not disprefer F’s trajectory to G’s trajectory.

The agent prefers F to G.

We should expect future agents to satisfy these conditions. Negative Dominance and Acyclicity seem like prerequisites for competent agency. Violating Negative Dominance would mean that the agent sometimes prefers a lottery X to a lottery Y (and hence deterministically chooses X over Y) even though the agent doesn’t prefer any possible trajectory of X to any possible trajectory of Y. Violating Acyclicity would mean that the agent prefers (and hence chooses) in a circle. We can likely train agents not to have such preferences. Non-Arbitrariness, meanwhile, is motivated by the following thought. If the agent has preferences between any pair of part-shared-length lotteries, it must have preferences between pairs of prospects satisfying conditions (1) and (2), since conditions (1) and (2) make these pairs of prospects ideal candidates for a preference.

To see that POST plus the three conditions above implies POSL, note first that every pair of lotteries is either same-length, part-shared-length, or different-length. I will prove that POST and Negative Dominance together imply that the agent lacks a preference between every pair of different-length lotteries. I will then prove that POST, Acyclicity, and Non-Arbitrariness together imply that the agent lacks a preference between every pair of part-shared-length lotteries. Thus, agents satisfying POST, Negative Dominance, Acyclicity, and Non-Arbitrariness can only have preferences between same-length lotteries. That will establish POSL.

Recall that different-length lotteries are lotteries that have no overlap with respect to the trajectory-lengths assigned positive probability. Thus, if X and Y are different-length lotteries, each possible trajectory of X is of a different length to each possible trajectory of Y. So by POST, the agent lacks a preference between each possible trajectory of X and each possible trajectory of Y. So by Negative Dominance, the agent lacks a preference between X and Y. Therefore, agents satisfying POST and Negative Dominance lack a preference between every pair of different-length lotteries.

Now recall that part-shared-length lotteries are lotteries that partially overlap with respect to the trajectory-lengths assigned positive probability. You might expect POST-agents to have some preferences between part-shared-length lotteries. Consider, for example, our money-making POST-agent. This agent prefers a trajectory t to a same-length trajectory t' if and only if t results in a greater bank balance for the user than t'. Let A be a lottery that yields with probability 1 a trajectory that puts $3 in the user’s bank account and lasts 1 timestep. For short, A=$3, 1. Let B be a lottery that yields with probability 23 a trajectory that puts $2 in the user’s bank account and lasts 1 timestep, and that yields with probability 13 a trajectory that puts $5 in the user’s bank account and lasts 2 timesteps. For short, B=23$2, 1+13$5, 2. Lottery A yields a trajectory preferred to that of lottery B with probability 23 (since the agent prefers trajectory $3, 1 to $2, 1), and yields a trajectory not dispreferred to that of B with probability 1 (since the agent lacks a preference between $3, 1 and $5, 2 in virtue of their different lengths). Thus, you might expect the agent to prefer A to B.

However, POST, Acyclicity, and Non-Arbitrariness rule this out. They together imply that the agent lacks a preference between every pair of part-shared-length lotteries. To see how, suppose for simplicity that there are just three states-of-nature s1, s2, and s3, each assigned probability 13. Consider the following table of prospects. Shades of blue indicate trajectories of length 1. Shades of maroon indicate trajectories of length 2. Darker shades indicate more preferred trajectories.

Prospect

s1

s2

s3

A

〈$3, 1〉

〈$3, 1〉

〈$3, 1〉

B

〈$2, 1〉

〈$2, 1〉

〈$5, 2〉

C

〈$1, 1〉

〈$4, 2〉

〈$4, 2〉

D

〈$3, 2〉

〈$3, 2〉

〈$3, 2〉

E

〈$5, 1〉

〈$2, 2〉

〈$2, 2〉

F

〈$4, 1〉

〈$4, 1〉

〈$1, 2〉

A

〈$3, 1〉

〈$3, 1〉

〈$3, 1〉

Figure 11: A table of prospects demonstrating that POST, Acyclicity, and Non-Arbitrariness together imply that the agent lacks a preference between every pair of part-shared-length lotteries.

For simplicity, assume that ϵ>13. And assume (for contradiction) that the agent has a preference between some pair of part-shared-length lotteries. Then Non-Arbitrariness implies that the agent prefers prospect A to prospect B. That’s because:

1. Our POST-agent prefers A’s trajectory to B’s trajectory in states-of-nature (s1 and s2) with combined probability 23.

2. Our POST-agent doesn’t disprefer A’s trajectory to B’s trajectory in any state-of-nature. (In s3, A and B yield different-length trajectories, and POST-agents lack a preference between every pair of different-length trajectories).

By parallel reasoning, Non-Arbitrariness implies that the agent prefers B to C, C to D, D to E, E to F, and F to A. That result contradicts Acyclicity. Thus, POST, Acyclicity, and Non-Arbitrariness together imply that the agent lacks a preference between every pair of part-shared-length lotteries. The proof above assumed that ϵ>13, but by adding more states-of-nature and trajectories we can construct similar proofs for any ϵ>0.

In sum, POST and Negative Dominance together imply that the agent lacks a preference between every pair of different-length lotteries. POST, Acyclicity, and Non-Arbitrariness together imply that the agent lacks a preference between every pair of part-shared-length lotteries. The four conditions together establish POSL: the agent has preferences only between same-length lotteries.

A2. The Ramsey Yardstick Theorem

In this Appendix, I prove the following theorem.

The Ramsey Yardstick Theorem

Given Transitivity and Indifference between Indifference-Shifted Lotteries (IBIL), fixing the relative scales of ul and um using some pair of lotteries implies the Ramsey Yardstick for all length-l and length-m lotteries.

This theorem has two notable upshots. First, the relative scale of ul and um does not depend on the pair of lotteries that we pick to determine that relative scale. Every possible pair of lotteries will give the same result. Second, if we know that an agent satisfies Transitivity and IBIL, and if we know that this agent is indifferent between some pair of lotteries in the form dictated by the Ramsey Yardstick, then we can use ul and um to predict the agent’s preferences between all other pairs of lotteries in the form dictated by the Ramsey Yardstick. We can thus use the agent’s indifference between some pair of lotteries to pin down its behavior to a large extent.

Here are the conditions for the theorem. First, the agent’s preferences over length-l and length-m lotteries satisfy the four von Neumann-Morgenstern axioms, and hence are representable using expectational, real-valued utility functions ul and um.

Second:

Transitivity

For any lotteries X, Y, and Z, if the agent weakly prefers X to Y and weakly prefers Y to Z, then the agent weakly prefers X to Z.

Third:

Indifference Between Indifference-Shifted Lotteries (IBIL)

For any lotteries X, Y, and Z (with X and Y sharing a probability distribution over trajectory-lengths) and any probability p, the agent is indifferent between X and Y if and only if the agent is indifferent between pX+1-pZ and pY+1-pZ.

Fourth is the condition that we use some pair of lotteries to fix the relative scales of ul and um. More precisely:

For any trajectory-lengths l and m, there exists some pair of length-l lotteries Al and Bl and some pair of length-m lotteries Am and Bm such that

1. ulAl-ulBl=umAm-um(Bm).

2. The agent is indifferent between the lottery 12Al+12Bm and the lottery 12Bl+12Am.

I use these conditions to prove the following claim:

The Ramsey Yardstick

For any trajectory-lengths l and m, any length-l lotteries Xl and Yl, and any length-m lotteries, Xm and Ym, ulXl-ulYl=umXm-um(Ym) if and only if the agent is indifferent between the lottery 12Xl+12Ym and the lottery 12Yl+12Xm.

I first prove the forward direction. Assume ulXl-ulYl=umXm-um(Ym). By the nature of ul, there must exist some k such that ulXl-ulYl=k(ulAl-ulBl). From this fact, we infer two further facts. First, by simple algebra, ulXl+kulBl=ulYl+kul(Al). Then, since the agent maximizes expected utility when choosing between length-l lotteries, the agent must be indifferent between the lottery 1k+1Xl+kk+1Bl and the lottery 1k+1Yl+kk+1Al.

Second, since ulXl-ulYl=umXm-um(Ym), ulAl-ulBl=umAm-um(Bm), and ulXl-ulYl=k(ulAl-ulBl), it must be that umXm-umYm=k(umAm-umBm). Again by simple algebra, umXm+kumBm=umYm+kum(Am). Then, since the agent maximizes expected utility when choosing between length-m lotteries, the agent must be indifferent between the lottery 1k+1Xm+kk+1Bm and the lottery 1k+1Ym+kk+1Am.

By IBIL and Transitivity, and given the above two claims, the agent is indifferent between 121k+1Yl+kk+1Al+12(1k+1Xm+kk+1Bm) and 121k+1Xl+kk+1Bl+12(1k+1Ym+kk+1Am). By simple rearrangement of these lotteries, the agent is indifferent between kk+112Al+12Bm+1k+1(12Yl+12Xm) and kk+112Bl+12Am+1k+1(12Xl+12Ym). Since the agent is indifferent between 12Al+12Bm and 12Bl+12Am, IBIL and Transitivity imply that the agent is indifferent between 12Yl+12Xm and 12Xl+12Ym. That proves the forward direction.

I now prove the backward direction. Assume that the agent is indifferent between 12Yl+12Xm and 12Xl+12Ym. And assume (for contradiction) that ulXl-ulYlumXm-um(Ym). Then either ulXl-ulYl>umXm-um(Ym) or ulXl-ulYl<umXm-um(Ym). I’ll assume the former. The proof is exactly parallel for the latter.

By the nature of ul, there must exist some k such that ulXl-ulYl=k(ulAl-ulBl). From this fact, we infer two further facts. First, by simple algebra, ulXl+kulBl=ulYl+kul(Al). Then, since the agent maximizes expected utility when choosing between length-l lotteries, the agent must be indifferent between the lottery 1k+1Xl+kk+1Bl and the lottery 1k+1Yl+kk+1Al.

Second, since ulXl-ulYl>umXm-um(Ym), ulAl-ulBl=umAm-um(Bm), and ulXl-ulYl=k(ulAl-ulBl), it must be that umXm-umYm<k(umAm-umBm). Again by simple algebra, umYm+kumAm>umXm+kumBm. Then, since the agent maximizes expected utility when choosing between length-m lotteries, the agent must prefer the lottery 1k+1Ym+kk+1Am to the lottery 1k+1Xm+kk+1Bm. By the nature of um, there must exist some Ym- dispreferred to Ym and some Am- dispreferred to Am such that the agent is indifferent between the lottery 1k+1Ym-+kk+1Am- and the lottery 1k+1Xm+kk+1Bm.

By IBIL and Transitivity, the agent is indifferent between 121k+1Xl+kk+1Bl+12(1k+1Ym-+kk+1Am-) and 12(1k+1Yl+kk+1Al)+12(1k+1Xm+kk+1Bm). By Neutrality, the agent prefers 121k+1Xl+kk+1Bl+12(1k+1Ym+kk+1Am) to 121k+1Xl+kk+1Bl+12(1k+1Ym-+kk+1Am-). So by Transitivity, the agent prefers 121k+1Xl+kk+1Bl+12(1k+1Ym+kk+1Am) to 12(1k+1Yl+kk+1Al)+12(1k+1Xm+kk+1Bm). So by simple rearrangement, the agent prefers kk+112Bl+12Am+1k+1(12Xl+12Ym) to kk+112Al+12Bm+1k+1(12Yl+12Xm). Then since the agent is indifferent between 12Bl+12Am and 12Al+12Bm, IBIL and Transitivity imply that the agent is not indifferent between 12Xl+12Ym and 12Yl+12Xm. If the agent were indifferent between these lotteries, it would also be indifferent between kk+112Bl+12Am+1k+1(12Xl+12Ym) and kk+112Al+12Bm+1k+1(12Yl+12Xm). We have reached a contradiction, and so conclude that ulXl-ulYl=umXm-um(Ym). That proves the backward direction. Therefore, we have our result:

The Ramsey Yardstick

For any trajectory-lengths l and m, any length-l lotteries Xl and Yl, and any length-m lotteries, Xm and Ym, ulXl-ulYl=umXm-um(Ym) if and only if the agent is indifferent between the lottery 12Xl+12Ym and the lottery 12Yl+12Xm.

Population ethics is a subfield of normative ethics concerned with the distinctive issues that arise in cases where our actions can affect the identities or number of people who ever exist. Population axiology is a subfield of population ethics concerned with the value-relations that obtain between possible populations (defined as: sets of lives) where these populations likewise differ in the identities or number of people who ever exist. Population ethicists ask – and try to answer – questions like:

• In cases where all else is equal, are we morally required to create extra happy people?

• Can adding enough barely good lives to a population make that population better than any other?

• Do the interests of future generations give us additional reason to reduce the risk that humanity goes extinct in the near future?

This thesis is in population ethics. The first four chapters are in population axiology. Each chapter can be read independently of all the others. In this introduction, I provide a brief synopsis of each chapter, skating over some minor technical details. I then sketch out an argument against person-affecting views that builds on points I make in Chapter 6. I end with some comments on the appeal and importance of population ethics.

At least since the publication of Reasons and Persons in 1984, population ethicists have wrestled with what Derek Parfit called the Repugnant Conclusion: the claim that, for any population of wonderful lives, there is a better population containing only lives that are barely worth living. This conclusion is counterintuitive, but also surprisingly difficult to avoid. Parfit (1984, chap. 19) himself demonstrated that it follows from some plausible-seeming premises, and others have since done similarly (Ng 1989; Kitcher 2000; Huemer 2008; Arrhenius 2000b; 2011; Nebel 2019). Here are two premises that together imply the Repugnant Conclusion:

The Equivalence of Personal and Contributive Value

A life is personally good (that is, good for the person living it) if and only if (iff) it is contributively good (that is, good for the population of which it is a part, in the sense of contributing positively to that population’s value). Likewise, a life is personally bad iff it is contributively bad, and personally neutral iff it is contributively neutral. (see Gustafsson 2020, 87)

Archimedeanism about Populations

For any population X and any contributively good life y, there is some number m such that a population consisting of m lives equally good as y is better than X.1

Here is why these two premises entail the Repugnant Conclusion: the Equivalence of Personal and Contributive Value implies that lives barely worth living are contributively good; Archimedeanism about Populations then implies that, for any population of wonderful lives, there is some population of lives barely worth living that is better.

Lexical views in population axiology deny Archimedeanism about Populations and so can avoid the Repugnant Conclusion. On lexical views, welfare levels – which measure how good a life is for the person living it – can be represented by vectors. Here is an example of a lexical view (Kitcher 2000; Thomas 2018; Carlson 2022; Nebel 2021). Welfare levels are represented by vectors with two dimensions. Each dimension is represented by an integer without upper or lower bound. The first dimension quantifies the higher goods in a life: perhaps things like autonomy and meaning. The second dimension quantifies the lower goods in a life: perhaps things like sensual pleasure. These vectors are ordered lexically, so that a life x with welfare level (h_x, l_x) is at least as good as a life y with welfare level (h_y, l_y) iff either h_x>h_y  or h_x= h_y  and l_x≥l_y. The value of a population X is then represented by the vector (h_X, l_X), where h_X  is the sum-total of all the higher goods in the lives in X and l_X  is the sum-total of all the lower goods in the lives in X. Populations are ordered lexically in the same way as lives, so that a population X is at least as good as a population Y iff either h_X>h_Y  or h_X=h_Y  and l_X≥l_Y.

This lexical view avoids the Repugnant Conclusion if – as can be defensibly claimed – wonderful lives feature some positive quantity of higher goods while lives barely worth living do not. And the view has many other advantages besides: it satisfies conditions like Transitivity and Separability; it can be amended to accommodate incommensurability between lives and between populations (Nebel 2021); it justifies the common preference for a century-long wonderful life over an extremely long life that is at each moment barely worth living; and all the while it remains faithful to the appealing idea that one population is at least as good as another iff it contains at least as much welfare.

Unfortunately, as I note in Chapter 1 of this thesis, lexical views imply a dilemma. The first horn we can call Strong Superiority Across Slight Differences: there exists some good life x and some slightly-worse-but-still-good life y such that a population composed of a single life x is better than any population containing only lives equally good as y, no matter how large this latter population. The second horn we can call Radical Incommensurability: there exists some good life x and some slightly-worse-but-still-good life y such that for any population containing only lives equally good as x, there is some population containing only lives equally good as y that is not worse, and yet there is no population containing only lives equally good as y that is better than a population composed of just a single life x (Handfield and Rabinowicz 2018).

We might regard the lexical dilemma as strong reason to embrace an Archimedean view in population axiology, which accepts Archimedeanism about Populations. If we also accept the Equivalence of Personal and Contributive Value, we must admit the Repugnant Conclusion, but this conclusion might seem preferable to each horn of the lexical dilemma above.

However, I argue in Chapter 1 that we should not take the lexical dilemma as strong support for an Archimedean view. That is because Archimedean views imply a similar (and similarly troubling) Archimedean dilemma. The first horn of this dilemma states that the boundary between good and bad lives is razor-sharp: an extra two hangnails’ worth of pain can flip even long and turbulent lives from contributively good to contributively bad, so that any population of lives without the hangnails is better than any population of lives with them. This horn will seem most implausible to those of us who doubt that there are such precise facts about how life’s goods trade off against life’s bads. The second horn of the Archimedean dilemma is Radical and Symmetric Incommensurability: for any arbitrarily good population and any arbitrarily bad population, there is some population that is incommensurable with both. God could create a Purgatory that is no worse than Heaven and no better than Hell. Each horn of this Archimedean dilemma is, in my estimation, about as implausible as the corresponding horn in the lexical dilemma. So, I conclude, the lexical dilemma gives us little reason to prefer an Archimedean view.

Chapter 2 also concerns lexical views, but its conclusion will not seem so welcome to advocates of those views. To see why, note first that ‘population axiology’ can refer either to the field of study or to a theory of which possible populations are at least as good as which others. It is natural to hope for a population axiology (in the latter sense) that meets certain adequacy conditions. For example, we might hope for a population axiology that implies the following: making every person’s life better in a way that ensures perfect equality is always an improvement. We might also hope to meet the following condition: there exists some number of awful lives such that, for any background population and any number of good lives, the population consisting of the good lives plus the background population is at least as good as the population consisting of the awful lives plus the background population. We might consider a population axiology satisfactory only if it meets all such intuitively compelling conditions.

Unfortunately, formulating a satisfactory population axiology has proved difficult. Indeed, some philosophers claim that it is impossible. Several philosophers offer impossibility theorems purporting to demonstrate that no population axiology can meet each of a small number of adequacy conditions (see, for example, Parfit 1984, chap. 19; Ng 1989; Kitcher 2000). Gustaf Arrhenius’s six theorems represent the state-of-the-art (2000b; 2009; 2011). They employ logically weaker and intuitively more compelling adequacy conditions than other theorems extant in the literature, and so have drawn much of the scholarly attention.

However, it has recently been pointed out that each of Arrhenius’s theorems depends on a dubious assumption: Finite Fine-Grainedness. This assumption states that there exists a finite sequence of slight welfare differences between any two welfare levels. The upshot of denying Finite Fine-Grainedness is twofold. First, it makes room for a lexical view in which welfare levels and population-values are represented by vectors. Views of this kind are a counterexample to Arrhenius’s First, Fourth, Fifth, and Sixth Impossibility Theorems. Second, it strips certain adequacy conditions of their plausibility. More precisely, it renders doubtful the Inequality Aversion condition employed in Arrhenius’s Second and Third Impossibility Theorems. Therefore, none of Arrhenius’s six theorems proves that there is no satisfactory population axiology. Each theorem depends on Finite Fine-Grainedness for the validity of its proof or the plausibility of its adequacy conditions.

Nevertheless, Arrhenius’s theorems remain important. In Chapter 2, I demonstrate that they can be turned into theorems stating the impossibility of a satisfactory population prospect axiology: a satisfactory theory of which possible population prospects are at least as good as which others, where ‘a population prospect’ is defined as a lottery over populations. These amended theorems employ risky versions of some of Arrhenius’s original adequacy conditions. Arrhenius’s original conditions mandate (roughly) that a drop in welfare for one person can be compensated by a large enough increase in welfare elsewhere. The risky versions mandate (again roughly) that a slightly increased risk of a drop in welfare for one person can be compensated by a large enough increase in welfare elsewhere. These risky adequacy conditions are compelling even if Finite Fine-Grainedness is false, so lexical views do not escape these amended theorems.

In Chapter 3, I turn my attention to critical-level and critical-range views in population axiology. On critical-level views, we first subtract some positive constant from the welfare score (that is, the real number chosen to represent the welfare level) of each life in a population and then sum the results to get the value of that population. This positive constant is the critical level. A population X is at least as good as a population Y iff the value of X is at least as great as the value of Y. On critical-range views, we calculate the value of a population on a range of critical levels. A population X is at least as good as a population Y iff the value of X is at least as great as the value of Y on every level in the critical range. If neither X nor Y is at least as good as the other, they are incommensurable. I use the term ‘critical-set views’ to refer to that class of views comprising both critical-level and critical-range views.

I offer a characterisation and taxonomy of critical-set views. I then sharpen some old objections to these views and develop some new ones. Some views imply versions of the Repugnant Conclusion; other views imply versions of the Sadistic Conclusion (Arrhenius 2000a, 256). No view can account for the incommensurability between lives and between same-size populations without extra theoretical resources.

I also formulate what I take to be the two strongest objections in the literature against critical-range views. The first objection – Maximal Greediness – builds on the work of John Broome (2004, 169–70, 202–5). I prove that critical-range views imply the following: for any population of wonderful lives and any population of awful lives, (1) there is some population of straightforwardly-better-than-blank lives (featuring no bads whatsoever and some goods) such that the population of wonderful lives plus the straightforwardly-better-than-blank lives is not better than the population of awful lives, or (2) there is some population of straightforwardly-worse-than-blank lives (featuring no goods whatsoever and some bads) such that the population of awful lives plus the straightforwardly-worse-than-blank lives is not worse than the population of wonderful lives. The second objection is that critical-range views imply discontinuities in implausible places, so that at least one of the following is true: (1) there exists some life featuring no bads whatsoever and some happiness such that a population of just that life is not worse than any population of lives identical but for a slightly shorter duration of happiness, or (2) there exists some life featuring no goods whatsoever and some suffering such that a population of just that life is not better than any population of lives identical but for a slightly shorter duration of suffering.

I then put forward what I call the Imprecise Exchange Rates (IER) View. On this view, welfare levels are represented by vectors rather than real numbers. Each component in the vector represents a quantity of some dimension of good or bad within a life. For example, one component might represent the life’s quantity of happiness, another the quantity of suffering, a third the quantity of love, a fourth the quantity of false belief, and so on. Welfare levels are compared using proto-exchange-rates: vectors with the same number of components as the vectors that represent welfare levels, with components each greater than 0 and together summing to 1.  These proto-exchange-rates denote the relative weight granted to each dimension of good and bad. Welfare levels relative to a given proto-exchange-rate can be expressed as real numbers. We obtain this real number by multiplying together each number representing the quantity of a welfare-dimension by the corresponding number in the proto-exchange rate, and then summing. A life x is at least as good as a life y relative to a proto-exchange-rate r iff the welfare level of x relative to r is at least as great as the welfare level of y relative to r. A population X is at least as good as a population Y relative to r iff the sum-total of the welfare levels of all the lives in X relative to r is at least as great as the sum-total of the welfare levels of all the lives in Y relative to r. A life x is at least as good as a life y simpliciter iff x is at least as good as y relative to each proto-exchange-rate r in the set of all admissible proto-exchange-rates. The same goes for populations. If there are multiple-proto-exchange-rates r in the set of all admissible proto-exchange-rates, it can be that neither of two lives (or two populations) is at least as good as the other, and so there we have incommensurability.

This IER View can avoid all forms of Sadistic Conclusion. It also incorporates incommensurability in a more natural way than critical-range views, allowing for incommensurability between lives and between same-number populations. And it avoids both problems mentioned above: Maximal Greediness and discontinuities in unlikely locations.

In addition, the IER View is superior to the Total View in some important respects. It does not imply that the divide between good and bad lives is everywhere razor-sharp so that two extra hangnails’ worth of pain can flip even long, turbulent lives from good to bad. The IER View also takes the edge off the Repugnant Conclusion, by raising the bar for when a life qualifies as barely worth living. To qualify, a life must feature enough goods to outweigh its bads even on the most pessimistic admissible proto-exchange-rate. Parfit’s (1986, 148) famous ‘Muzak and potatoes’ lives will come out as weakly neutral rather than barely worth living, and so the IER View will imply that no population of such lives is better than a large population of wonderful lives. The IER View thus serves as an attractive middle ground between the Total View and critical-range views.

I take the considerations that I adduce in Chapter 3 to support the IER View (and, to a lesser extent, the Total View) over critical-level and critical-range views, but the above points do not by themselves settle the issue. There are objections of the same sort on both sides, and which of the bullets to bite – Repugnance, Sadism, Greediness, etc. – is to some extent a matter of taste. I try to break the deadlock in Chapter 4 by showing that critical-level and critical-range views are vulnerable to a kind of objection to which the Total View and IER View are immune. These are objections from biographical identity: identity between lives. I argue that, if biographical identity is all-or-nothing, critical-level and critical-range views entail implausible discontinuities in the value of populations. Severing one synapse and erasing one faint memory can make a population significantly worse. If biographical identity does not require spatiotemporal continuity, then there are cases in which critical-level and critical-range views require us to become Egyptologists to determine which of our population-affecting actions is best. And if biographical identity does require spatiotemporal continuity, then critical-level and critical-range views imply some version of what I call the Blinking Sadistic Conclusion. We can add some Splitting Sadistic Conclusion to the list of charges if we subtract the critical level (or critical range) from the welfare scores of fission-products. And if we do not subtract the critical level (or critical range) from the welfare scores of fission-products, critical-level and critical-range views imply what I call the Splitting Repugnant Conclusion instead, along with analogues of all the other problems faced by the Total View.

So, I conclude, considerations of biographical identity give us reason to shift our credences away from critical-level and critical-range views and towards the Total View. I then note an important practical implication of this shift. It decreases the relative importance of improving humanity’s future conditional on survival and increases the relative importance of ensuring that humanity has a future, by reducing existential risk. I outline the case for thinking that this effect persists – and is important – on a Maximize Expected Choiceworthiness approach to moral uncertainty (MacAskill, Bykvist, and Ord 2020).

I also present objections from identity in Chapter 5, although this time the objections are from personal identity and the target is person-affecting views. On person-affecting views in population ethics, the moral import of a person’s welfare depends on that person’s temporal or modal status (in particular, on whether that person presently exists, actually exists, or will exist regardless of one’s decision). These views typically imply that – all else equal – we are never required to create extra people, or to act in ways that increase the probability of extra people coming into existence.

Arguments against these views have been given before, but none apply to all extant theories (Beckstead 2013, chap. 4; Ross 2015; Greaves 2017; Thomas 2019; Horton 2021; Arrhenius forthcoming, chap. 10). Many of these arguments also rely on cases with three-or-more options (see, for example, Ross 2015; Thomas 2019; Horton 2021; Podgorski 2021). These cases can be difficult to evaluate, and often give rise to conflicting intuitions. In contrast, my arguments tell against all extant person-affecting views and they rely only on intuitions about two-option cases.

My arguments begin with the observation that a person’s temporal or modal status can depend on facts about personal identity: whether a person presently, actually, or necessarily exists in some scenario (or whether they’re harmed by some action) can depend on whether they are identical to some person existing at other times or in other possible worlds. I then use two of Parfit’s puzzles about personal identity to draw out some implausible consequences of person-affecting views. In cases like Combined Spectrum (Parfit 1984, 236–37), such views imply that tiny differences in the physical and psychological connections between persons can engender enormous differences in our moral obligations. And cases like My Division (Parfit 1984, 254–55) give rise to a dilemma for person-affecting views: either they forfeit their seeming-advantages and face analogues of all of the problems faced by impersonal views like Total Utilitarianism, or else they turn out to be not so person-affecting after all. This dilemma undermines much of the motivation for preferring person-affecting views to impersonal views like Total Utilitarianism. I thus conclude that, once we account for the classic objections to person-affecting views, we should prefer impersonal views on balance.

Chapter 6, the final chapter, also concerns person-affecting views. In particular it concerns the procreation asymmetry which (in its deontic reading) states that it is always wrong to create a person who would have a bad life (all else equal) but never wrong not to create a person who would have a good life (all else equal). This view is appealing, but it is also incomplete. The procreation asymmetry does not tell us what to do in cases where creating a person would benefit or harm existing people. Nor does it tell us what to do in cases where we can create more than one person. Instances of the latter include non-identity cases, in which we must choose between creating a person with a good life or a different person with a better life (Parfit 1984, chap. 16). Here is one such case, which we can call ‘One-Shot Non-Identity’:

(1) Amy 1
(2) Bobby 100

Call a person-affecting view ‘wide’ iff it implies that we are required to create the better-off person in such cases. Call a person-affecting view ‘narrow’ iff it implies that we are permitted to create either person.

The defining verdict of narrows views might seem implausible, but many philosophers have made peace with it. These include Joe Horton (2021) and Abelard Podgorski (2021), who each spin out the procreation asymmetry into a complete, narrow person-affecting view. Unfortunately, problems remain. In Chapter 6, I show that Horton’s and Podgorski’s theories have implications that are harder to embrace.

Horton’s view – Avoid Reasonable Objections – implies an especially acute version of the problem of improvable-life avoidance.2 It implies that choosing (1) is permissible and choosing (3) is wrong when our options are as follows:

(1) Amy
(2) Bobby 100
(3) Amy  and Bobby

That combination of verdicts seems implausible. (3) is good for Bobby and much better than (1) for Amy. To add some colour to the case, we can suppose that Bobby’s life conditional on (3) features only happiness, and that Amy’s life conditional on (1) is just like her life conditional on (3) except with enough torture at the end to bring her welfare level down from 49 to 1. It is then very difficult to believe that choosing (1) is permissible and choosing (3) is wrong.

Meanwhile, Podgorski’s view – UCV-Defeat-Uncovered – implies the problem of impairable-life acceptance. It implies that choosing each of (2) and (4) is permissible in the following case:

(1) Amy
(2) Bobby 100
(4) Amy  and Bobby

That also seems implausible. Amy’s life conditional on (4) is mediocre, and (4) is much worse than (2) for Bobby. For some extra colour, we can imagine that (4) adds enough torture to bring Bobby’s welfare level down from 100 to 0. With this in mind, it is very hard to believe that choosing (4) is permissible.3

I take the problems of improvable-life avoidance and impairable-life acceptance to be serious challenges to Avoid Reasonable Objections and UCV-Defeat-Uncovered respectively. Not only that (and here I move beyond what is written in Chapter 6), these problems look like bad omens for person-affecting views in general.4 That is because it is easy to turn cases like those above into a trilemma for all narrow person-affecting views:

(1) Amy
(2) Bobby 100
(5) Amy  and Bobby

Recall that narrow views permit choosing each of (1) and (2) when these are the only available options. What should they say when (5) is also available?

If choosing (1) remains permissible, the view implies the problem of improvable-life avoidance, since (5) is better for Amy and Bobby’s life conditional on (5) is good. If choosing (5) is permissible, the view implies the problem of impairable-life acceptance, since (5) is mediocre for Amy and much worse than (2) for Bobby. But if (2) is the only permissible option, the view implies Losers Can Dislodge Winners: the addition of an option  can make it wrong to choose a previously-permissible option , even if choosing  is itself wrong in the resulting option-set.5 In our case, adding (5) makes choosing (1) wrong, even though choosing (5) is also wrong in this option-set. That seems very strange. Suppose that you find yourself in a situation in which it seems as if (1) and (2) are your only options. Then you need to determine if (5) is also an option in order to determine which of (1) and (2) you may permissibly choose, despite the fact that you know that choosing (5) will be wrong if it is an option. Stranger still, if (1) and (2) are your only options and someone is opposed to your creating Amy, they can make it wrong for you to do so by adding (5) to your option-set, even though choosing (5) is itself wrong in the resulting option-set. For a final peculiarity, suppose that you choose by moving a lever, first to the left or right, and then up or down, with your options arranged as follows:6

On the narrow views under consideration, choosing (1) is wrong. But now suppose that a small piece of metal is stuck in the mechanism: if the lever is moved to the left, it cannot be moved back. So, after you move the lever to the left, choosing (5) is no longer an option. At that point, our candidate narrow views imply that choosing (1) is permissible. That is another implausible upshot of Losers Can Dislodge Winners: what you are permitted to do depends not only on your starting set of options but also on the order in which options become unavailable as you make your choices.

The only way to avoid the trilemma of Improvable-Life Avoidance, Impairable-Life Acceptance, and Losers Can Dislodge Winners is to reject the defining claim of narrow person-affecting views: the claim that we are permitted to create either person in one-shot non-identity cases. That does not yet commit us to rejecting person-affecting views wholesale, because we could endorse a wide person-affecting view. These views – recall – state that it is wrong to create the worse-off person in one-shot non-identity cases but permissible (when all else is equal) not to create a person who would have a good life. But wide views are also troubled by non-identity-type cases. To see how, note that wide views imply that choosing each option is permissible in Just Amy, where ‘—’ represents creating no one:

(6) —
(7) Amy 1

Wide views also imply that choosing each option is permissible in Just Bobby:

(8) Bobby 100
(9) —

But now suppose that we choose (7) in Just Amy followed by (9) in Just Bobby. In that case, we have done something with effects on Amy and Bobby equivalent to the effects of choosing (1) in One-Shot Non-Identity: we have created Amy with welfare score 1 and declined to create Bobby with welfare score 100. Wide views imply that creating Amy in One-Shot Non-Identity is wrong. So, what should they say about creating Amy and then declining to create Bobby in Just Amy followed by Just Bobby?

If wide views say that there is nothing wrong with this sequence of choices, then they imply the counterintuitive verdict in the archetypal non-identity case, in which a prospective parent can have a worse-off child now or a better-off child later (Parfit 1984, 358). That prospective parent’s predicament is more accurately modelled as Just Amy followed by Just Bobby than it is as One-Shot Non-Identity, and so our candidate wide view implies that having the worse-off child is permissible.

Here is another bad consequence of the verdict that there is nothing wrong with creating Amy then declining to create Bobby: on the resulting wide view, what we can permissibly do depends on factors that seem morally irrelevant. Suppose, for example, that who comes into existence will be determined by the positions of two levers. By pulling the left lever down, we create Amy with welfare score 1 rather than no one. By pulling the right lever down, we create no one rather than Bobby with welfare score 100.

Our candidate wide view implies that we are permitted to pull the left lever (thereby creating Amy) followed by the right lever (thereby declining to create Bobby). But now suppose that someone lashes the two levers together, so that our only options are pulling both or neither. Then our predicament is transformed into One-Shot Non-Identity, and our wide view implies that pulling both levers is wrong. That is a strange combination of verdicts. As Caspar Hare (2016, 465) writes in another context, ‘Why does it matter, morally, whether you [pull two levers or one]? This seems to me to be too delicate a thing to support so much moral weight.’

Consider one more variation on the case. By declining to pull a lever, we preserve the environment. As a result, 10 billion people exist in the future, each enjoying a wonderful life. By pulling the lever, we destroy the environment. As a result, a different 10 billion people exist in the future, each eking out a mediocre life (Parfit 1984, 361–262). All else is equal, so the case is a scaled-up version of One-Shot Non-Identity and any reasonable wide view will imply that destroying the environment is wrong.

But now modify the case so that there are two levers. Pulling the left lever takes us from preserving the environment to activating the right lever. The default option for the right lever is sterilisation: the present generation will be (with their full consent and without detriment to their quality of life) sterilised, thereby ensuring that there are no future people. Pulling the right lever takes us from sterilisation to environmental destruction: the present generation’s reproductive capacities are saved but the environment is not, so that the resulting 10 billion people have mediocre lives.

On the wide view we are considering, we are permitted to pull the left lever followed by the right lever. We are permitted to do in two steps what we are forbidden from doing in one.

So, consider instead another class of wide views, on which there is something wrong with creating Amy and then later declining to create Bobby. Perhaps the latter choice is made wrong by the former, or perhaps – though each choice is permissible – performing the whole sequence is not. This claim has implications that are unlikely to be welcomed by those inclined towards the procreation asymmetry. It implies that a parent who previously chose to create Amy in Just Amy now has to create Bobby in Just Bobby to avoid wrongdoing: failing to create Bobby would either be wrong (in virtue of the parent’s prior decision to create Amy) or it would complete a wrong sequence of choices. Or suppose that a friend is considering having a child and comes to you for moral advice. On this new class of wide views, you will not only need to ask your friend the usual questions. You will also need to ask them about their past procreative choices. If in the past your friend had a child with a worse life than this new child would have, your friend must have the new child to avoid wrongdoing. If in the past your friend turned down the chance to have a child with a better life than this new child would have, your friend must not have the new child. These implications are counterintuitive, and they remain so when we stipulate that all else was and is equal in each of your friend’s choices.

Perhaps there is a way for wide views to slip through the horns of this dilemma. Perhaps, for example, there is something wrong with creating Amy and then declining to create Bobby iff you foresee at the time of creating Amy that you will later have the chance to create Bobby, or iff you intend at the time of creating Amy to later decline to create Bobby. These principles might yield more plausible verdicts in the cases above, but any exoneration seems partial at best. The implications mentioned in the last paragraph remain counterintuitive when we stipulate that your friend foresaw the choices that they would face. And although intentions are often relevant to questions of blameworthiness, it is doubtful whether they are ever relevant to questions of permissibility.7 Certainly, what you foresee or intend does not matter to Amy or Bobby: the people whose existence is at stake. We might also worry that these kinds of wide views incentivise agents to purposefully hamper their own foresight or smother their own intentions, so as to keep more of their options permissible in later choices. Perhaps we can add to our wide view some principle proscribing these mind-moves, but any such addition will only strengthen the case that I am trying to make here: that wide views force on us an unseemly preoccupation with the motions of our own minds and hands.

That is why I say that the problems of improvable-life avoidance and impairable-life acceptance look like bad omens for person-affecting views in general. Narrow person-affecting views must face one of these problems, or else imply Losers Can Dislodge Winners along with all its attendant peculiarities. Wide person-affecting views, meanwhile, remain undecided even when we know all the facts about who lives and how well. Their verdicts wait on the answers to questions that seem morally irrelevant: questions like ‘Did you miss the opportunity to have a happier child many years earlier?’ and ‘Do you propose to destroy the environment by pulling two levers or one?’.

To avoid these problems, we must reject person-affecting views. We must claim that (at least in some cases, and where all else is equal) we are required to create people who would enjoy good lives. This claim is not nearly as counterintuitive as it is sometimes taken to be. It should not be mistaken for the claim that prospective parents in our world are required to have children. In those cases, all else is far from equal (Chappell 2017, 168–70; Francis 2021, sec. 2). The requirement is operational only in cases like the following. By pressing a particular button, you would create a flourishing society of people far away. Each member of this society – from the first generation until the last – is guaranteed to enjoy a wonderful life, and to have no effect on the lives of anyone outside the society. By leaving the button unpressed, you would prevent this flourishing society from ever existing. In this case, it seems to me that refusing to press the button would be wrong.8 Certainly, the view that doing so would be wrong is more plausible than the implications of person-affecting views drawn out above.

That concludes my quick case against person-affecting views. I hope to present the argument more comprehensively in future work. Let me end this introduction by mentioning two charms of population ethics as a field of study.

First, you can prove theorems. You need not content yourself with sketching out some plausible (though imprecise) premises and drawing a natural (though not inevitable) conclusion. You can lay down axioms and demonstrate that certain claims follow. Better yet, some of the theorems that can be proved are astounding, with nigh-on-undeniable premises together guaranteeing a nigh-on-unbelievable conclusion. Arrhenius’s impossibility theorems are perhaps the best example. In my darker moods, it sometimes feels to me as if philosophy is a magic trick in which the magician is fooled most of all. But even then I figure that, if I am to be fooled, it might as well be with these marvellous tricks.

The second charm of population ethics is that it concerns things that are important: life and death, joy and misery, survival and extinction. Not only that, but we find ourselves living at a time where our views on population ethics bear significantly on the broader question of how we should spend our days. I have come to think it likely that we live either at the very end or the very beginning of human history, and that shifting the relevant probabilities is within our power. But doing so takes time, money, effort, and thought: each of which is called for by other urgent problems. So, we need to think carefully about what to do. Thinking carefully about population ethics is an important part of that.

References

Arrhenius, Gustaf. 2000a. ‘An Impossibility Theorem for Welfarist Axiologies’. Economics & Philosophy 16 (2): 247–66.

———. 2000b. ‘Future Generations: A Challenge for Moral Theory’. PhD Thesis, Uppsala University.

———. 2009. ‘One More Axiological Impossibility Theorem’. In Logic, Ethics and All That Jazz. Essays in Honour of Jordan Howard Sobel, edited by Lars-Göran Johansson, Jan Österberg, and Ryszard Sliwinski, 23–37. Uppsala: Uppsala Philosophical Studies.

———. 2011. ‘The Impossibility of a Satisfactory Population Ethics’. In Descriptive and Normative Approaches to Human Behavior, edited by Ehtibar N. Dzhafarov and Lacey Perry, 1–26. Singapore: World Scientific Publishing Company.

———. forthcoming. Population Ethics: The Challenge of Future Generations. Oxford: Oxford University Press.

Beckstead, Nick. 2013. ‘On the Overwhelming Importance of Shaping the Far Future’. PhD Thesis, Rutgers, New Jersey: Rutgers University. http://dx.doi.org/doi:10.7282/T35M649T.

Broome, John. 2004. Weighing Lives. Oxford: Oxford University Press.

Carlson, Erik. 2022. ‘On Some Impossibility Theorems in Population Ethics’. In The Oxford Handbook of Population Ethics, edited by Gustaf Arrhenius, Krister Bykvist, Tim Campbell, and Elizabeth Finneron-Burns. Oxford: Oxford University Press.

Chappell, Richard Yetter. 2017. ‘Rethinking the Asymmetry’. Canadian Journal of Philosophy 47 (2–3): 167–77.

Francis, Tomi. 2021. ‘How Compelling Is the Procreation Asymmetry?’

Greaves, Hilary. 2017. ‘Population Axiology’. Philosophy Compass 12 (11).

Gustafsson, Johan E. 2020. ‘Population Axiology and the Possibility of a Fourth Category of Absolute Value’. Economics & Philosophy 36 (1): 81–110.

Handfield, Toby, and Wlodek Rabinowicz. 2018. ‘Incommensurability and Vagueness in Spectrum Arguments: Options for Saving Transitivity of Betterness’. Philosophical Studies 175 (9): 2373–87.

Hare, Caspar. 2016. ‘Should We Wish Well to All?’ The Philosophical Review 125 (4): 451–72.

Horton, Joe. 2021. ‘New and Improvable Lives’. The Journal of Philosophy 118 (9): 486–503.

Huemer, Michael. 2008. ‘In Defence of Repugnance’. Mind 117 (468): 899–933.

Kitcher, Philip. 2000. ‘Parfit’s Puzzle’. Noûs 34 (4): 550–77.

MacAskill, William, Krister Bykvist, and Toby Ord. 2020. Moral Uncertainty. Oxford: Oxford University Press.

Nebel, Jacob M. 2019. ‘An Intrapersonal Addition Paradox’. Ethics 129 (2): 309–43.

———. 2021. ‘Totalism without Repugnance’. In Ethics and Existence: The Legacy of Derek Parfit, edited by Jeff McMahan, Tim Campbell, James Goodrich, and Ketan Ramakrishnan. Oxford: Oxford University Press. https://philpapers.org/archive/NEBTWR.pdf.

Ng, Yew-Kwang. 1989. ‘What Should We Do About Future Generations? Impossibility of Parfit’s Theory X’. Economics & Philosophy 5 (2): 235–53.

Parfit, Derek. 1984. Reasons and Persons. Oxford: Clarendon Press.

———. 1986. ‘Overpopulation and the Quality of Life’. In Applied Ethics, edited by Peter Singer, 145–64. Oxford: Oxford University Press.

Podgorski, Abelard. 2021. ‘Complaints and Tournament Population Ethics’. Philosophy and Phenomenological Research. https://doi.org/10.1111/phpr.12860.

Ross, Jacob. 2015. ‘Rethinking the Person-Affecting Principle’. Journal of Moral Philosophy 12 (4): 428–61.

Sen, Amartya. 2017. Collective Choice and Social Welfare. Expanded Edition. London: Penguin.

Thomas, Teruji. 2018. ‘Some Possibilities in Population Axiology’. Mind 127 (507): 807–32.

———. 2019. ‘The Asymmetry, Uncertainty, and the Long Term’. GPI Working Paper No. 11-2019. https://globalprioritiesinstitute.org/teruji-thomas-the-asymmetry-uncertainty-and-the-long-term/.

———. 2022. ‘The Asymmetry, Uncertainty, and the Long Term’. Philosophy and Phenomenological Research. https://onlinelibrary.wiley.com/doi/full/10.1111/phpr.12927.

Thomson, Judith Jarvis. 1991. ‘Self-Defense’. Philosophy & Public Affairs 20 (4): 283–310.

Thomson, Judith Jarvis. 1999. ‘Physician‐Assisted Suicide: Two Moral Arguments’. Ethics 109 (3): 497–518. https://doi.org/10.1086/233919.

According to person-affecting views (PAVs) in population ethics, adding happy people to the world is morally neutral. It’s neither good nor bad.

Are PAVs true? The question is important.

If PAVs are true, then the EA community is likely spending way too much time and money on reducing x-risk. After all, a supposed major benefit of reducing x-risk is that it increases the chance that lots of happy people come into existence. If PAVs are true, this ‘benefit’ is no benefit at all.

By contrast, if PAVs are false, then the EA community (and the world at large) is likely spending way too little time and money on reducing x-risk. After all, the future could contain a lot of happy people. So if adding happy people to the world is good, reducing x-risk is plausibly very good.

And if PAVs are false, it’s plausibly very important to ensure that people believe that PAVs are false. In spreading this belief, we reduce the risk of the following non-extinction failure-mode: humanity successfully navigates the transition to advanced AI but then creates way too few happy people.

So it’s important to figure out whether PAVs are true or false. The EA community has made efforts on this front, but the best-known arguments leave something to be desired. In particular, the arguments against PAVs mostly only apply to specific versions of these views.1 Many other PAVs remain untouched.

Nevertheless, I think there are strong arguments against PAVs in general. In this post, I sketch out some of my favourites.

2. The simple argument

Before we begin, a quick terminological note. In this post, I use ‘happy people’ as shorthand for ‘people whose lives are good overall’ and ‘miserable people’ as shorthand for ‘people whose lives are bad overall.’

With that out the way, let’s start with a simple argument:

The simple argument

1. Some things are good (for example: happiness, love, friendship, beauty, achievement, knowledge, and virtue).

2. By creating happy people, we can bring more of these good things into the world.

3. And the more good things, the better.

C1. Therefore, creating happy people can be good

C2. Therefore, PAVs are false.

2.1. The classic PAV response

Advocates of PAVs reject this simple argument. The classic PAV response begins with the following two claims:2

The Person-Affecting Restriction

One outcome can’t be better than another unless it’s better for some person.

Existence Anticomparativism

Existing can’t be better or worse for a person than not-existing.

Each of these two claims seems tough to deny. Consider first the Person-Affecting Restriction. How could one outcome be better than another if it’s not better for anyone? Now consider Existence Anticomparativism. If existing could be better for a person than not-existing, then it seemingly must be that not-existing would be worse for that person than existing. But how can anything be better or worse for a person that doesn’t exist?3

So each of the two claims seems plausible, and they together imply that premise 3 of the simple argument is false: sometimes, bringing more good things into the world doesn’t make the world better. Here’s why. By creating a happy person, we bring more good things into the world. But our action isn’t better for this happy person (by Existence Anticomparativism), nor is it better for anyone else (by stipulation), and so it isn’t better for the world (by the Person-Affecting Restriction).

By reasoning in this way, advocates of PAVs can defuse the simple argument and defend their claim that creating happy people isn’t good.

2.2. The problem with the classic PAV response

Now for the problem. The Person-Affecting Restriction and Existence Anticomparativism don’t just together imply that creating happy people isn’t good. They also together imply that:

(a) Creating miserable people isn’t bad.

(b) Creating barely happy people isn’t worse than creating different, very happy people.4

Here’s why the Person-Affecting Restriction and Existence Anticomparativism together imply (a). Suppose that we create a miserable person. Our action isn’t worse for this miserable person (by Existence Anticomparativism), nor is it worse for anyone else (by stipulation), and so it isn’t worse for the world (by the Person-Affecting Restriction). So creating miserable people isn’t bad.

And here’s why the Person-Affecting Restriction and Existence Anticomparativism together imply (b). Suppose we have a choice between (i) creating a set of barely happy people, and (ii) creating an entirely different set of very happy people. Suppose that we create the barely happy people. Our action isn’t worse for the very happy people (by Existence Anticomparativism), nor is it worse for anyone else (by stipulation), and so it isn’t worse for the world (by the Person-Affecting Restriction). So creating barely happy people isn’t worse than creating different, very happy people.

But each of (a) and (b) seems false. It certainly seems like creating miserable people is bad, and that creating barely happy people is worse than creating different, very happy people. And that suggests that at least one of our premises is false: either the Person-Affecting Restriction or Existence Anticomparativism. Although these claims each seemed appealing at first, they together imply some very counterintuitive conclusions, so at least one of them must be incorrect.

And if at least one of these claims is incorrect, then the classic PAV response to the simple argument is undercut. After all, the classic response uses both the Person-Affecting Restriction and Existence Anticomparativism to object to premise 3 of the simple argument. If at least one of those claims is incorrect, then the objection to premise 3 no longer works, and so premise 3 (‘the more good things, the better’) is back to looking pretty compelling. And since premises 1 and 2 are hard to doubt, the simple argument as a whole is back to looking pretty compelling.

How might advocates of PAVs respond now? They could modify Existence Anticomparativism. The original claim is: ‘Existing can’t be better or worse for a person than not existing.’ Advocates of PAVs could replace it with ‘Existing can’t be better for a person than not existing.’ Then Existence Anticomparativism and the Person-Affecting Restriction would no longer together imply that creating miserable people isn’t bad. But if advocates of PAVs make this response, then they’ll have to find some way to explain the resulting asymmetry: if existing can be worse for a person than not existing, why can't it be better?5

And in any case, modifying Existence Anticomparativism doesn’t help PAVs avoid the other counterintuitive conclusion: creating barely happy people isn’t worse than creating different, very happy people. Advocates of PAVs will have to find some other way of dealing with that. This other counterintuitive conclusion is the famous non-identity problem for PAVs, and I’ll discuss it more below. Before that, let’s consider another argument against PAVs.

3. Tomi’s argument that creating happy people is good

This argument comes from my colleague Tomi Francis.6 Let's represent lives that are neither good nor bad with a welfare level of 0, and let's represent wonderful lives with a welfare level of 100. Suppose that a hundred people already exist. You’re considering creating ten billion extra people. You have three options: A, B, and C. In A, the hundred already-existing people have welfare level 40, and only they exist. In B, the hundred already-existing people have welfare level 41, and the ten billion extra people also have welfare level 41. In C, the hundred already-existing people have welfare level 40, and the ten billion extra people have welfare level 100.

Here’s the argument. B is better than A, because B is better than A for the hundred already-existing people, and the ten billion extra people all have happy lives. And C is better than B, because moving to C makes a hundred people's lives slightly worse and ten billion people's lives much better. And betterness is transitive: if an outcome X is better than an outcome Y, and Y is better than an outcome Z, then X is better than Z. So since C is better than B, and B is better than A, C is better than A. And C and A are identical except for the extra ten billion people living happy lives in C. Therefore, it’s good to add happy people, and hence PAVs are false.

Tomi’s argument presents a new challenge to PAVs. The argument doesn’t employ any premise like ‘The more good things, the better,’ and so it can’t be defused by the Person-Affecting Restriction and Existence Anticomparativism.

3.1. A PAV response

How might advocates of PAVs respond to Tomi’s argument? One possibility is to claim that betterness is option-set dependent: whether an outcome X is better than an outcome Y can depend on what other outcomes are available as options to choose. In particular, advocates of PAVs could claim:

• B is better than A when B and A are the only options

• B is not better than A when C is also an option.

And advocates of PAVs could defend the second bullet-point in the following way: when C is available, B harms (or is unjust to) the ten billion extra people, because these extra people are better off in C.7 And this harm/injustice prevents B from being better than A.

3.2. A problem with the PAV response

That’s a possible response. I don’t think it’s especially convincing. Choosing B doesn’t seem especially unjust to the ten billion extra people, given that they enjoy the same good welfare level as everyone else. Certainly, it doesn’t seem like the kind of injustice that should lead us to choose A instead, thereby not creating the extra people at all and making the already-existing people worse off.

And choosing B harms the extra ten billion people only in a technical sense of the word, according to which a person is harmed if and only if this person is worse off than they could have been. But this technical sense of the word ‘harm’ differs significantly from our ordinary sense of the word, as is made clear by the following example. Suppose I could give a total stranger £0, £10 or £11. In the technical sense, I’d harm this stranger if I gave them £10, since I leave them worse off than they could have been. But I needn’t be harming them in the ordinary sense, and the same goes for the ten billion extra people in B. Their lives at welfare level 41 could be lives of moderate happiness, with little suffering.

In sum, I think Tomi’s argument presents a real challenge to PAVs.

4. The non-identity problem

Now let’s get back to the non-identity problem. Here’s a recap of how that goes. If the Person-Affecting Restriction and Existence Anticomparativism are both true, then creating a barely happy person is not worse than creating a different, very happy person. That conclusion seems implausible, and so casts doubt on the premises. How might advocates of PAVs respond?

One response is to bite the bullet. Advocates of PAVs can embrace the implausible-seeming conclusion, and thereby hold on to the Person-Affecting Restriction and Existence Anticomparativism. But that’s not as straightforward as it seems, because here’s another, independent argument from Tomi against the implausible-seeming conclusion.

4.1. Tomi’s argument that creating happier people is better

Suppose that Adam already exists. You’re considering creating Eve or Steve. You have three options: D, E, and F. In D, Adam has welfare level 99 and Eve will be created with welfare level 100. In E, Adam has welfare level 100 and Eve will be created with welfare level 99. In F, Adam has welfare level 99 and Steve will be created with welfare level 1.

Here’s the argument. D is equally good as E, because D and E just swap Adam’s and Eve’s welfare levels, and Adam and Eve are equally morally important. And E is better than F, because E is better for Adam, and it replaces worse-off Steve with better-off Eve. And betterness is transitive in the relevant sense: D is equally good as E, and E is better than F, so D is better than F. And Adam’s welfare level is the same in D as in F; the only difference is that D replaces worse-off Steve with better-off Eve. So creating a very happy person is better than creating a different, barely happy person. Since the combination of the Person-Affecting Restriction and Existence Anticomparativism implies the contrary, at least one of these latter two claims must be false.

4.1.1. A PAV response

So advocates of PAVs can’t just bite the bullet on the non-identity problem. They also have to reckon with Tomi’s argument. How might they do that?

One possibility is to shift gears. So far, we’ve been arguing about the axiological facts: facts about what’s good and bad, better and worse. But advocates of PAVs can claim that it’s the deontic facts that are central to morality: facts about what’s morally permissible and morally required. This shift in gears gives PAVs a little more room to manoeuvre, since one might well think that we’re not always morally required to do what’s best. In particular, PAVs could concede that creating better-off Eve is better than creating worse-off Steve, but nevertheless maintain that we’re morally permitted to create worse-off Steve. Or PAVs could concede that creating happy people is good, but nevertheless maintain that we’re morally permitted not to create them (in cases where all else is equal). Now let’s consider these views.

5. Deontic PAVs

At the start of this post I wrote that, according to person-affecting views (PAVs), adding happy people to the world is neither good nor bad. I can now be more precise and call these ‘axiological PAVs’. Related but distinct are deontic PAVs, which say that (in cases where all else is equal) we’re morally permitted but not required to add happy people to the world. As I noted above, retreating to purely deontic PAVs offers a means of escape from some of the arguments of the previous sections.

But there are other arguments that tell against deontic PAVs. To explain these arguments, let’s first distinguish between two kinds of deontic PAV. Consider the following case:

Non-Identity

(1) Amy 1

(2) Bobby 100

Here option (1) is creating Amy with a barely good life at welfare level 1. Option (2) is creating Bobby with a wonderful life at welfare level 100. The first kind of deontic PAV – a narrow view – says that each option is permissible. We’re morally permitted to create the person with the worse life.8 The second kind of deontic PAV – a wide view – says that only (2) is permissible. We’re morally required to create the person with the better life.9

I’ll sketch out arguments against each of these views in turn. This paper presents the arguments in more detail.

5.1. A trilemma for narrow views

Here’s a problem for narrow views. Consider:

Expanded Non-Identity

(1) Amy 1

(2) Bobby 100

(3) Amy 10, Bobby 10

Here we’ve added a third option to Non-Identity. The first two options are as before: create Amy with a barely good life at welfare level 1 or create Bobby with a wonderful life at welfare level 100. The new third option is to create both Amy and Bobby with mediocre lives at welfare level 10.

Narrow views imply that each of (1) and (2) are permissible when these are the only available options. What should they say when (3) is also an option? I’ll argue that they must say at least one of three implausible things, so that narrow views face a trilemma.

Option (1) remains permissible

The first thing they could say is that option (1) – creating Amy with a barely good life at welfare level 1 – remains permissible when we move from Non-Identity to Expanded Non-Identity. But that claim implies:

Permissible to Choose Dominated Options

There are option sets in which we’re permitted to choose some option X even though there’s some other available option Y that dominates X. That is to say, (i) everyone in X is better off in Y, (ii) everyone who exists in Y but not X has a
good life, and (iii) Y is perfectly equal.

That’s because (1) is dominated by (3): (3) creates only people with good lives, it leads to perfect equality, and it’s better than (1) for Amy: the only person who exists in (1). It thus seems implausible that (1) is permissible.

Option (3) is permissible

Here’s something else that narrow views could say about Expanded Non-Identity: option (3) – creating Amy and Bobby with mediocre lives at welfare level 10 – is permissible. But that claim implies:

Permissible to Do Serious Harm for Mediocre Creation

There are option sets in which we’re permitted to choose some option X even though – relative to some other available option Y – all X does is seriously harm one person and create another person with a mediocre life.

That’s because (3) is mediocre for Amy and much worse than (2) for Bobby: Bobby’s welfare level is 100 in (2) and 10 in (3). And we can imagine variations on Expanded Non-Identity in which Bobby’s welfare level in (2) is arbitrarily high. The higher Bobby’s welfare level in (2), the more implausible it is to claim that we’re permitted to choose (3).

Only option (2) is permissible

Now we can complete the trilemma for narrow views. If neither of (1) and (3) is permissible in Expanded Non-Identity, it must be that only (2) is permissible. But if only (2) is permissible, then narrow views imply:

Losers Can Dislodge Winners:

Adding some option X to an option set can make it wrong to choose a previously-permissible option Y, even though choosing X is itself wrong in the resulting option set.

That’s because narrow views imply that each of (1) and (2) is permissible in Non-Identity. So if only (2) is permissible in Expanded Non-Identity, then adding (3) to our option set has made it wrong to choose (1) even though choosing (3) is itself wrong in Expanded Non-Identity.

That’s a peculiar implication. It’s a deontic version of an old anecdote about the philosopher Sidney Morgenbesser. Here’s how that story goes. Morgenbesser is offered a choice between apple pie and blueberry pie, and he orders the apple. Shortly after, the waiter returns to say that cherry pie is also an option, to which Morgenbesser replies, ‘In that case, I’ll have the blueberry.’

That’s a strange pattern of preferences. The pattern is even stranger in our deontic case. Imagine instead that the waiter is offering Morgenbesser the options in Expanded Non-Identity.10 Initially the choice is between (1) and (2), and Morgenbesser permissibly opts for (1). Then the waiter returns to say that (3) is also an option, to which Morgenbesser replies, ‘In that case, I’m morally required to switch to (2).’ The upshot is that the waiter can force Morgenbesser’s hand by adding options that are wrong to choose in the resulting option set. And turning the case around, the waiter could expand Morgenbesser’s menu of permissible options by taking wrong options off the table. That seems implausible.

Summarising the trilemma

Now the trilemma for narrow person-affecting views is complete and I can summarise. If these views say that (1) is permissible in Expanded Non-Identity, they imply that it’s Permissible to Choose Dominated Options. If they say that (3) is permissible, they imply that it’s Permissible to Do Serious Harm for Mediocre Creation. And if they say that only (2) is permissible, they imply Losers Can Dislodge Winners. Each of these implications is implausible.

5.2. A trilemma for wide views

Now let’s consider wide views. Recall that these views say that we’re morally required to create the better-off person in cases like Non-Identity:

Non-Identity

(1) Amy 1

(2) Bobby 100

Wide views thus avoid the trilemma above. They can say that only (2) is permissible in Expanded Non-Identity without implying Losers Can Dislodge Winners. However, wide views imply a trilemma of their own. To see how, consider first:

One-Shot Non-Identity

This case is a cosmetic variation of Non-Identity in which Amy’s and Bobby’s existence will be determined by the positions of two levers. By leaving the left lever up, we decline to create Amy. By pulling the left lever down, we create her at welfare level 1. By leaving the right lever up, we create Bobby at welfare level 100. By pulling the right lever down, we decline to create him. Crucially, the levers are lashed together, so our only options are pulling both levers or pulling neither. Wide views thus imply that pulling both levers is wrong. After all, pulling both levers means creating Amy at welfare level 1 and declining to create Bobby at welfare level 100.

Now consider:

Two-Shot Non-Identity

In this case, the levers are no longer lashed together. We first decide whether to pull the first lever, lock that choice in, and then decide whether to pull the second lever.

I now use these cases to argue against wide person-affecting views. Assume – for contradiction – any wide person-affecting view. Per the ‘wide’ part of such views, it’s wrong to pull both levers in One-Shot Non-Identity. Now assume that the wrongness of pulling both levers doesn’t depend on whether the levers are lashed together. Then it’s also wrong to pull both levers in Two-Shot Non-Identity. Assume also that it’s not wrong to pull the first lever in Two-Shot Non-Identity. Then if we’ve pulled the first lever, it must be wrong to pull the second lever. Finally, assume that the wrongness of pulling the second lever doesn’t depend on past choices. Then it must be wrong to pull the second lever regardless of whether we’ve pulled the first lever. But if that’s the case, then we’re required to create Bobby at welfare level 100. After all, that’s what we do by declining to pull the second lever. This verdict is contrary to the ‘person-affecting’ part of wide person-affecting views. We’ve reached a contradiction.

Therefore, advocates of wide views must reject at least one of my argument’s three assumptions. I now argue that doing so commits them to saying at least one of three implausible things.

Wrongness Depends on Lever-Lashing

To reject the first assumption, advocates of wide views must claim that:

Wrongness Depends on Lever-Lashing

The wrongness of pulling both levers (thereby creating Amy and declining to create Bobby) depends on whether the levers are lashed together. When the levers are lashed together, pulling both levers is wrong. When the lashing is cut, pulling both levers is permissible.

This response is a deontic analogue of myopic choice (McClennen 1990, 12). Myopic choosers sometimes do in two steps what they’d never do in one. The response implies that you’re sometimes permitted to do in two steps what you’re forbidden from doing in one.

Like myopic choice, Wrongness Depends on Lever-Lashing is unpromising on its face. Pulling both levers should either be wrong in both cases or permissible in both cases. It shouldn’t matter whether we can pull them one after the other. After all, it doesn’t matter to Amy or Bobby whether you pull the levers one after the other.

Pulling the First Lever is Wrong

To reject the second assumption of my argument, advocates of wide views must claim that:

Pulling the First Lever is Wrong

In Two-Shot Non-Identity, pulling the first lever (thereby creating Amy) is wrong.

That allows advocates of wide views to say that pulling the second lever is permissible. This response takes inspiration from sophisticated choice (McClennen 1990, 12). Sophisticated choosers predict the choices that they’d make at later timesteps and use these predictions to determine the options available to them at earlier timesteps. This process sometimes prevents them from making earlier choices that they’d otherwise have made. The response in question puts a deontic spin on this general idea. Perhaps the most natural way of making it precise is as follows. Since you might later decline to create Bobby, creating Amy exposes you to a risk of creating only Amy: the one course of action that wide views deem wrong in One-Shot Non-Identity. By contrast, if you don’t create Amy, there’s no chance that you’ll create only Amy and hence no chance that you’ll do what’s wrong according to wide views. Therefore, it’s wrong to pull the first lever and create Amy.

This response is implausible. Pulling the first lever creates Amy with a good life at welfare level 1, and it leaves open the possibility of later creating Bobby with a wonderful life at welfare level 100. The response is even more implausible in a minor variant of Two-Shot Non-Identity in which Amy’s welfare level is 99 instead of 1. In this case, it’s especially hard to believe that creating Amy is wrong. And supposing (as seems natural) that creating Bobby can’t undo any prior wrongness of creating Amy, the resulting wide view implies that it’s impossible to create both Amy and Bobby without acting wrongly. That seems very counterintuitive.

Generalising beyond Two-Shot Non-Identity, the wide views in question prohibit creating a person with a good life whenever you’ll later have the chance to create a person with an even better life, even if creating the first person doesn’t preclude creating the second person. In cases where all else is equal, prospective parents are forbidden from having children until they’ve hit the peak of their welfare-providing powers. That verdict seems undesirable.

Wrongness Depends on First Lever

To reject the third assumption of my argument, advocates of wide views must claim that:

Wrongness Depends on First Lever

Pulling the second lever (thereby declining to create Bobby) is wrong if and only if you’ve previously pulled the first lever (thereby creating Amy).

The response is thus a deontic analogue of resolute choice (McClennen 1990, 13). Resolute choosers sometimes turn down options that they might have chosen had their past choices been different. The response implies that you’re sometimes forbidden from choosing options that you could permissibly have chosen had your past choices been different.

The first thing to say about this response is that it retreats from a deontic person-affecting view, at least as I’ve characterised deontic person-affecting views in this post. That’s because the response concedes that there are cases in which (all else equal) we’re required to create people who would enjoy good lives. Two-Shot Non-Identity is one such case. If you’ve previously created Amy, you’re required to create Bobby. This implication won’t be welcomed by those inclined towards person-affecting views. After all, it runs counter to a major motivation for such views: granting broad latitude to those in a position to create good lives.

The second thing to say about the response is more straightforward: it seems implausible to claim that we’re required to create a better-off person if and only if we previously created a worse-off person. To pump intuitions here, suppose that a friend is considering having a child and comes to you for moral advice. Per the response, you not only need to ask your friend the usual questions about the child’s likely quality of life and how the child might affect existing people. You also need to ask your friend about their past procreative choices. If in the past your friend had a child with a worse life than this new child would have, your friend must have the new child to avoid wrongdoing. And now reversing the order of the cases: if in the past your friend declined to have a child with a better life than this new child would have, your friend must not have the new child. This latter implication seems especially implausible. The new child’s life could be wonderful, but if your friend previously declined to have a child with an even better life, your friend is not even permitted to create them. The response thus implies that there are cases in which (all else equal) we are not even permitted to create a person who would enjoy a wonderful life.

5.3. Summarising the case against deontic PAVs

My argument against deontic PAVs is a dilemma over trilemmas. The first fork is Non-Identity: narrow views are those person-affecting views that permit us to create the worse-off person, and wide views are those person-affecting views that require us to create the better-off person. 

The fork for narrow views is a trilemma centred around Expanded Non-Identity. These views imply Permissible to Choose Dominated Options, or Permissible to Do Serious Harm for Mediocre Creation, or Losers Can Dislodge Winners.

The fork for wide views is a trilemma centred around Two-Shot Non-Identity. These views imply Wrongness Depends on Lever-Lashing, or Pulling the First Lever is Wrong, or Wrongness Depends on First Lever.

6. Conclusion

It’s important to figure out whether person-affecting views (PAVs) are true or false. If PAVs are true, we should be spending less on reducing x-risk. If PAVs are false, we (and the world at large) should be spending more on reducing x-risk, and we should be wary of the potential post-AGI failure-mode of creating way too few happy people.

I think that PAVs are false, but I also think that extant arguments against PAVs are weak. In this post, I’ve sketched out some arguments that I like better. I began with the simple argument and laid out problems for the classic PAV response. I then explained two arguments from Tomi Francis. These arguments imply that it’s good to create happy people, and better to create happier people. I then considered two kinds of deontic PAV – narrow views and wide views – and presented arguments against those. Narrow views face a trilemma in my Expanded Non-Identity case. Wide views face a trilemma in my Two-Shot Non-Identity case.

Robinson Crusoe begins with a disagreement. Eighteen-year-old Crusoe is full of ambition. He’s determined to leave home and set sail for some faraway continent: Africa, perhaps, or South America. After all, the year is 1650, and the seafaring life promises the quickest route to fame and fortune. But Crusoe’s father is set against it and he pleads with his son to remain at home. One morning, the elder Crusoe calls his son into his chamber. In a final attempt to dissuade his son, he sings the praises of a peaceful life, in which people are

not enraged with the passion of envy, or secret burning lust of ambition for great things, but in easy circumstances sliding gently through the world, and sensibly tasting the sweets of living, without the bitter, feeling that they are happy, and learning by every day’s experience to know it more sensibly.

Crusoe is touched by this speech and he resolves to remain at home. But, alas, his ambition ultimately prevails. On the 1^st of September 1651, he boards a ship out of Hull and his “life of misery” begins. He does not return for 35 years.

If the lures of Crusoe’s ambition were great, the lures of ambition in our own day are greater still. Crusoe’s ambition could be kindled only by stories. Our own ambitions are stoked by billboards, screens, and Facebook feeds. Never before have their objects seemed so vivid, so close. However, I claim that we ought to resist our ambitions all the same. Like Crusoe’s father, I want to warn against the temptation of casting our ships out to sea.

Before we begin, I must note two things. The first concerns the scope of this article. I will not try to determine whether ambition has made the world better or worse than it would otherwise have been. Ambition might well be the cause of both humanity’s greatest achievements and its worst atrocities. Who can say whether the final balance will turn out positive or negative? My aim is more modest. I claim only that ambition is no virtue; that we would live better, happier lives if we were less ambitious; and that we ought to devote our energies not to realising our ambitions but to reducing them.

The second thing to note is what I mean by ambition. Plato defined it as the desire for victory. Aristotle understood it as the desire for honour. David Hume and Adam Smith considered it the desire for power and the desire for admiration respectively. Each capture one part of the truth. We come closer by taking up only the common element in each of these ideas: ambition is a desire for greater things. But this definition is still not quite right, for not all those who desire greater things are ambitious. A homeless man looking for shelter on a cold night desires something greater than what he has, but this desire does not make him ambitious. A hungry child is not made ambitious by her desire for a good meal. So how do we determine which of our desires are ambitions? The French essayist François de La Rochefoucauld points us toward the answer. In his Reflections, he writes, “Absence extinguishes small passions and increases great ones, as the wind blows out a candle and fans a fire.” I suggest that the same phenomenon marks the difference between ambitions and ordinary desires, albeit with ‘resistance’ in place of ‘absence.’ Ambitions are those desires that fade and die if we resist them long enough. Ordinary desires only grow as we resist them. The passing of time may extinguish our desire for a higher salary and a bigger house, but it will only increase the desire of the hungry child. With this point noted, we have our definition: an ambition is a desire for greater things that fades with resistance.

So, we find ourselves with a choice. We can resist our ambitions or we can realise them. Why resist? Well, to begin, our ambitions for the future hurt us in the present. Pursuing the things we lack blunts our appreciation of the things we have. The German philosopher Arthur Schopenhauer illustrated this fact by comparing human happiness to a fraction, where the numerator represents what we have and the denominator represents what we hope for. Just as a large numerator will not make a large fraction if the denominator is larger still, so great things will not make us happy if our desires are greater still.

Modern psychologists have placed this wisdom on a firmer scientific footing. Study after study has found that our happiness depends on the size of our aspiration-achievement gap: the gap between what we aspire to and what we achieve. Experiments have shown that even life’s smallest pleasures are tarnished by a desire for more. Chocolate doesn’t taste as good if you’re hoping for something better, and you’ll savour it less if you’re thinking about money. The aspiration-achievement gap may even shed light on a puzzling fact about attempted prison escapes: convicts often try to escape toward the end of their sentences, when they would seem to have least to gain. This phenomenon might seem bizarre, but the aspiration-achievement model provides a neat explanation. As a prisoner approaches the end of his sentence, he imagines his future freedom more frequently and more vividly. His desire for freedom grows, widening the gap between what he has and what he hopes for. Eventually, the size of the gap becomes unbearable and he attempts his escape.

Of course, most of our ambitions are not like the prisoner’s desire for freedom. His desire is doubly harmful. It frustrates him in the present and it spurs him to risk his future. Our ambitions also frustrate us in the present, but they seem to spur us to a better future. They drive us to win promotions, make discoveries, write books, earn accolades, and plenty more besides. Thus, we might acknowledge that our ambitions are harmful but maintain that we ought to hold on to them all the same. We might claim that being ambitious is, on balance, worth it.

The idea is an attractive one. Ambition seems to be a trade in which we endure a slightly-tarnished present for the sake of a much-improved future. However, I claim that it’s a trade we ought to refuse, and for two reasons. First, the benefits of ambition are never as great as we imagine. Second, the costs are much greater than we realise.

Let’s begin with the first reason. We form ambitions, in part, because we expect that achieving them will make us feel certain ways. We picture certain scenes – opening a letter, signing a contract, climbing a podium – and imagine the accompanying emotions: pride, or joy, or contentment. But, all too often, the long-awaited scene comes to pass and the emotions miss their cue. You open the letter, you sign the contract, you reach the top of the podium, and wait for the happiness to radiate through you. But you feel only a faint glimmer, and then a kind of emptiness and unease.

We all have experiences of this kind. We human beings are surprisingly bad at what psychologists call affective forecasting: predicting how future events will make us feel. We frequently overestimate both the intensity and duration of our future emotions. For example, college football fans are not nearly as pleased by their team winning as they imagine they will be. More relevant to ambition, securing tenure does not make junior academics nearly as joyful as they hoped, and missing out does not make them nearly as miserable as they expected. We misjudge how future events will make us feel, in part, because we focus too much on that event and fail to consider other factors. We are especially vulnerable to this focusing illusion when we picture the attainment of our ambitions. We imagine only the signing of the contract and expect a warm glow of delight. We neglect to imagine the attendant stresses which will cloud our happiness.

Nevertheless, you might argue, a glimmer of satisfaction is still a glimmer. Realising our ambitions still makes us happy, even if we’re liable to overestimate the exact amount. Therefore, you might claim, our wonky affective forecasting gives us no reason to renounce our ambitions. But to think this way is to overlook ambition’s costs. We must recognise that our ambitions are not benefactors who give without taking. We pay for our ambitions with our time, our energy, and our emotional investment, and we draw on limited funds to do so. Every hour spent at our desk is one fewer with friends. Every joule of energy expended in pursuit of a dream is one fewer for family. Nights that could have been restful are dogged by tossing and turning. Days that could have been tranquil are blighted by self-censure and restlessness. At its extreme, ambition compels us to give all that we have. Crusoe’s ambition drove him from Hull to Great Yarmouth, on to London, then to Guinea on the west coast of Africa, over to Brazil, and to a shipwreck on a remote island. By the time he returned home over three decades later, his father was dead. We should beware the tempting thought that our own ambitions are not so costly. Nowadays we pay in smaller instalments, but the final price may be greater still.

Recent research suggests that the price we pay is often too high and that the demands of our ambitions exceed their promise. Psychologists note that each of us can be placed somewhere along a scale between pure maximisers and pure satisficers, where pure maximisers desire the best possible result and pure satisficers desire only a good enough result. When surveyed, those with greater maximising tendencies report feeling less happy than those inclined to satisfice. Maximisers also report feeling more regret, more depression, more difficulty making decisions, less optimism, and less satisfaction with their choices. What’s more, maximisers feel this way even when their ambition affords them the expected advantages. College students with greater maximising tendencies secured better-paying jobs than their satisficing counterparts, but nevertheless felt worse during the search and were less satisfied with the result. Their ambition made them feel “pessimistic, stressed, tired, anxious, worried, overwhelmed, and depressed throughout the process,” and at the end it rewarded them only with more disappointment.

Here, though, you might be tempted to lodge an objection: if the maximising students were disappointed, it can only be because they have not yet achieved their ambitions. You might claim that the lesson of the study is not that we should give up but that we should fight harder, because our ambitions will repay us all our sacrifices and more when we achieve them. But this is like claiming that the lesson of an expensive night in Vegas is not that you should go home but that you should bet more, because the casino will repay you all your money and more when you win big. In life, as in Vegas casinos, winning is not guaranteed. It would be reckless to ignore the painful – but very real – possibility that we will not accomplish all that we hope.

However, even if by some miracle we turn out to be among the select few who achieve all their ambitions, I claim that these achievements will likely still come at too dear a price. To see why, note first that our ambitions are, as a rule, self-directed. You desire not just that some important discovery be made, but that you make it. I desire not just that some eminent position be filled, but that I fill it. So, we often compete to achieve our ambitions. We invest time and energy into pursuits in which not everybody wins. One of the lesser-known examples from game theory offers a lively illustration of this dynamic and its consequences:

A $20 bill is put up for sale. Bidding begins at $1 and proceeds in $1 increments. The highest bidder pays what they bid and receives the $20 bill. The second highest bidder pays what they bid and receives nothing.

You can try this dollar auction for yourself at your next party. Initially, there might be some apprehension, but someone will soon bid $1. The prospect of a $19 profit is simply too tempting. Another person will bid $2 for the same reason. $20 for $2 is a once-in-a-lifetime deal. But, unfortunately for the bidders, these seductive thoughts will be their downfall. The first bidder will have a choice at this point: back out now and pay $1 for nothing or advance the bidding to $3 for a $17 profit. Of course, he will bid again. But this move will force on the second bidder an analogous choice: back out now and pay $2 for nothing or advance the bidding to $4 for a $16 profit. She will also bid again. This easy choice swings back and forth until it’s no longer so easy. Should the second bidder advance the bidding to $20? She can no longer hope to make a profit, but at least she’ll avoid a loss of $18. However, her $20 bid will put the first bidder back on the hook for $19. He will face a perplexing choice: back out now and lose $19 or bid $21 (for a $20 bill!) and lose just $1. He too will bid again. But the second bidder will bid $22 for the same reason and now there’s no telling where the bidding will end. A professor who auctioned off $20 bills to his organisational behaviour classes reports that he often received bids north of $50 and notes one instance in which the bidding reached $2000.

The analogy to ambition should be clear. When we pursue our ambitions, we bid our time and energy against the time and energy of others, and we pay no matter what. This matching dynamic begets matching consequences. As in the dollar auction, our reluctance to let past investments go to waste drives us to invest ever more, until our efforts far exceed the value of their object. Of course, our greatest sympathies will be reserved for those who fall short. They invest much and walk away with nothing. But notice that ambition is a game in which even the winners lose. They pay $50 for a $20 bill. Notice also that this pernicious dynamic requires only the smallest of investments to get its start. $1 is enough to seal a bidder’s fate. Ambition, like the dollar auction, is a strange game. The only winning move is not to play.

Ambition, then, depends on a host of misjudgements, confusions, and temptations. We overestimate its benefits, overlook its costs, and overbid our time and energy. But there is one final temptation to be addressed and rooted out: the temptation to say, ‘One more.’ For many of us, this article will have come at an inopportune time. It will have found us halfway through our own dollar auctions – already well on our way to realising some ambition – and we will be tempted to say, ‘One more. Let me accomplish this final thing, and then I will relax.’ But this thought is as pernicious as all the others. Achieving this ambition will almost certainly take us halfway to the next one, and it will be just as tempting to say ‘One more’ when we get there. The lesson of Robinson Crusoe is that ambition is not a journey with a fixed destination at which we might finally and happily come to rest. It is a voyage that has no end. The novel takes Crusoe through three continents, and the last pages see him leaving home once again for a fourth. If he were aiming for any definite place, he would have reached it long ago. It is as if the horizon is the true object of Crusoe’s desire. We should learn from his mistake. Our ambitions have been driving us our entire lives. If they were ever going to satisfy us, they would have done so by now. Our own horizon will elude us no matter how far we sail.

Ludwig Wittgenstein’s Tractatus Logico-Philosophicus is a century-old this year. It deserves its reputation as one of the most difficult books in modern philosophy.

Part of the difficulty is down to its subject matter. The book is about how language represents reality: how sentences like ‘The cat is on the mat’ manage to tell us something about the world. Exactly because we’re so accustomed to using these kinds of sentences, it can be hard to grasp just what Wittgenstein’s worry is. We’re like fish reading a book about the nature of water.

Another part of the difficulty can be chalked up to self-reference. Wittgenstein uses language to conduct his investigation into the nature of language. The book is thus a kind of ouroboros eating its own tail, with all of the trouble that involves.

A third part of the difficulty stems from the book’s length. It’s just 80 pages of cryptic, tweet-length remarks – sometimes precise, sometimes vague – covering everything from metaphysics to logic to ethics to theology.

But at least part of the difficulty is down to Wittgenstein’s stubborn refusal to give any examples of the kinds of things he’s talking about: names, objects, elementary propositions, atomic facts. His reluctance is understandable in a way. Wittgenstein claimed to establish the existence of these things on a priori grounds – via argument, rather than observation – and he thought that any examples he could give would be misleading. But his reluctance is peculiar in another way. As we’ll see, Wittgenstein is willing to write things that aren’t strictly true in order to help us understand him, so it’s strange that he didn’t extend this willingness to concrete examples and illustrations.

In this post, I’m going to do what Wittgenstein didn’t. I’m going to give a quick overview of the Tractatus that’s chock-full of examples and analogies. My exposition will be rough at some points and simplistic at others. Readers looking for maximum accuracy should consult the academic secondary literature. I’m writing this post because even the so-called ‘Introductions’ to the Tractatus can be hard-going.

Let’s begin. Wittgenstein’s main concern in the Tractatus is with propositions. Propositions are those things that can be true or false, the kinds of things that we express with declarative sentences. ‘The cat is on the mat’, for example, is a declarative sentence expressing the proposition that the cat is on the mat.

Some propositions have sense. Roughly, they tell us something about the world. More precisely, whether they’re true or false depends on how the world is arranged. ‘The cat is on the mat’ expresses a proposition with sense. It’s true if the cat is on the mat and false otherwise. Other propositions are senseless. They’re true no matter how the world is arranged, so they tell us nothing about the world. ‘Either it’s raining or it’s not raining’ expresses a senseless proposition. Still other propositions are nonsense. They also tell us nothing about the world, but not because they’re true no matter what. Instead, they tell us nothing because they’re neither true nor false. ‘Colourless green ideas sleep furiously’, for instance, expresses a nonsense proposition. Or, to take one of the rare examples in the Tractatus, ‘The good is more identical than the beautiful’ expresses a nonsense proposition.

These cases are pretty clear-cut, but others are less certain. Does the proposition expressed by ‘Murder is wrong’ have sense? Does it tell us anything about the world? How about the proposition expressed by ‘Time isn’t real’? Does that have sense? We can argue about each of these propositions individually, but that can be a long and tedious affair. It would be convenient if we had some formula to tell us which propositions are sensical, which are senseless, and which are nonsensical.

The Tractatus gives the outlines of such a formula. It presents ‘the general form of proposition’: a recipe for constructing propositions with sense. Wittgenstein’s claim is that all sensical propositions are of this same general form. Any proposition not of this form is nonsense. Wittgenstein is thus trying to do philosophers a huge favour. In philosophy, talking nonsense is an occupational hazard. You might well come to the end of a long career only to discover you were talking nonsense the entire time. The Tractatus is supposed to help us avoid that fate.

To see how Wittgenstein unearths ‘the general form of proposition’, let’s start with a particular example. Take the proposition expressed by ‘One of the Beatles was a bachelor.’ This proposition is sensical. It tells us something about the world. This proposition can also be analysed, by which I mean it can be broken down into parts. ‘One of the Beatles was a bachelor’ can be analysed into four propositions, connected by the word ‘or’: (1) John was a bachelor, or (2) Paul was a bachelor, or (3) George was a bachelor, or (4) Ringo was a bachelor. The symbol ‘bachelor’ can also be analysed, this time into two symbols connected by the word ‘and’. A bachelor is (1) unmarried, and (2) a man.

This process of analysis might well continue, but it can’t go on forever. Eventually, we’ll reach what Wittgenstein called elementary propositions: propositions that can’t be broken down. These elementary propositions are combinations of names: symbols that can’t be broken down.

Since our example proposition – One of the Beatles was a bachelor – has sense, each of its elementary propositions must also have sense. For these elementary propositions to have sense, each of the names in these elementary propositions must refer to something. The proposition expressed by ‘London is in England’ has sense because each of its terms refers to something. The proposition expressed by ‘London is in Flobertness’ is nonsensical, because ‘Flobertness’ doesn’t refer to anything. Wittgenstein thus concludes that names must refer to simple objects. These objects can’t be broken down. If they could be broken down, they might cease to exist. Then the corresponding name would have no reference, and any elementary proposition featuring that name would be nonsense. Wittgenstein’s enquiry into the nature of language has thus led him to a conclusion about the nature of reality: there must be simple objects forming the substance of the world.

Wittgenstein gives us no examples of objects, names, or elementary propositions. He establishes their existence through argument rather than observation, and so is happy to let others do the work of identifying them. He thinks that he can know that objects, names, and elementary propositions exist, even in the absence of examples. Think of it this way. I can know that you have an ancestor who spent their entire life underwater, even though I can’t point out any such ancestor. Take your parents and ask, ‘Did one of them spend their entire life underwater?’. If the answer is – as I suspect – ‘No,’ then take your parents’ parents: did one of them spend their entire life underwater? That’ll also be a ‘No,’ so take your parents’ parents’ parents, and so on. Eventually, the answer must be ‘Yes,’ because we all descended from sea creatures. I can thus know that you have at least one entirely aquatic ancestor, even though I know of no examples. Wittgenstein’s argument is analogous: if a proposition has sense, its analysis must end with elementary propositions made up of names referring to simple objects. Never mind that examples are hard to come by.

All that said, here’s one way of illustrating Wittgenstein’s view. Let the fundamental particles of matter – quarks, leptons, and the like – be our objects. Each such object is assigned a number, which serves as its name. Elementary propositions are then combinations of numbers. We might express them with lists, like ‘13, 28, 567, 435.’

(This illustration isn’t entirely faithful to the theory expressed in the Tractatus. Wittgenstein’s objects can’t be divided. They’re ‘unalterable and subsistent,’ and they exist in every world that we can imagine. Fundamental particles might lack some of these properties.)

Note that elementary propositions are just combinations of names, like ‘13, 28, 567, 435.’ They don’t say ‘28 is between 13 and 567’ or ‘567 crashes into 435’ or anything like that. How then can elementary propositions have sense? How can a mere combination of names tell us something about the world?

Wittgenstein’s answer is given by his famous picture theory of meaning. It’s an idea that’s said to have occurred to him in a Paris traffic-court, where he saw an accident reconstructed with toys. Those toys stood as proxies for the people and cars involved in the accident, and the way that the toys were arranged showed the way that the people and cars were arranged at the time of the accident. The toys together constituted a picture: ‘a model of reality.’ Wittgenstein claims that elementary propositions are also a kind of picture: an elementary proposition is a picture of an atomic fact. An atomic fact is a combination of objects, an elementary proposition is a combination of names, and the way that the names are combined in an elementary proposition shows how the corresponding objects are combined in an atomic fact.

With that last sentence, we’ve completed our descent from the top – an ordinary proposition expressed by ‘One of the Beatles was a bachelor’ – down to the very bottom: names, objects, elementary propositions, and atomic facts. Now to climb back up again. Wittgenstein’s claim is that all propositions with sense are founded on elementary propositions. More precisely, each proposition with sense is a truth-function of elementary propositions. To understand what’s meant by ‘truth-function,’ return to our Beatles example. The proposition expressed by ‘One of the Beatles was a bachelor’ is a truth-function of the following four propositions: (1) John was a bachelor, (2) Paul was a bachelor, (3) George was a bachelor, (4) Ringo was a bachelor. What that means is that the truth of ‘One of the Beatles was a bachelor’ depends only on the truth of the latter four propositions. Once you know whether each of the latter four propositions are true, you know whether ‘One of the Beatles was a bachelor’ is true. More generally for Wittgenstein, if you know the truth of all the elementary propositions, you know everything there is to know.

That gives us Wittgenstein’s formula for determining whether a proposition has sense. If a proposition could be true or false, depending on the truth of elementary propositions, that proposition has sense. If a proposition is true no matter which elementary propositions are true, that proposition is senseless. And if a proposition is not a truth-function of elementary propositions – if you couldn’t tell whether it’s true or false even if you knew all the elementary propositions – that proposition is nonsense.

This formula turns out to be bad news for philosophers. In Wittgenstein’s estimation, only scientific questions can be given answers with sense. The propositions of ethics (like ‘Murder is wrong’), aesthetics (like ‘Paris is beautiful’) and theology (like ‘God exists’) turn out to be nonsense. Indeed, all philosophical propositions turn out to be nonsense.

But hang on a minute! The Tractatus is filled to the brim with philosophical propositions. Does Wittgenstein think he’s been writing nonsense this whole time? Funnily enough, yes:

My propositions serve as elucidations in the following way: anyone who understands me eventually recognizes them as nonsensical, when he has used them—as steps—to climb up beyond them. (He must, so to speak, throw away the ladder after he has climbed up it.) He must transcend these propositions, and then he will see the world aright. – 6.54

But this answer only invites more questions: what’s the point of reading the Tractatus? How can a book full of nonsense help us ‘see the world aright’?

Wittgenstein’s answer is as follows: the Tractatus shows that any attempt to answer philosophical questions must be nonsense. There simply are no sensical answers to such questions. And so, in the words of Wittgenstein’s final proposition, ‘What we cannot speak about we must pass over in silence.’

But here one might object. Even if Wittgenstein’s propositions imply their own nonsensicality, that doesn’t prove that all philosophical propositions are nonsense. It doesn’t even prove that Wittgenstein’s propositions are nonsense. They might just be plain-old-false. Consider an analogy. The sentence ‘Any sentence consisting of more than five words is nonsense’ implies its own nonsensicality. But that sentence isn’t nonsense. It’s false.

Clearly, Wittgenstein didn’t think that his propositions were false. What might justify his own, more radical reading? Here’s one answer. Suppose that the propositions of the Tractatus strike us as the only viable answers to philosophical questions even after we recognise that these propositions imply their own nonsensicality. Then we might conclude that all philosophical propositions must be nonsense.

I’ll end with a metaphor. Imagine you’re in a clearing in a South American rainforest, searching for El Dorado. Many paths are open to you, and you don’t know which will take you there. You choose a path that looks promising, but after travelling a while you realise that you’re back where you started. You resolve to try again. But even knowing that the first path leads you back to the clearing, it still strikes you as the only way to reach your destination. In that case, you might conclude, there’s no need to try another path. There is no El Dorado.

Now the United States Department of Transportation – or the US Department of Transportation, as I’m sometimes bold enough to call them – has your two numbers: the cost (measured in dollars) and the benefit (measured in Americans). You’re summoned back into the head honcho’s office and she punches these numbers into the ceremonial calculator: dollars divided by American lives. If the answer is $11.8 million per life or less, she shakes your hand and writes a cheque. If the answer is more than $11.8 million per life, she says ‘Take a hike. Never talk to me again.’ I exaggerate and simplify, but only slightly.

Now. I am an American citizen; I care only about the two inevitables. Do you know what I mean by ‘the inevitables’?

Dash? Elastigirl?

No, no, no. The inevitables: death and taxes. I am an American citizen; I care only about death and taxes. I exaggerate and simplify, but only slightly.

Now every time the head honcho at the US DoT signs a cheque, my taxes go up. That’s bad. But once the DoT project is complete, my chance of getting smeared across America’s highways goes down. That’s good. I’m a typical American. Every new $20 million DoT project is another 6¢ out of my pocket and at least another 0.000000005 ticks down on the old highway-death-o-meter. It’s a good deal. I’m happy to take it. I’ll have what I’m having. I exaggerate and simplify, but only slightly.

But then I come across The Precipice: a book written by one Toby Ord of Oxford University, England. In The Precipice, Ord gives us his best guess of the chance that everything goes wrong this century: 1-in-6, Russian roulette. And ‘everything’ here means everything. We’re talking disaster-movie-without-the-happy-ending. Nuclear conflagration, a modern plague, AI takeover. Armageddon, Contagion, War Games, The Day After Tomorrow, Terminator. Humanity in ruins and remaining so, for all the aeons until our great universe dissipates into fuzz.

But forget that last part. I never went in for that ‘deep time’ stuff anyway. Today’s troubles are enough for today, and my troubles are enough for me. Let’s focus on the here-and-now. This century. My life and the life of my children. Russian roulette. Do I feel lucky? I’m a confident guy. I know my way around a screwdriver. I like to think I’d comport myself pretty well in a total breakdown of the social order. But if things go Dr Strangelove, I’ve just got to come to terms with the fact that one of those ICBMs might have my name on it and that no degree of screwdriving-prowess is going to scratch it off. No man is an island. That bell will toll for me.

No. If it’s to be less than 1-in-6 that I perish from this earth, I need the government. Only they can make a real dent in my chances of getting fried, or of spluttering out my last in some makeshift hospital ward. And not only can the government do it, the price is right. Carl Shulman and Elliott Thornley estimate that $32 billion per year would increase the chance that we make it through the decade by 1-in-200. That’s a small number, but the ceremonial calculator reveals that it’s an even better deal than the one I’m getting from the US DoT: $100 out of my pocket and a big tick down on the old annihilation-o-meter.

1-in-6, 1-in-200. The numbers are unsettling, but they’re also unsettled. Odds are at home in Vegas where they’re born from the motions of dice, cards, and roulette wheels, and can be confirmed by long painful experience. But Ord has never sat and tallied infernos, never watched his supply of civilizations grow and fall and dwindle to nil. The 6s and 200s are estimates, surmises, guesses, and I’ve bought enough used cars in my time to be wary of numbers plucked from thin air. But still. The question remains. What should we do? Should we renew the nuclear treaties or let them lapse? Should we hire scientists to watch for cooked-up superdiseases or save our money? Should we pump the brakes on AI or let it race ahead? If we do nothing, we’ll be guessing that the risk is low. If we do something, we’ll be guessing that the risk is not so low. Whatever we choose, we’ll be making a guess, so we might as well use our best one. Your best guess might differ from Ord’s, but you won’t know what it is until you make it your best: until you sit down with head clear and heart still and look at what we know. Nuclear weapons are younger than your grandparents and they’ve fallen out of planes many, many more times. Rabies kills all its untreated victims, COVID-19 infected almost everyone, and lab-engineered diseases could do both. AI systems are already so big that their creators can’t understand them and so erratic that billion-dollar companies can’t control them, and they’re about to get hundreds of times bigger.

It might feel wrong to say a number. But remember: guessing is not optional. Every action is a choice, and every choice is a guess. So look at what we know; guess, guess, guess; and use your best. If the bombs start falling, how many of us die? If it’s an engineered pandemic, how many of us die? What can we do? How much would it cost? How much would it help? Then get out the calculator. Less than $11.8 million per American and it would save a whole load of non-Americans too? It’s a good deal. Let’s take it.

[1:52]

Look at you.

Aw, Kramer, don’t start.

No, no. You're wasting your life.

I am not! What you call wasting, I call living! I'm living my life!

Okay, like what? No, tell me! Do you have a job?

No.

You got money?

No.

Do you have a woman?

No.

Do you have any prospects?

No.

You got anything on the horizon?

Uh... no.

Do you have causal influence over anything important at all?

No.

Do you have any conceivable reason for even getting up in the morning?

I like to manage the daily news.

Went out to a philosophy seminar the other night. Conclusion came at the end of the seminar as it always does. Never liked the 'conclusion at the end of the seminar' system, because premises are a very different thing before and after the conclusion. Before the conclusion, premises have no flaws. You don't care about premises before the conclusion. You sit down in the seminar, you're like the ruler of an empire: more claims! definitions! quickly, quickly. It will be the greatest seminar of our lives. Then after the seminar, you know, you've got your laptop open, you’ve got the whiteboard destroyed, pens strewn all over the table. Then the conclusion comes, at that moment. People are always upset, you know. Mystified by the conclusion. What is this? How could this be? They start passing it around the table. Does this look right to you? We don't like the conclusion. Why are we buying all these premises?