QALYs/$ are more intuitive than $/QALYs

Cross-posted to the effective altruism forum.

Summary

Cost-effectiveness estimates are often expressed in $/QALYs instead of QALYs/$. But QALYs/$ are preferable because they are more intuitive. To avoid small numbers, we can renormalise to QALYs/$10,000, or something similar.

Cost-effectiveness estimates are often expressed in $/QALYs.

Four examples:

GiveWell, “Errors in DCP2 cost-effectiveness estimate for deworming”:1

Eventually, we were able to obtain the spreadsheet that was used to generate the $3.41/DALY estimate. That spreadsheet contains five separate errors that, when corrected, shift the estimated cost effectiveness of deworming from $3.41 to $326.43. We came to this conclusion a year after learning that the DCP2’s published cost-effectiveness estimate for schistosomiasis treatment – another kind of deworming – contained a crucial typo: the published figure was $3.36-$6.92 per DALY, but the correct figure is $336-$692 per DALY. (This figure appears, correctly, on page 46 of the DCP2.)

DCP3, “Cost-Effectiveness of Interventions for Reproductive, Maternal, Newborn, and Child Health”:

dcp3_qaly

Michael Dickens, “Charities I would like to see”:

This would cost about $5 per rat per month plus an opportunity cost of maybe $500 per month for the time spent, which works out to another $5 per rat per month. Thus creating 1 rat QALYs costs $120 per year, which is $240 per human QALYs per year.

Deworming treatments cost about $30 per DALY. Thus a rat farm looks like a fairly expensive way of producing utility.

GiveWell, “Mass Distribution of Long-Lasting Insecticide-Treated Nets (LLINs)” uses cost per life saved:

LLIN distribution is one of the most cost-effective ways to save lives that we’ve seen. Our best guess estimate comes out to about $3,000 per equivalent under-5 year old life saved (or, excluding developmental impacts, $7,500 per life saved) using the total cost per net in the countries we expect AMF to work over the next few years.

QALYs/$ are preferable to $/QALYs

As long as we compare opportunities to do good by looking at the ratio of their cost-effectiveness, $/QALYs is equivelent to QALYs/$.

However, even if we know that we ought to be using ratios of cost-effectiveness, our System 1 may sometimes implicitly be using differences (subtractions) of cost-effetiveness. This can lead to problems when using $/QALYs which are entirely avoided if we use QALYs/$.

Suppose we have 20 charities - whose cost-effectiveness follows a log-normal distribution. I have plotted bar graphs of these values expressed in $/QALYs and in QALYs/$.

d_per_q Looking at this graph, we are immediately attracted to the right-hand side. That’s where the big, visible differences in bar height are. So we feel that the high-hand side is where most of the action is. We may have the intuition that most of the gains are to be had by switching away from from charities like , , and , in favour of charities like , and . This is because we would implicitly be using differneces instead of ratios.

In reality, of course, what’s crucial is the left-hand side of the graph. Charity produces about 9 times more value than charity , while charity is only 1.5 times better than charity .

q_per_d If we had used QALYs/$, this would have been easier to see. Here, the importance of picking the best charity (rather than a merely good one) stands out visually.

When we use QALYs/$, both products and subtractions give us the correct result. That is why QALYs/$ are preferable.

Small numbers

One potential problem with using QALYs/$ is that we end up with very small numbers. Small numbers can be unintuitive. It’s hard to picture 0.05 and 0.1 of something, and easy to picture 20 and 10 of something.

But this problem can easily be solved by multiplying the small numbers by a large constant. This is what we did with the Oxford Prioritisation Project, and it’s also what Toby Ord does in “The moral imperative towards cost-effectiveness”.

Further reading

By the way, this exact phenomenon is well documented in the domain of car fuel efficiency. See “The MPG Illusion”, Science Vol. 320, Issue 5883, pp. 1593-1594, DOI: 10.1126/science.1154983.

Bastian Stern also has posts explaining how $/QALYs create problems when we use arithmetic means, and when we look at proportional improvements between charities. This is not suprising, since arithmetic means and proportions are essentially based on subtraction.

Recommendation

Wherever possible, we should stop using $/QALYs and use QALYs/$10,000, or something similar.

  1. Of course, there are also many examples of people correctly using QALYs/$. See for instance “The moral imperative towards cost-effectiveness”, or chapter 3 of “Doing Good Better”. 

June 15, 2017

Self-locating beliefs vs loss of discriminating power

sleeping beauty

In Adam Elga’s 2000 paper “Self-locating belief and the Sleeping Beauty problem”, he opens with:

In addition to being uncertain about what the world is like, one can also be uncertain about one’s own spatial or temporal location in the world. My aim is to pose a problem arising from the interaction between these two sorts of uncertainty, solve the problem, and draw two lessons from the solution.

His answer to the sleeping beauty problem is 1/3. But this violates conditionalisation and reflection. His diagnosis is that this has to do with the self-locating nature of the beliefs:

The answer is that you have gone from a situation in which you count your own temporal location as irrelevant to the truth of H, to one in which you count your own temporal location as relevant to the truth of H. […] [W]hen you are awakened on Monday, you count your current temporal location as relevant to the truth of H: your credence in H, conditional on its being Monday, is 1/ 2, but your credence in H, conditional on its being Tuesday, is 0. On Monday, your unconditional credence in H differs from 1/ 2 because it is a weighted average of these two conditional credences — that is, a weighted average of 1/2 and 0.

But Arntzenius (2003) shows that the problem has nothing to do with the self-locating nature of the beliefs and everything to do with the loss of discriminating power of experiences.

Strict conditionalization of one’s degrees of belief upon proposition X can be pictured in the following manner. One’s degrees of belief are a function on the set of possibilities that one entertains. Since this function satisfies the axioms of probability theory it is normalized: it integrates (over all possibilities) to one. Conditionalizing such a function on proposition X then amounts to the following: the function is set to zero over those possibilities that are inconsistent with X, while the remaining nonzero part of the function is boosted (by the same factor) everywhere so that it integrates to one once again. Thus, without being too rigorous about it, it is clear that conditionalization can only serve to “narrow down” one’s degree of belief distribution (one really learns by conditionalization). In particular a degree of belief distribution that becomes more “spread out” as time passes cannot be developing by conditionalization, and a degree of belief distribution that exactly retains its shape, but is shifted as a whole over the space of possibilities, cannot be developing by conditionalization.

So we need to distinguish problems with spreading from problems with shifting.

shifting

Self-locating beliefs undergo shifting in a perfeclty straightforward manner which has nothing to do with sleeping beauty type cases:

suppose that one is constantly looking at a clock one knows to be perfect. […] At any given moment one’s degrees of belief […] will be entirely concentrated on one temporal location, namely, the one that corresponds to the clock reading that one is then seeing. And that of course means that the location where one’s degree of belief distribution is concentrated is constantly moving.

spreading

Beliefs can undergo spreading when the situation is such that there is a loss of discriminating power of experiences over time. In Shangri-La1,

there are two distinct possible experiential paths that end up in the same experiential state. That is to say, the traveler’s experiences earlier on determine whether possibility A is the case (Path by the Mountain), or whether possibility B is the case (Path by the Ocean). But because of the memory replacement that occurs if possibility B is the case, those different experiential paths merge into the same experience, so that that experience is not sufficient to tell which path was taken. Our traveler therefore has an unfortunate loss of information, due to the loss of the discriminating power of his experience.

The same thing is happening in sleeping beauty, contra Elga:

In the case of Sleeping Beauty, the possibility of memory erasure ensures that the self-locating degrees of belief of Sleeping Beauty, even on Monday when she has suffered no memory erasure, become spread out over two days.

It just so happened that Elga chose an example in which self-locating beliefs are “counted as relevant to the truth of H”. This caused confusion.

implication for bayesianism

The lesson from Arntzenius (2003) is that conditionalisation, understood as ereasing and re-normalising, is not a necessary condition of ratioanlity.

It’s a mistake to think of bayesian rationality as conditionalisation. The key maxim, that implied by Bayes’ theorem, is: ‘At each time, apportion your credences to your evidence’.

We can think of this as having an ‘original’ or ‘ur-prior’ credence distribution. At each time, you should update that ur-prior based on your total evidence E. E can come to contain less information, (you “lose evidence”) in cases of fogetting or loss of discriminating power. When you lose evidence, your credence distribution undergoes spreading.

constraints on ur-priors?

What norms constrain the ur-priors of rational agents? One possibility is the following. Think of possible worlds. Within each possible world, there are many experience-moments: one for each observer location and each time. When one is uncertain about about one’s own spatial or temporal location, one is uncertain about which experience-moment one finds oneself in within a possible world.

So one set of possible constraints are:

  • In accordance with the principal principle, your credence in each possible world should be equal to the objective chance of that world.
  • In accordance with a principle of indifference, your should apportion your credence in a world equally between all its possible experience-moments.
  1. “Every now and then, the guardians to Shangri La will allow a mere mortal to enter that hallowed ground. You have been chosen because you are a fan of the Los Angeles Clippers. But there is an ancient law about entry into Shangri La: you are only allowed to enter, if, once you have entered, you no longer know by what path you entered. Together with the guardians, you have devised a plan that satisfies this law. There are two paths to Shangri La, the Path by the Mountains, and the Path by the Sea. A fair coin will be tossed by the guardians to determine which path you will take: if heads you go by the Mountains, if tails you go by the Sea. If you go by the Mountains, nothing strange will happen: while traveling you will see the glorious Mountains, and even after you enter Shangri La, you will forever retain your memories of that Magnificent Journey. If you go by the Sea, you will revel in the Beauty of the Misty Ocean. But, just as you enter Shangri La, your memory of this Beauteous Journey will be erased and be replaced by a memory of the Journey by the Mountains. Suppose that in fact you travel by the Mountains. How will your degrees of belief develop? Before you set out your degree of belief in heads will be 1/2. Then, as you travel along the Mountains and you gaze upon them, your degree of belief in heads will be one. But then, once you have arrived, you will revert to having degree of belief 1/2 in heads. For you will know that you would have had the memories that you have either way, and hence you know that the only relevant information that you have is that the coin was fair. This seems a bizarre development of degrees of belief. For as you are traveling along the Mountains, you know that your degree of belief in heads is going to go down from one to 1/2. You do not have the least inclination to trust those future degrees of belief. Those future degrees of belief will not arise because you will acquire any evidence, at least not in any straightforward sense of “acquiring evidence.” Nonetheless, you think you will behave in a fully rational manner when you acquire those future degrees of belief. Moreover, you know that the development of your memories will be completely normal. It is only because something strange would have happened to your memories had the coin landed tails that you are compelled to change your degree of belief to 1/2 when that counterfactual possibility would have occurred.” 

June 13, 2017

Comment un individu peut-il faire une différence? Une mauvaise réponse, et deux bonnes

J’entends souvent des dialogues de ce genre quand il s’agit de faire un effort personnel pour aider les autres:

Alice: Face à toute la souffrance dans le monde, je me sens impuissante. Même si je changeais mon comportement, mon action individuelle ne résoudrait pas nos problèmes. Par exemple, même si je faisais un don pour aider un agriculteur pauvre au Kenya, d’autres ne donneront rien, et ce n’est pas grâce à moi que nous allons éliminer la pauvreté. Ce n’est pas à moi, mais aux puissants de ce monde d’agir.

Bernard: Si tout le monde raisonnait comme toi, nous ne ferions jamais rien pour aider les plus vulnérables. Au contraire, si chacun agit à son niveau, nous pouvons éliminer la pauvreté ensemble. Ainsi, en prenant partie à une action sociale, chacun peut changer les choses. Tu n’es donc pas impuissante.

Alice et Bernard font tous deux erreur. Nous ne sommes pas impuissants à être solidaires, mais ce n’est pas pour les raisons avancées bar Bernard.

Nous sommes tous des individus

J’ai de la sympathie pour la point de vue de Bernard. Alice doit avoir tort, car porter son raisonnement à sa conclusions aurait pour conséquence qu’il faudrait arrêter de travailler à tous les grands problèmes de l’humanité. Mais la réponse de Bernard est fallacieuse.

Nous sommes tous des individus. Je ne suis pas une société, vous n’êtes pas un état, aucun d’entre nous n’est un groupe ou une institution. Les actions dont nous pouvons décider sont des actions individuelles. Bien sûr, des individus peuvent influencer ces groupes, mais encore une fois, le choix de cette influence est finalement individuel. C’est le choix de voter, de se rendre à la réunion communale, ou d’aller manifester. La réplique “si tous agissent à leur niveau…” n’est pas opérante car personne ne peut décider si tous agissent ou non. Nous pouvons seulement décider d’agir nous-mêmes, ou de tenter de convaincre d’autres d’agir.

Distinguer deux objections

Afin d’expliquer pourquoi Alice a réellement tort, il faut distinguer deux objections différentes qui pourraient se cacher derrière ce discours.

Rapellons-nous la phrase d’Alice:

Même si je changeais mon comportement, mon action individuelle ne résoudrait pas nos problèmes.

Une première interprétation de cette phrase est la suivante:

Les petits changements ne vont pas résoudre les grands problèmes : une victime de la pauvreté de moins ne vas pas affecter le développement économique d’un pays pauvre; un végétarien ne va pas mettre fin à l’élevage industriel.

A ce premier type d’objection il y a une réponse simple et correcte. Ceux qui objectent de cette façon utilisent implicitement une fraction de ce type :

Par exemple, faire un don à une ONG luttant contre l’extrême pauvreté au Kenya est intuitivement rattaché au problème de la pauvreté dans le monde. Devenir végétarien correspond au but de mettre fin à l’élevage industriel. Dans ces deux cas, le dénominateur de la fraction est très grand : votre impact individuel ne représente qu’une partie infime du problème que vous souhaitez résoudre. Mais il est fallacieux d’utiliser cette fraction. Pour décider si une action en vaut la peine, il faut comparer ses bénéfices à ses coûts, et non pas à la taille d’un autre problème. Si vous souhaitez maximiser votre impact positif, c’est simplement le numérateur qu’il faut maximiser. Le dénominateur est sans conséquence. L’habitant du village Kenyan souhaite seulement atteindre une vie meilleure, le niveau de la pauvreté mondiale lui importe peu. La poule élevée en batterie souhaite seulement échapper à sa vie de souffrance, quoi qu’il arrive à la production mondiale de viande[1]. La question qu’il faut plutôt vous poser est quelle serait la manière de les aider le plus possible.

Un semblant de paradoxe

Ce dont je souhaite en réalité parler ici est la seconde interprétation de l’objection d’Alice, plus sophistiquée. Au lieu d’avancer que votre impact personnel est très petit par rapport à une autre grandeur, l’argument est cette fois bien que l’effet de votre action individuelle est nul.

Imaginons qu’Alice ne fasse le discours suivant: “Réfléchis à ce qui se déroule réellemet lorsque tu agis à une échelle individuelle. Quand tu décides de ne pas acheter de viande au supermarché, le paquet que tu laisses sur l’étagère est composé d’animaux qui sont déjà morts. Ce n’est donc pas eux que tu aides. Mais tu raisonnes ainsi: “si je n’achète pas de viande, le supermarché s’ajustera à cette réduction de demande en commandant moins de viande, ce qui finira par réduire l’offre de viande et ainsi le nombre d’animaux en élevage”. Mais ce raisonnement n’est pas correct. Les décisions du supermarché ne dépendent pas de ton choix. Le supermarché réduira sa demande de viande uniquement s’il observe une réduction importante de la consommation, par exemple 2% de son inventaire. Ta réduction individuelle sera trop petite pour affecter le choix du supermarché. Les gérants ne remarqueront même pas ton choix, masqué par les variations aléatoires de la consommation de viande.

De même, regardons comment fonctionne l’ONG. Elle opère actuellement des écoles dans 5 villages. Pour établir une école dans un sixième village, il lui faut 100 000€ par an de dons supplémentaires pour payer les enseignants. Si elle n’en recoit que 99 999€, elle ne pourra pas les payer, et conservera l’argent dans son compte en banque. À moins que l’ONG ne soit déjà à 90 000€ ou plus, un don de 10 000€ n’aura donc aucun impact.

Un troisième exemple: imaginons qu’un tremblement de terre ne frappe le Népal. Une ONG népalaise organise en urgence l’achat de matériel médical à une entreprise pharmaceutique Indienne. Pour des raisons logistiques, l’entreprise ne vend le matériel qu’en incréments de 100 000€. Comme pour l’école dans le village Kenyan, il est très improbable que votre don soit celui qui permet tout juste à l’ONG d’acheter un incrément de plus.”

Alice a raison d’observer que la grande majorité des petits dons ou un petits changements de comportement n’ont aucun impact. Pourtant, un petit nombre d’entre eux ont un impact demesuré. Par exemple, le donateur marginal qui fait passer l’ONG de 99 999€ à 100 000€ donne lieu à la construction d’une école avec un seul euro!

Fonction en escalier Une fonction en escalier. Ici, certains mouvements vers la droite le long de l’axe des abcisses (les dons ) n’ont aucun effet sur l’impact de l’organisation, alors que certains très petits mouvements ont un grand impact en passant d’un palier à un autre.

Il faudrait donc, en théorie, tout faire pour être le donateur marginal, ou le consommateur qui s’abstient d’acheter l’unique portion de viande qui ferait basculer la décision du supermarché.

La solution du paradoxe se trouve dans le fait qu’un ciblage si précis est impossible en pratique. Dans les exemples simpifiés ci-dessus, il faudrait connaitre en détail la situation financière des ONG ou l’inventaire du supermarché, et de plus prédire le comportement de tous les autres donateurs et conommateurs. De surcroît, en réalité les procédures de décision qu’appliquent ces institutions sont bien plus complexes que je ne l’ai fait paraître. Il faudrait prendre en compte toute la chaîne de production de la viande, ou encore toutes les opportunités et contraintes auxquelles fait face l’ONG.

Plusieurs fonctions en escalier

En réalité, nous sommes ignorants quant à laquelle d’un grand nombre de fonctions en escalier correspondent à la réalité. En moyennant ces fonctions, nous arrivons à nouveau à une fonction linéaire.

Nous sommes donc totalement ignorants quant à l’identité du donateur marginal. Nous conaissons seulement la probabilité d’être au point de bascule, par exemple 10% avec un don de 10 000€ et des incréments de 100 000€. Il serait possible de réduire notre incertitude quant à l’identité du point de bascule, par exemple en construisant une modéalisation mathématique extrêmement complexe de la chaine de production de la viande. Mais en pratique cette procédure serait bien plus couteuse que le don lui-même.

Alice oublie de mettre en balance cette probablité de succès avec la taille du bénéfice potentiel. Autrement dit, si la probabilité de succès est et la taille du bénéfice , la quantité qui nous intéresse est l’espérance mathématique . Alice comment l’erreur de ne considérer que . En général, dans ce type de cas, est très petit mais est très grand, et ce de manière proportionelle. Par exemple, si , , alors correspond à un dixième de la valeur d’une école: exactement le même ratio que celui entre la taille du don (10 000€) et le cout d’une école (100 000€).

  1. Une autre manière de voir cet argument est d’observer que la valeur du dénominateur est de toute façon arbitraire. Il catégorise votre action comme visant à résoudre un problème dont les limites sont arbitrairement choisies. Le don contre la pauvreté pourrait aussi bien être rattaché au but de rendre les plus heureux les habitants du village ou agit l’ONG. Le choix de devenir végétarien pourrait être reformulé comme visant à aider les poules de votre région plutôt que tous les animaux d’élevage industriel dans le monde. La fraction est alors plusieurs milliers de fois plus grande, mais il n’y a aucune raison de préférer un dénominateur à l’autre. 

June 12, 2017

A better formalism for interpreting confidence intervals

When we take a sample mean, we should think of it as a random variable, and our measured sample mean as a realisation of that random variable. The sample mean is a random variable because it is the result of random sampling. Repeated sampling involves observing repeated realisations of the random variable.

We should think of confidence intervals around this mean as realisations of a random interval, an interval whose bounds are random variables rather than real numbers. This is an attractive formalism because it resolves many confusions around the interpretation of confidence intervals.

Suppose the true population mean is the number . The mean of a random sample from this population is the random variable . Then the random interval

has an approximately 95% probability of containing .

Suppose in our sample takes the realisation and takes the realisation . So an instance of the above random interval is the confidence interval:

The confidence interval either contains or does not contain .

In full, my proposed interpretation schema is:

,

is a realisation of

,

and the probability

.

This formalism has several advantages:

  • robustness: distinguishing random intervals from confidence intervals means it’s much harder to get confused into making an incorrect probabilistic statement about the non-probabilistic object .
  • parsiomy: we express everything we want using only probabilities, random variables, and intervals, three well-understood notions.
  • relevance: our interpretation only involves the objects we actually have (a random interval and a confidence interval). We need not make reference to (hypothetical) repeated sampling.

The ugly and the bad

Unfortunately, my preferred formalism does not appear to be popular. Let me show some of the alternatives I have seen and explain their downsides and how my proposal does better.

1

Oxford department of statistics:

The interval is random, not the parameter. Thus, we talk of the probability of the interval containing the parameter, not the probability of the parameter lying in the interval.

This is the worst example, and is admittedly rarely seen in print. But in speech I’ve seen it used often, even by academics who were trying to explain the correct interpretation of confidence intervals! The problem with this of course is that once you write it down in mathematical language, the probability of the interval containing the parameter is exactly the same object as the probability of the parameter lying in the interval. In our example it is simply . It is equal to 1 or 0.

2

Quantitative Economics lecture notes for Oxford undergraduates:

“Were this procedure to be repeated on multiple samples, the calculated confidence interval (which would differ for each sample) would encompass the true population parameter 95% of the time.”

I don’t like this because:

  • It invokes the clunky counterfactual “were this procedure to be repeated”. What if it’s impossible to take repeated samples? We still want to be able to make statements about our confidence interval.
  • It doesn’t have a clear mathematical formalisation. how do I write “95% of the time” in terms of probabilities?
  • The actual confidence interval we have is nowhere mentioned. For what is supposed to be an interpretation of that object, that’s a little confusing.

My formalism solves these three problems.

3

Wikipedia:

“There is a 90% probability that the calculated confidence interval from some future experiment encompasses the true value of the population parameter.”

Similar complaint here: why do we need to refer to future experiments? We want an interpretation of the confidence interval we actually have.

4

Harvard University:

For this reason, for a 95% CI, we say that we have 95% confidence that the interval will cover the true population mean. We use the term ‘confidence’ instead of probability because although the sample mean is random, the single interval we calculate is fixed. We also cannot talk about the probability that the population mean will lie within a certain interval, since it is also fixed.

This needlessly introduces the new concept of ‘confidence’, which is bound to cause confusion. It’s much better to use probabilities, a concept we already understand and for which we have a formal notation.

June 10, 2017

Consistent Vegetarianism and the Suffering of Wild Animals - Journal of Practical Ethics

A revised version of the essay I wrote for the Uehiro Prize has been published in the Journal of Practical Ethics.

Abstract:

Ethical consequentialist vegetarians believe that farmed animals have lives that are worse than non-existence. In this paper, I sketch out an argument that wild animals have worse lives than farmed animals, and that consistent vegetarians should therefore reduce the number of wild animals as a top priority. I consider objections to the argument, and discuss which courses of action are open to those who accept the argument.
May 25, 2017