In this episode, I am joined by Michael Wellman and Uday Rajan. Michael is a Professor of Computer Science & Engineering at the University of Michigan; and Uday is a Professor of Business Administration and Chair and Professor of Finance and Real Estate at the same institution. Our conversation focuses on the ethics of autonomous trading agents on financial markets. We discuss algorithmic trading, high frequency trading, market manipulation, the AI control problem and more.
Where does the self begin and end? Do I extend no further than my fingertips and toes? Or do I exist beyond my body? Is my self bound up with and constituted by my biology or does it extend into the material artifacts in my environment?
These are inherently philosophical questions, and ones with considerable practical import. Last year, my old laptop computer was destroyed by a virus. I had had the same computer for nearly a decade. I tend to get attached to my stuff. I never backed it up. I used it write my PhD thesis; I kept years worth of calendar appointments and meeting notes stored in its memory banks; I had a vast library of research papers, complete with annotations. I used it every day. It was essential part of my cognitive landscape: I could not work without it. And it was all gone within twenty four hours. It felt like a part of me was erased from history. And yet, that is not how most people would see it. They would argue that it was not part of me that was erased; it was, rather, part of my property. That property is alienable and distinguishable from who I am. So what happened to me was an offence against my property, not an offence against my person.
But it doesn’t feel that way to me. It feels like I was personally assaulted by the virus. What’s more, as I and others grow more and more attached to our technological artifacts, and as we grow more and more dependent on them to participate in everyday life, feelings of this sort can be expected to multiply. Should this change how we conceive of the boundaries between the personal and the proprietary?
In their article ‘Is Having Your Computer Compromised A Personal Assault?’, J. Adam Carter and S. Orestis Palermos take on this very question. They present an argument for an extended account of personal assault, one that would count attacks against certain types of technological artifact as personal assaults. They do so by reference to extended cognition theory. I want to examine their argument in this post.
1. The Argument for Extended Assault
I want to start by setting out the argument for extended assault that they present in the middle of the article. The argument works like this:
(1) Intentional harm to a part of a person which is responsible for her mental and other faculties constitutes personal assault.
(2) Our mental faculties can be partly constituted by external artifacts, so long as these artifacts have been appropriately integrated into our overall cognitive system.
(3) Therefore, having our integrated epistemic artifacts intentionally compromised plausibly qualifies as a case of personal assault.
Obviously, premise (2) is the real centrepiece of this argument. To be persuaded of this premise requires some background in extended cognition theory. I’ll cover that in a moment. Before doing so, however, it is important to give premise (1) its due regard.
As the authors themselves note, premise (1) is not an uncontroversial definition of personal assault. Many legal systems recognise a set of offences against the person. These offences are graded in terms of their seriousness In England and Wales, for example, the following five offences form the backbone of the system of offences against the person:
Assault: The reckless or intentional causing of another to apprehend the imminent use of unlawful force.
Battery: The reckless or intentional application of unlawful force to another.
Assault Occasioning Bodily Harm: A battery or assault that causes actual bodily harm to another.
Malicious Wounding or GBH: The unlawful and malicious wounding or inflicting of grievous bodily harm against another.
Intentional Wounding or GBH: The unlawful and intentional wounding or causing of grievous bodily harm to another.*
The three most serious of these offences all explicitly require the use of force against the body and not just the person. ‘Wounding’, for instance, is defined as the breaking of the layers of the skin. And grievous bodily harm is usually clarified by reference to paradigmatic cases such as the breaking of bones or the rupturing of internal organs. The two less serious offences (assault and battery) are a little bit more ambiguous in their definitions, but they are usually interpreted as requiring some application (or threat of application) of force to the body. This creates a problem for the argument for extended assault because it suggests that the current approach to assault (in England and Wales at least) is very much body-centric. It will require quite the conceptual shift to move us away from this body-centric view.
Or will it? Carter and Orestis beg to differ. They suggest that the current approach to assault is more open to extension than might first appear to be the case. And there is something to this. Current approaches to assault occasioning bodily harm already include the infliction of a recognised mental illness within their scope. Since the mental realm arguably does extend beyond the walls of the body, this would seem to allow for extended assault. Tampering with the objects in someone’s environment could cause them considerable mental distress and so could be punishable as a form of assault under the existing approach. Furthermore, as the authors point out, extended assault is not ruled out by leading philosophical conceptions of personhood. They point, in particular, to Charles Taylor, who argued in his 1991 work The Ethics of Authenticity that we should identify the person with their biological constitution.
All of this is to say that premise (1) might appear to stretch the boundaries of existing approaches to assault, but it may not stretch them all that far, and may simply continue an expansion that is already well under way.
What about premise (2)?
2. A Primer on Extended Cognition Theory
To understand premise (2) we need to get into some of the intricacies of extended cognition theory. I’ve looked at aspects of it before, focusing in particular on Clark and Chalmers ‘extended mind’ hypothesis and the criticisms thereof. I’ll be briefer here. Carter and Orestis don’t try to defend extended cognition theory in their article. They merely offer an overview of it and consider what follows if it is true.
Let’s start with an example. Suppose you want to solve a mathematical problem involving the multiplication of two numbers (467 x 35) for example. If you are remarkably gifted (or have learned some of the mental techniques for doing so) you might be able to solve this problem in your head. If you are like the rest of us, you will rely on some external artifacts to help you solve the problem. Assuming you don’t have a calculator (and that’s an assumption we’ll work with for now) you’ll probably grab a pen and paper and solve the problem using the techniques you learned at school. So, first you’ll put something like this down on the page:
467
x 35
______
And then you’ll solve the problem by multiplying 7, 6 and 4 by 5, in that order, carrying over and writing down the answers under the line. You should get ‘2335’. You’ll then skip a line, write in ‘0’ and multiply 7, 6, and 4 by 3 (carrying over and writing down the answers as you go). You should get ‘14010’. You’ll then add those two numbers together to give you the answer of ’16345’.
Now think about how you solved the problem in more detail. You performed a reasonably complex cognitive operation (the multiplication of two numbers) but you didn’t do so inside your own head. Instead, you formed an extended cognitive loop with a set of artifacts in your environment (the pen and paper). It was that loop that performed the cognitive operation.
The essence of extended cognition theory is that this kind of thing happens quite often. Cognition is rarely a purely brain-bound phenomenon. It is, instead, an extended phenomenon - something that happens through a collaboration between a biological agent and the environment in which they live.
But surely that doesn’t mean that the pen and paper you used to solve the multiplication problem are part of you? If someone destroyed that pen and paper you would hardly be driven to say that it amounted to a personal assault? Not all cognitive loops are equal. Sometimes brain-based cognition is just ‘embedded’ in an environment: the brain performs a cognitive operation in collaboration with some set of artifacts, but the two systems remain ontologically distinct. On other occasions, the collaboration between the brain and the environment is far more intimate and interdependent. On those occasions, you no longer have two distinct systems: you have one. This is when the brain and the set of artifacts become ‘integrated’ (or truly ‘extended’). It is only on those occasions that the concept or idea of extended personhood becomes more compelling.
How can we tell the difference? Carter and Orestis (and others) appeal to a standard drawn from dynamical systems theory:
DST standard for extension: For two systems to give rise to an overall extended (coupled) system there must be non-linear interactions/interdependency (via feedback loops) between the parts of those system.
Complement: If there is only linear dependence between the two systems, there is not one overall extended system .
Why is this a good standard? Two reasons are singled out by the authors. The first is that when systems are coupled through non-linear interactions it gives rise to emergent phenomena (i.e. new properties emerge that would not emerge if the systems were not coupled) and second, it becomes impossible to decompose the inputs and outputs of the system into the two original subsystems. In the latter sense, it becomes practically and conceptually necessary to posit one system rather than two. This only really happens when there is continuous and reciprocal interaction between the two systems. This means there is a pretty high standard when it comes to claiming that you form an extended cognitive system with objects in your environment.
But maybe it’s not that high. Would the pen and paper example given earlier count? Carter and Orestis suggest that it would:
[O]ur everyday interaction with telescopes, microscopes, laptops, smartphones and pen and paper when solving complex mathematical problems, the way blind people interact with their canes and Tactile Visual Substitution Systems would all qualify as genuine cases of cognitive extension. In such cases, the completion of the relevant cognitive task does involve dense feedback loops between the individual agent and her artifact. According, in the lift of DST, in such cases, we may indeed have to talk of the presence of an overall extended cognitive system that consists of both the individual agent and her artifact.
That seems to suggest that if I snatch away your pen and paper I am (possibly) guilty of an extended personal assault. Is that not a counterintuitive result? Do we really want to push the concept of extended assault that far?
3. Is Extended Assault Too Counterintuitive?
This is probably the main criticism one can have of the argument for extended assault. If we really do form closely coupled systems with cognitive artifacts like laptops, and pens and paper, then personal assault is a pervasive phenomenon. Far more pervasive than we currently realise. It seems like more work needs to be done to convince us to be so promiscuous in the application of the concept.
One thing that might help in this regard is to highlight the expansive nature of existing, legally-recognised forms of assault, as well as some of the safety mechanisms used to limit that expansion. Look back to the definitions I gave earlier on of assault and battery. These definitions are pretty expansive on their face. An assault is anything that causes you to apprehend the application of force; a battery is anything involving the application of force. In neither case does there have to be harm in order for the offence to occur. This means that assault and battery are pervasive phenomena — they happen all the time, perhaps on a daily basis. This doesn’t result in the absurdly promiscuous prosecution of assault and battery because: (a) most people ignore minor infractions and don’t bring them to the attention of the authorities; (b) the authorities don’t bother to pursue and prosecute minor infractions; and (c) the courts limit the scope of the offences by focusing on ‘unlawful’ force, not all types of force. This latter modification is particularly important in practice because it prevents everyday bumps (e.g. in a crowded train) from counting. Presumably, similar exclusion mechanisms could operate in the case of extended assault. Not every minor interference with integrated cognitive artifacts would have to count.
There are, however, some other concerns people might have about how we integrate with physical artifacts that make assault against those artifacts quite different from assault against the body. Take my opening example of the laptop that was destroyed by the virus. It’s probably worth saying now that the incident I described never actually happened. It was just a story. The reality is that I back-up most of my important files and programs to the ‘cloud’. So even if my physical laptop was destroyed, I wouldn’t lose the important things that enable me to function on a daily basis. I’d still argue that I am cognitively integrated with the laptop, but I can easily replace the laptop and retain the same level of cognitive integration. In that case, would we really want to call the destruction of the laptop ‘personal assault’? An objection emerges here:
Fungibility Objection: Even if we form dynamically integrated systems with certain cognitive artifacts, many of those artifacts are readily replaceable/fungible if destroyed (i.e. you can substitute in an equivalent artifact and retain the same functionality). It seems wrong to label the destruction of such readily fungible artifacts a type of personal assault.
This is an intuitively appealing objection, but as Carter and Orestis point out it has a number of weaknesses. First, it may not be the case that the artifact is readily fungible. Some people might create or customise artifacts so that they are highly personalised and non-fungible. Second, accepting fungibility as a block on the application of assault could have troubling consequences. Imagine in the future that prosthetic limbs are widely available and are equivalent (possibly superior) in their functionality to biological limbs. This means that our biological limbs are readily replaceable/fungible. If someone hacked off your leg in this world, would it still count as assault? If we embrace the fungibility standard, it wouldn’t. Surely that’s a counterintuitive result? Finally, if cloud-based storage is what prevents the destruction of a laptop from being a form of personal assault, it raises the thorny question about the status of cyberattacks on those cloud-based systems. If your files and programs are deleted from the cloud, should that count as personal assault? Possibly, if you accept extended cognition theory.
Okay, that’s it for this post. Hopefully, this is enough to convince you that the concept of extended assault is, at the very least, worthy of greater scrutiny. I’ve only really scratched the surface of the issues in this post. I’d recommend reading the full paper for more. In particular, I’d recommend reading the two interesting case studies (involving actual legal trials) that the authors use to illustrate the concept in more detail.
* Students of criminal law will know that there a little bit more to this offence than I am letting on in this definition. I’m ignoring the complications associated with resisting arrest for the purposes of this discussion.
I have a new paper. It appears as a chapter in the book Surviving the Machine Age, which is edited by Kevin LaGrandeur and James Hughes. The book is, I believe, unique in how it brings together several different perspectives on what should and will happen to society in an era of rampant technological unemployment. It's a little bit pricy, but I would recommend it for purchase by university libraries and the like.
Details about my chapter are below, along with links to a pre-publication draft.
Title: Building a Postwork Utopia: Technological Unemployment, Life Extension and the Future of Human Flourishing
Abstract: Populations in developed societies are rapidly aging: fertility rates are at all-time lows while life expectancy creeps ever higher. This is triggering a social crisis in which shrinking youth populations are required to pay for the care and retirements of an aging majority. Some people argue that by investing in the right kinds of lifespan extension technology – the kind that extends the healthy and productive phases of life – we can avoid this crisis (thereby securing a ‘longevity dividend’). This chapter argues that this longevity dividend is unlikely to be paid if lifespan extension coincides with rampant technological unemployment. This does not mean that we should not pursue lifespan extension, but it does mean that the argument in its favor needs to rest on other grounds. After articulating these grounds, the chapter proceeds to consider the implications this has for our vision of the extended life, postwork utopia. It argues that this vision may need to be reconceived and suggests that one plausible reconception involves prioritizing the role of games in the well-lived life.
Free will skepticism (FWS) is the view that free will does not exist, at least not in the sense required for basic moral desert. In other words, it is the view that no one is responsible for what they do in the sense that they deserve punishment and blame for what they do. That, at any rate, is how FWS is characterised by Derk Pereboom in his book Free Will, Agency and Meaning in Life. Most of that book is dedicated to defending FWS and addressing its implications for our systems of responsibility and punishment. This is understandable given that those are the practical applications most widely-discussed in the literature on free will and moral responsibility.
But in the final chapter of the book, Pereboom turns his attention to an oftentimes neglected topic: the implications of FWS for personal relationships and meaning. Most of us want to live good lives. We want to have meaningful personal relationships and we want to participate in and contribute to projects of great worth. Many people have the nagging suspicion that free will is essential to those things. If we lack free will, then we do not control our fates. And if we do not control our fates, then surely we must resign ourselves to whatever happens and not take credit for our participation in those things. This leads some people to favour pragmatic or instrumental justifications for the belief in free will. They argue that even if the epistemic grounds for believing in free will are unpersuasive we should still believe in it because the pragmatic stakes are so high: if we cease to believe, much of what we value about our current lives will ebb away.
Pereboom tries to resist this resignation. He argues that FWS need not threaten personal relationships and meaning. In this post, I want to examine how he defends this point of view. I do so by focusing on his arguments pertaining to the role of free will in meaningful personal relationships.
1. The Role of Free Will in Meaningful Personal Relationships
The alleged connection between free will and meaningful achievement is easy to understand; the connection between free will and meaningful personal relationships is a little bit more opaque. The classic defence of the connection comes from the work of Peter Strawson. Strawson was famous for his ‘reactive attitudes’ account of moral responsibility. According to this view, the essence of moral responsibility lies in the expression of reactive attitudes (e.g. resentment, indignation, anger, disappointment etc.). Ascriptions of responsibility justify the expression of these attitudes. What’s more, these reactive attitudes are important precisely because they are bound-up with our interpersonal relationships. We need to express them because we care so much about what other people have done to us.
Some of these reactive attitudes seem to require moral desert in order for them to be justifiably directed at another person. Resentment and indignation, in particular, seem to require this. If you wrong me by betraying my trust, I may resent you for it, but my resentment would be inapt if you do not deserve moral blame for what you did. This creates a problem for the proponent of FWS. They don’t believe in moral desert and so it seems like they can never justifiably feel resentment or indignation towards another for what they have done. Is that a big deal? Some argue that it is because resentment and indignation are integral to meaningful relationships. Here’s the argument:
(1) In order to have meaningful personal relationships you must have the potential to justifiably express resentment and indignation towards your relationship partner.
(2) In order to justifiably express resentment and indignation towards another moral desert must exist.
(3) On FWS, moral desert does not exist.
(4) Therefore, on FWS, we cannot have meaningful personal relationships.
Most of the argumentative action centres on premise (1). Pereboom wants to argue that meaningful personal relationships do not require the potential for resentment and indignation. His strategy is twofold: he first wants to argue that these emotions can cause a great deal of harm (as well as any supposed good) and he then wants to argue that there are other emotions that play a similar role in interpersonal relationships and do not require moral desert. Defenders of the argument want to resist both claims.
Let’s look at how one might defend premise (1). It does seem odd to claim that resentment and indignation are integral to meaningful personal relationships. After all, these are emotions that only need to be expressed when things are going badly, i.e. when your relationship partner has wronged you in some way. Furthermore, emotions of this sort often have long-term destructive potential. Resentment often builds up over time and reveals itself in highly toxic interactions. These interactions can be sufficient to bring a relationship to a crashing end. Surely a flourishing personal relationship would be devoid of these emotions?
Defenders of premise (1) try to offer a subtle defence of the charms of resentment and indignation. One way they do this is to argue that resentment and indignation are essential motivators towards justice. It is only when we feel these powerful emotions that we really desire to right a wrong that has been done. Another, and possibly more interesting way, is to suggest that resentment and indignation are integral to something else that is essential to a meaningful personal relationship. Seth Shabo follows this path by arguing that resentment and indignation are essential to emotional vulnerability and that emotional vulnerability is, in turn, an essential part of a mature, loving relationship.
This is an intriguing idea. Vulnerability is often thought to increase the value of a practice. For example, suppose you like to go running every day. Over time you build up a ‘running streak’ (i.e. a number of continuous days of running). That running streak gets both more valuable and more vulnerable over time. If you continue to run every day, its value is maintained, but if you fail to go running just one day, you will break the streak and its value will dissipate. The streak is highly precarious and highly valuable. Shabo is suggesting that personal relationships are somewhat similar. We invest a lot of emotional energy and trust in them. This makes them highly valuable and highly vulnerable. If our relationship partner betrays our trust, our emotional investment is (at least partly) bankrupted; but if we didn’t invest all that emotional energy, the relationship would not be so valuable. In short, we have to make a risky investment, to gain the real value; and so we have to open ourselves up to the potential of resentment and indignation to have a meaningful relationship. Shabo puts it like this:
It is plausible to think that an ordinary susceptibility to hurt feelings is at least characteristic of the sort of emotional investment required for mature love. One important reason for this is that mature love involves a significant degree of emotional vulnerability to the other…And, the thought continues, hurt feelings often beget resentment; for, like resentment, they are a reaction to the sense that one has been treated rudely, inconsiderately, disrespectfully, callously and so on.
(5) Emotional vulnerability is essential to a meaningful personal relationship.
(6) Emotional vulnerability requires susceptibility to resentment and indignation.
(1) [Therefore], in order to have a meaningful personal relationship you must have the potential to justifiably express resentment and indignation toward your relationship partner.
2. Resisting Resentment and Indignation
Is this line of argument any good? Should the FWS resign themselves to less meaningful and fulfilling personal relationships as a result of their dismissal of free will? As mentioned above, Pereboom thinks not. He accepts that justifiably directing resentment and indignation towards another requires moral blame, but he resists the claim that rejecting this prevents us from having meaningful personal relationships. He has two main arguments. I’ll give them names for ease of references. The first is the 'hidden costs' argument:
(7) Hidden Costs: While the expression of resentment and indignation might have some positive value it also has significant disvalue and this is often ignored by its proponents.
I sketched this argument earlier on. The idea underlying it is that resentment and indignation can actually be destructive of personal and relational well-being. Pereboom puts it like this:
But expression of resentment and indignation is apt to have harmful effects. It often fails to contribute to the well being of those to whom it is directed. Frequently it is intended to cause physical or emotional pain, and can give rise to destructive resistance instead of reconciliation. As a result, it has the potential to damage or destroy relationships.
While it is probably impossible to conduct a comprehensive cost-benefit analysis of the consequences of resentment and indignation, this argument does smack of plausibility and does force us to reconsider the pro-resentment/indignation reasoning. Pereboom then adds to that another argument:
(8) Substitute emotions: Even if resentment and indignation are not justified under FWS, there are other possible emotions that perform similar or equally valuable functions, e.g. sorrow, sadness, disappointment etc.
The idea here is that resentment and indignation are supposed to perform certain functions within a relationship: communicate one’s upset or shock at what another has done; send them a signal that their behaviour needs to change; release tension; and so on. Pereboom is simply claiming that similar functions can be performed by other emotions. It certainly seems plausible to claim this (though the dividing line between different emotions can be difficult to draw).
The combination of these two arguments provides a robust defence of the FWS position, but it does not respond to Shabo’s more sophisticated argument in favour of resentment and indignation. That argument made a special appeal to the role of emotional vulnerability in loving relationships and claimed that susceptibility to resentment and indignation was important if one was to enter into that state of vulnerability. Pereboom’s responds to this by introducting a counterexample:
(9) Parent-child counterexample: “Very commonly, teenagers go through a period when they have attitudes of disregard and disrespect for parents, expression of which can result in deeply hurt feelings. But often such expressions of disregard and disrespect do not occasion the parents’ resentment but rather their disappointment and sadness… [these emotions] are…manifestations of vulnerability on the part of the parent. Crucially, they are also personal, since the teenager’s attitudes toward his parent matter to them in their own right…
This looks like a persuasive counterexample to me. It points to a particular type of relationship — one that is intensely personal and meaningful to many people — that involves expressions of emotional vulnerability, but does not require resentment and indignation. This is as it should be. I think many people would argue that being resentful or indignant at your child’s behaviour is highly inappropriate. If this relationship can survive, and thrive, without resentment and indignation, it’s not clear why other relationships cannot do the same. Thus the attempt to bind emotional vulnerability to resentment and indignation is weaker than it first appears.
This, however, points to another way in which to understand the objections to the FWS position. Perhaps the critics are not really concerned with the justifiability of particular reactive attitudes and their importance in interpersonal relationships; perhaps they are really concerned with their practical necessity. Maybe it's the case that our innate (or cultural) drives to resentment and indignation are too powerful too resist? Maybe it’s not possible for people to have relationships without them? This casts the FWS appeal to substitute emotions into doubt.
(10) Impracticality: People have powerful drives toward resentment and indignation; it is not possible for them to rely on substitute emotions alone.
This is obviously a very different style of critique to the ones considered above (though, note, this is probably closer to what Shabo was arguing in his article about love and resentment). It moves us away from the normative debate about what is needed for meaningful interpersonal relationships into a debate about what is practically possible. Given that FWS is grounded in a sensitivity to what is and is not possible for creatures like us, it should come as no surprise to find that this objection is not particularly troublesome for the proponent of that view. Pereboom is a fan of what Shabo calls the ‘containment policy’. This policy accepts that we may feel resentful or indignant on occasion; it accepts that we are not morally perfect creatures; it simply argues that we should take steps to minimise those feelings (and substitute in alternative emotions) and not afford them any social or moral legitimacy when we do experience them. The goal of the FWS proponent is thus not to demand the impossible, but to encourage us to take steps toward a more positive grounding for our interpersonal relationships.
(11) Containment policy: FWS does not demand the impossible: we may continue to feel resentment and indignation. We should simply take steps to minimise those feelings and not afford them any social or moral legitimacy.
That brings us to the end of this post. There is more to be said, of course. Pereboom’s discussion of these issues is more nuanced than I am able to convey in this summary. Hopefully, what I have written gives a general sense of his position and how it is defended. Suffice to say, I find it generally persuasive. Although I may not fully embrace FWS (I’m somewhat on the fence), I do think that the case made for reactive attitudes like resentment and indignation is quite weak. Indeed, I'm always surprised by people who think that emotions such as indignation or resentment are positive and humane. Would they say the same about jealousy and rage? If not, why not? My sense, as someone who experiences these emotions from time to time, is that they destroy far more than they nourish and sustain. To suggest that they play an essential part in the well-lived life is deeply counterintuitive to me.
Patrick Lin started it. In an article entitled ‘The Ethics of Autonomous Cars’ (published in The Atlantic in 2013), he considered the principles that self-driving cars should follow when they encountered tricky moral dilemmas on the road. We all encounter these situations from time to time. Something unexpected happens and you have to make a split second decision. A pedestrian steps onto the road and you don’t see him until the last minute: do you slam on the brakes or swerve to avoid? Lin made the obvious point that no matter how safe they were, self-driving cars would encounter situations like this, and so engineers would have to design ‘crash-optimisation’ algorithms that the cars would use to make those split second decisions.
In a later article Lin explained the problem by using a variation on the famous ‘trolley problem’ thought experiment. The classic trolley problem asks you to imagine a trolley car hurtling out of control down a railroad track. If it continues on its present course, it will collide with and kill five people. You can, however, divert it onto a sidetrack. If you do so, it will kill only one person. What should you do? Ethicists have debated the appropriate choice for the last forty years. Lin’s variation on the trolley problem worked like this:
Imagine in some distant future, your autonomous car encounters this terrible choice: it must either swerve left and strike an eight-year old girl, or swerve right and strike an 80-year old grandmother. Given the car’s velocity, either victim would surely be killed on impact. If you do not swerve, both victims will be struck and killed; so there is good reason to think that you ought to swerve one way or another. But what would be the ethically correct decision? If you were programming the self-driving car, how would you instruct it to behave if it ever encountered such a case, as rare as it may be?
There is certainly value to thinking about problems of this sort. But some people worry that, in focusing on individualised moral dilemmas such as this, the framing of the ethical challenges facing the designers of self-driving cars is misleading. There are important differences between the moral choice confronting the designer of the crash optimisation system (whether it be programmed from the top-down with clearly prescribed rules or the bottom-up using some machine-learning system) and the choices faced by drivers in particular dilemmas. Recently, some papers have been written drawing attention to these differences. One of them is Hin-Yan Liu’s ’Structural Discrimination and Autonomous Vehicles’. I just interviewed Hin-Yan for my podcast about this and other aspects of his research, but I want to take this opportunity to examine the argument in that paper in more detail.
1. The Structural Discrimination Problem
Liu’s argument is that the design of crash optimisation algorithms could lead to structural discrimination (note: to be fair to him, Lin acknowledged the potential discriminatory impact in his 2016 paper).
Structural discrimination is a form of indirect discrimination. Direct discrimination arises where some individual or organisation intentionally disadvantages someone because they belong to a particular race, ethnic group, gender, class (etc). Once upon a time there were, allegedly, signs displayed outside pubs, hotels and places of employment in the UK saying ‘No blacks, No Irish’. The authenticity of these signs is disputed, but if they really existed, they would provide a clear example of direct discrimination. Indirect discrimination is different. It arises where some policy or practice has a seemingly unobjectionable express intent or purpose but nevertheless has a discriminatory impact. For example, a hairdressing salon that had a policy requiring all staff to show off their hair to customers might have discriminatory impact on (some) potential Muslim staff (I took this example from Citizen’s Advice UK).
Structural discrimination is a more generalised form of indirect discrimination whereby entire systems are set up are structured in such a way that they impose undue burdens on particular groups. How might this happen with crash optimisation algorithms? The basic argument works like this:
(1) If a particular rule or policy is determined with reference to factors that ignore potential forms of discrimination, and if that rule is followed in the majority of circumstances, it is likely to have an unintended structurally discriminatory impact.
(2) The crash optimisation algorithms followed by self-driving cars are (a) likely to be determined with reference to factors that ignore potential forms of discrimination and (b) are likely to be followed in the majority of circumstances.
(3) Therefore, crash optimisation algorithms are likely to have an unintended discriminatory impact.
The first premise should be relatively uncontroversial. It is making a probabilistic claim. It is saying that if so-and-so happens it is likely to have a discriminatory impact, not that it definitely will. The intuition here is that discrimination is a subtle thing. If we don’t try to anticipate it and prevent it from happening, we are likely to do things that have unintended discriminatory effects. Go back to the example of the hairdressing salon and the rule about uncovered hair. Presumably, no one designing that rule thought they were doing anything that might be discriminatory. They just wanted their staff to show off their hair so that customers would get a good impression. They didn’t consciously factor in potential forms of bias or discrimination. This is what created the potential for discrimination.
The first part of premise one is simply saying that what is true in the case of the hair salon is likely to be true more generally. Unless we consciously direct our attention to the possibility of discriminatory impact, it will be sheer luck whether we avoid it. That might not be too problematic if the rules we designed were limited in their application. For example, if the rule about uncovered hair for staff only applied to one particular hairdressing salon, then we have some problem but it would fall far short of structural discrimination. There would be discrimination in the particular salon, but that discrimination would not spread across society as whole. Muslim hairdressers would not be excluded from work at all salons. It is only when the rule is followed in the majority of cases that we get the conditions in which structural discrimination can breed.
This brings us to premise two. This is the critical one. Are there any reasons to accept it? Looking first to condition (a), there are indeed some reasons to believe that this will be the case. The reasons have to do with the ‘trolley problem’-style framing of the ethical challenges facing the designers of self-driving cars. That framing encourages us to think about the morally optimal choice in a particular case, not at a societal level. It encourages us to pick the least bad option, even if that option contravenes some widely-agreed moral principle. A consequentialist, for example, might resolve the granny vs. child dilemma in favour of the child based on the quantity of harm that will result. They might say that the child has more potentially good life years ahead of them (possibly justifying this by reference to the QALY standard) and hence it does more good to save the child (or, to put it another way, less harm to kill the granny). The problem with this reasoning is that in focusing purely on the quantity of harm we ignore factors that we ought to consider (such as the potential for ageism) if we wish to avoid a discriminatory impact. As Liu puts it:
[A]nother bling spot of trolley problem ethics…is that the calculus is conducted with seemingly featureless and identical “human units”, as the variable being emphasised is the quantity of harm rather than its character or nature.
We could try to address this problem by getting the designers of the algorithms to look more closely at the characteristics of the individuals that might be affected by the choices made by the cars, but this will then lead us to the second problem, namely the fact that whatever solution we hit upon is likely to be multiplied and shared across many self-driving cars, and that multiplication and sharing is likely to exacerbate any potentially discriminatory effect. Why is this? Well, presumably car manufacturers will standardise the optimisation algorithms they offer on their cars (not least because the software that actually drives the car is likely to be cloud-based and to adapt and learn based on the data collected from all cars). This will result in greater homogeneity in how cars respond to trolley-problem like dilemmas, which will in turn increase any potentially discriminatory effect. For example, if an algorithm does optimise by resolving the dilemma in favour of the child, we get a situation in which all cars using that algorithm favour children over grannies, and so an extra burden is imposed on grannies across society as a whole. They face a higher risk of being killed by a self-driving car.
There are some subtleties to this argument that are worth exploring. You could reject it by arguing that there will still presumably be some diversity in how car manufacturers optimise their algorithms. So, for example, perhaps all BMWs will be consequentialist in their approach whereas all Audis will be deontological. This is likely to result in a degree of diversity but perhaps much less diversity than we currently have. This is what I think is most interesting about Liu’s argument. In a sense, we are all running crash-optimisation algorithms in our heads right now. We use these algorithms to resolve the moral dilemmas we face while driving. But as various experiments have revealed, the algorithms humans use are plural and messy. Most people have intuitions that make them lean in favour of consequentialist solutions in some cases and deontological solutions in others. Thus the moral choices made at an individual level can shift and change across different contexts and moods. This presumably creates great diversity at a societal level. The differences across the different car manufacturers is likely to be more limited.
This is, admittedly, speculative. We don’t know whether the diversity we have right now is so great that it avoids any pronounced structural discrimination in the resolution of moral dilemmas. But this is what is interesting about Liu’s argument: It make an assumption about the current state of affairs (namely that there is great diversity in the resolution of moral dilemmas) that might be true but is difficult to verify until we enter a new state of affairs (one in which self-driving cars dominate the roads) and see whether there is a greater discriminatory impact or not. Right now, we are at a moment of uncertainty.
Of course, there might be technical solutions to the structural discrimination problem. Perhaps, for instance, crash optimisation algorithms could be designed with some element of randomisation, i.e. they randomly flip back-and-forth between different moral rules. This might prevent structural discrimination from arising. It might seem odd to advocate moral randomisation as a solution to the problem of structural discrimination, but perhaps a degree of randomisation is one of the benefits of the world in which we currently live.
2. The Immunity Device Thought Experiment
There is another nice feature to Liu’s paper. After setting out the structural discrimination problem, he introduces a fascinating thought experiment. And unlike many philosophical thought experiments, this is one that might make the transition from thought to reality.
At the core of the crash optimisation dilemma is a simple question: how do we allocate risk in society? In this instance the risk of dying in a car accident. We face many similar risk allocation decisions already. Complex systems of insurance and finance are set up with the explicit goal of spreading and reallocating these risks. We often allow people to purchase additional protection from risk through increased insurance premiums, and we sometimes allocate/gift people extra protections (e.g. certain politicians or leaders). Might we end up doing the same thing when it comes to the risk of being struck by a self-driving car? Liu asks us to imagine the following:
Immunity Device Thought Experiment:‘It would not be implausible or unreasonable for the manufacturers of autonomous vehicles to issue what I would call here an “immunity device”: the bearer of such a device would become immune to collisions with autonomous vehicles. With the ubiquity of smart personal communication devices, it would not be difficult to develop a transmitting device to this end which signals the identity of its owner. Such an amulet would protect its owner in situations where an autonomous vehicle finds itself careening towards her, and would have the effect of deflecting the care away from that individual and thereby divert the car to engage in a new trolley problem style dilemma elsewhere.'
(Liu 2016, 169)
The thought experiment raises a few important and interesting questions. First, is such a device technically feasible? Second, should we allow for the creation of such a device? And third, if we did, how should we allocate the immunity it provides?
On the first question, I agree with what Liu says. It seems like we have the underlying technological infrastructure that could facilitate the creation of such a device. It would be much like any other smart device and would simply have to be in communication with the car. There may be technical challenges but they would not be insurmountable. There is a practical problem if everybody managed to get their hands on an immunity device: that would, after all, defeat the purpose. But Liu suggests a work around to this: have a points-based (trump card) rating system attached to the device. So people don’t get perfect immunity; they get bumped up and down a ranking order. This changes the nature of the allocation question. It’s no longer who should get such a device but, rather, how the points should be allocated.
On the second question, I have mixed views. I feel very uncomfortable with the idea, but I can’t quite pin down my concern. I can see some arguments in its favour. We do, after all, have broadly analogous systems nowadays whereby people get additional protection through systems of social insurance. Nevertheless, there are some important disanalogies between what Liu imagines and other forms of insurance. In the case of, say, health insurance, we generally allow richer people to buy additional protection in the form of higher premiums. This can have negative redistributive consequences, but the gain to the rich person does not necessarily come at the expense of the poorer person. Indeed, in a very real sense, the rich person’s higher premium might be subsidising the healthcare of the poorer person. Furthermore, the protection that the rich person buys may never be used: it’s there as peace of mind. In the case of the immunity device, it seems like the rich person buying the device (or the points) would necessarily be doing so at the expense of someone else. After all, the device provides protection in the event of a self-driving car finding itself in a dilemma. The dilemma is such that the car has to strike someone. If you are buying immunity in such a scenario it means you are necessarily paying for the car to be diverted so that it strikes someone else. This might provide the basis for an objection to the idea itself: this is something that we possibly should not allow to exist. The problem with this objection is that it effectively applies the doctrine of double effect to this scenario, which is not something I am not comfortable with. Also, even if we did ban such devices, we would still have to decide how to allocate the risk burden: at some stage you would have to make a choice as to who should bear the risk burden (unless you adopt the randomisation solution).
This brings us to the last question. If we did allow such a device to be created, how would we allocate the protection it provides. The market-based solution seems undesirable, for the reasons just stated. Liu considers the possibility of allocating points as a system of social reward and punishment. So, for example, if you commit a crime you could be punished by shouldering an increased risk burden (by being pushed down the ranking system). That seems prima facie more acceptable than allocating the immunity through the market. This is for two reasons. First, we are generally comfortable with the idea of punishment (though there are those who criticise it). Second, according to most definitions, punishment involves the intentional harming of another. So the kinds of concerns I raised in the previous paragraph would not apply to allocation-via-punishment: if punishment is justified at all then it seems like it would justify the intentional imposition of a risk burden on another. That said, there are reasons to think that directly harming someone through imprisonment or fine is more morally acceptable than increasing the likelihood of their being injured/killed in a car accident. After all, if you object to corporal or capital punishment you may have reason to object to increasing the likelihood of bodily injury or death.
Okay, that brings us to the end of this post. I want to conclude by recommending Liu's paper. We discuss the ideas in it in more detail in the podcast we recorded. It should be available in a couple of weeks. Also, I should emphasise that Liu introduces the Immunity Device as a thought experiment. He is definitely not advocating its creation. He just thinks it helps us to think through some of the tricky ethical questions raised by the introduction of self-driving cars.
In this episode, I talk to Mark Coeckelbergh. Mark is a Professor of Philosophy of Media and Technology at the Department of Philosophy of the University of Vienna and President of the Society for Philosophy and Technology. He also has an affiliation as Professor of Technology and Social Responsibility at the Centre for Computing and Social Responsibility, De Montfort University, UK. We talk about robots and philosophy (robophilosophy), focusing on two topics in particular. First, the rise of the carebots and the mechanisation of society, and second, Hegel's master-slave dialectic and its application to our relationship with technology.
Polynesian sailors developed elaborate techniques for long-distance sea travel long before their European counterparts. They mapped out the elevation of the stars; they followed the paths of migrating birds; they observed sea swells and tidal patterns. The techniques were often passed down from generation to generation through the medium of song. They are still taught to this day (in some locations). In 1976, there was a famous proof of their effectiveness when Mau Piailug, a practitioner of the techniques, steered a traditional sailing canoe nearly 3,000 miles from Hawaii to Tahiti without relying on more modern methods of navigation.
These Polynesian sailing techniques provide a perfect real-world illustration of distributed cognition theory. According to this theory, cognition is not something that takes place purely in the head. When humans want to perform cognitive tasks, they don’t simply represent and manipulate the cognition-relevant information in their brains, they also co-opt features of their environment to assist them in the performance of cognitive tasks. In the case of the Polynesian sailors, it was the migrational patterns of birds, the movements of the sea and the elevation of the stars that assisted the performance. It was also the created objects and cultural products (e.g. songs) that they used to help to offload the cognitive burden and transmit the relevant knowledge down through the generations. In this manner, the performance of the cognitive task of navigation became distributed between the individual sailor and the wider environment.
Generally speaking, there are three features of the external environment that can assist in the performance of a cognitive task:
Cognitive Artifacts: Intentionally designed objects that are used in the performance of the task, e.g. a map, a calendar, an abacus, or a textbook.
Naturefacts: Natural objects, events or states of affairs that get co-opted into the performance of a cognitive tasks, e.g. the paths of migrating birds and the elevation of the stars.
Other Cognitive Agents: Other humans (or, possibly, robots and AI) that can perform cognitive tasks in collaboration/cooperation with one another.
I think it is important to understand how all three of these cognitive-assisters function and to appreciate some of the qualitative differences between them. One thing that distributed cognition theory enables you to do is to appreciate the complex ecology of cognition. Because cognition is spread out across the agent and its environment, the agent becomes structurally coupled to that environment. If you tamper with or alter one part of the external cognitive ecology it can have knock-on effects elsewhere within the system, changing the kinds of cognitive task that need to be performed, and altering the costs/benefits associated with different styles of cognition (I discussed this, to some extent, in a previous post). Understanding how the different cognitive assisters function provides insight into these effects.
In the remainder of this post, I want to take a first step towards understanding the complexity of our cognitive ecology by taking a look at Richard Heersmink’s proposed taxonomy of cognitive artifacts. This taxonomy gives us some insight into one of the three relevant features of our cognitive ecology (cognitive artifacts) and enables us to appreciate how this feature works and the different possible forms it can take.
The taxonomy itself is fairly simple to represent in graphical form. It divides all cognitive artifacts into two major families: (i) representational and (ii) ecological. It then breaks these major families down into a number of sub-types. These sub-types are labelled using a somewhat esoteric conceptual vocabulary. The labels make sense once you have mastered the vocabulary. The remainder of this post is dedicated to explaining how it all works.
1. Representational Cognitive Artifacts
Cognition is an informational activity. We perform cognitive tasks by acquiring, manipulating, organising and communicating information. Consequently, cognitive artifacts are able to assist in the performance of cognitive tasks precisely because they have certain informational properties. As Heersmink puts it, the functional properties of these artifacts supervene on their informational properties. One of the most obvious things a cognitive artifact can do is represent information in different forms.
’Representation’ is a somewhat subtle concept. Heersmink adopts CS Peirce’s classic analysis. This holds that representation is a triadic relation between an object, sign and interpreter. The object is the world that the sign is taken to represent, the sign is that which represents the world, and the interpreter is the one who determines the relation between the sign and the object. To use a simple example, suppose there is a portrait of you hanging on the wall. The portrait is the sign; it represents the object (in this case you); and you are the interpreter. The key thing about the sign is that it stands in for something else, namely the represented object. Signs can represent objects in different ways. Some forms of representation are straightforward: the sign simply looks like the object. Other forms of representation are more abstract.
Heersmink argues that there are three main forms of representation and, as a result, three main types of representational cognitive artifact. The first form of representation is iconic. An iconic representation is one that is isomorphic with or highly similar to the object it is representing. The classic example of an iconic cognitive artifact is a map. The map provides a scaled down picture of the world. The visual imagery on the map is supposed to stand in a direct, one-to-one relation with the features in the real world. A lake is depicted as an blue blob; a forest is depicted as a mass of small green trees, a mountain range is depicted as a series of humps, coloured in different ways to represent their different heights.
The second form of representation is indexical. An indexical representation is one that is causally related to the object it is representing. The classic example of an indexical cognitive artifact would be a thermometer. The liquid within the thermometer expands when it is heated and contracts when it is cooled. This results in a change in the temperature reading on the temperature gauge. This means there is a direct causal relationship between the information represented on the temperature gauge and the actual temperature in the real world.
The third form of representation is symbolic. A symbolic representation is one that is neither iconic nor indexical. There is no discernible relationship between the sign and the object. The form that the sign takes is arbitrary and people simply agree (by social convention) that it represents a particular object or set of objects. Represented language is the classic example of a symbolic cognitive artifact. The shapes of letters and the order in which they are presented bears no direct causal or isomorphic relationship to the objects they describe or name (pictographic or ideographic languages are different). The word ‘cat’, for example, bears no physical similarity to an actual cat. There is nothing about those letters that would tell you that they represented a cat. You simply have to learn the conventions to understand the representations.
The different forms of representation may be combined in any one cognitive artifact. For example, although maps are primarily iconic in nature, they often include symbolic elements such as place-names or numbers representing elevation or distance.
2. Ecological Cognitive Artifacts
The other family of cognitive artifacts are ecological in nature. This is a more difficult concept to explain. The gist of the idea is that some artifacts don’t merely provide representations of cognition-relevant information; rather, they provide actual forums in which information can be stored and manipulated. The favourite example of this — one originally posed by the distributed cognition pioneer David Kirsh — is the game of Tetris. For those who are not familiar, Tetris is a game in which you must manipulate differently shaped ‘bricks’ (technically known as ‘zoids’) into sockets or slots at the bottom of the game screen so that they form a continuous line of zoids. Although you could, in theory, play the game by mentally rotating the zoids in your head, and then deciding how to move them on the game screen, this is not the most effective way to play the game. The most effective way to play the game is simply to rotate the shapes on the screen and see how they will best fit into the wall forming at the bottom of the screen. In this way, the game creates an environment in which the cognition-relevant manipulation of information is performed directly. The artifact is thus its own cognitive ecology.
Heersmink argues that there are two main types of ecological cognitive artifact. The first is the spatial ecological artifact. This is any artifact that stores information in its spatial structure. The idea behind it is that we encode cognition-relevant information into our social spaces, thereby obviating the need to store that information in our heads. A simple example would be the way in which we organise clothes into piles in order to keep track of which clothes have been washed, which need to be washed, which have been dried, and which need to be ironed. The piles, and their distribution across physical space, stores the cognition-relevant information. Heersmink points out that the spaces in which we encode information need not be physical/real-world spaces. They can also be virtual, e.g. the virtual ‘desktop’ on your computer or phonescreen.
The other kind of ecological cognitive artifact is the structural artifact. I don’t know if this is the best name for it, but the idea is that some artifacts don’t simply encode information into physical or virtual space; they also provide forums in which that information can be manipulated, reorganised and computed. The Tetris gamescreen is an example: it provides a virtual space in which zoids can be rearranged and rotated. Another example would be scrabble tiles: constantly reorganising the tiles into different pairs or triplets makes it easier to spot words. The humble pen and paper can also, arguably, be used to create structures in which information can be manipulated and reorganised (e.g. writing out the available letters and spaces when trying to solve a crossword clue).
3. Conclusion
This then is Heersmink’s taxonomy of cognitive artifacts. One thing that is noticeable about it (and this is a feature, not a bug) is that it focuses on the properties of the artifacts themselves, not on human uses. It is, thus, an artifact-centred taxonomy not an anthropomorphic one. Also the taxonomy does not divide the world of cognitive artifacts into a set of jointly exhaustive and mutually exclusive categories. As is clear from the descriptions, particular artifacts can sit within several of the categories at one time.
Nevertheless, I think the taxonomy is a useful one. It sheds light on the different ways in which artifacts can figure in our cognitive tasks, it makes us more sensitive to the rich panoply of cognitive artifacts we encounter in our everyday lives, and it can shed light on the propensity of these artifacts to enhance our cognitive performance. For example, symbolic cognitive artifacts clearly have a higher cognitive burden associated with them. The user must learn the conventions that determine the meaning of the representations before they can effectively use the artifact. At the same time, the symbolic representations probably allow for more complex and abstract cognitive operations to be performed. If we relied purely on iconic forms of representation we would probably never have generated the rich set of concepts and theories that litter our cognitive landscapes.
