'Asimov's Laws of Robotics' - the Laws of Robotics 2013

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMMy purpose here is to determine whether or not Asimov's fiction vindicates the laws he expounded. Does hesuccessfully demonstrate that robotic technology can be applied in a responsible manner to potentiallypowerful, semi-autonomous and, in some sense intelligent machines? To reach a conclusion, we mustexamine many issues emerging from Asimov's fiction.HistoryThe robot notion derives from two strands of thought, humanoids and automata. The notion of a humanoid(or human- like nonhuman) dates back to Pandora in The Iliad, 2,500 years ago and even further. Egyptian,Babylonian, and ultimately Sumerian legends fully 5,000 years old reflect the widespread image of thecreation, with god- men breathing life into clay models. One variation on the theme is the idea of the golem,associated with the Prague ghetto of the sixteenth century. This clay model, when breathed into life, became auseful but destructive ally.The golem was an important precursor to Mary Shelley's Frankenstein: The Modern Prometheus (1818). Thisstory combined the notion of the humanoid with the dangers of science (as suggested by the myth ofPrometheus, who stole fire from the gods to give it to mortals). In addition to establishing a literary traditionand the genre of horror stories, Frankenstein also imbued humanoids with an aura of ill fate.Automata, the second strand of thought, are literally "self- moving things" and have long interested mankind.Early models depended on levers and wheels, or on hydraulics. Clockwork technology enabled significantadvances after the thirteenth century, and later steam and electro- mechanics were also applied. The primarypurpose of automata was entertainment rather than employment as useful artifacts. Although many patternswere used, the human form always excited the greatest fascination. During the twentieth century, several newtechnologies moved automata into the utilitarian realm. Geduld and Gottesman 8 and Frude 2 review thechronology of clay model, water clock, golem, homunculus, android, and cyborg that culminated in thecontemporary concept of the robot.The term robot derives from the Czech word robota, meaning forced work or compulsory service, orrobotnik, meaning serf. It was first used by the Czech playwright Karel Çapek in 1918 in a short story andagain in his 1921 play R. U. R., which stood for Rossum's Universal Robots. Rossum, a fictional Englishman,used biological methods to invent and mass- produce "men" to serve humans. Eventually they rebelled,became the dominant race, and wiped out humanity. The play was soon well known in English- speakingcountries.DefinitionUndeterred by its somewhat chilling origins (or perhaps ignorant of them), technologists of the 1950sappropriated the term robot to refer to machines controlled by programs. A robot is "a reprogrammablemultifunctional device designed to manipulate and/or transport material through variable programmedmotions for the performance of a variety of tasks"9. The term robotics, which Asimov claims he coined in1942 10 refers to "a science or art involving both artificial intelligence (to reason) and mechanical engineering(to perform physical acts suggested by reason)"11.As currently defined, robots exhibit three key elements:http://www.rogerclarke.com/SOS/Asimov.htmlPage 4 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMdeath. It is not for him to decide. He may not harm a human - variety skunk or variety angel." 7 On the otherhand they might not, as when a robot tells a human, "In conflict between your safety and that of another, Imust guard yours." 22 In another short story, robots agree that they "must obey a human being who is fit bymind, character, and knowledge to give me that order." Ultimately, this leads the robot to "disregard shapeand form in judging between human beings" and to recognize his companion robot not merely as human butas a human "more fit than the others." 18 Many subtle problems can be constructed. For example. a personmight try forcing a robot to comply with an instruction to harm a human (and thereby violate the first law) bythreatening to kill himself unless the robot obeys.How is a robot to judge the trade- off between a high probability of lesser harm to one person versus a lowprobability of more serious harm to another? Asimov's stories refer to this issue but are somewhatinconsistent with each other and with the strict wording of the first law.More serious difficulties arise in relation to the valuation of multiple humans. The first law does not evencontemplate the simple case of a single terrorist threatening many lives. In a variety of stories, however,Asimov interprets the law to recognize circumstances in which a robot may have to injure or even kill one ormore humans to protect one or more others: "The Machine cannot harm a human being more than minimally,and that only to save a greater number" 23 (emphasis added). And again: "The First Law is not absolute.What if harming a human being saves the lives of two others, or three others, or even three billion others?The robot may have thought that saving the Federation took precedence over the saving of one life." 24These passages value humans exclusively on the basis of numbers. A later story includes this justification:"To expect robots to make judgments of fine points such as talent, intelligence, the general usefulness tosociety, has always seemed impractical. That would delay decision to the point where the robot is effectivelyimmobilized. So we go by numbers." 18A robot's cognitive powers might be sufficient for distinguishing between attacker and attackee, but the firstlaw alone does not provide a robot with the means to distinguish between a "good" person and a "bad" one.Hence, a robot may have to constrain a "good" attackee's self- defense to protect the "bad" attacker fromharm. Similarly, disciplining children and prisoners may be difficult under the laws, which would limitrobots' usefulness for supervision within nurseries and penal institutions. 22 Only after many generations ofself- development does a humanoid robot learn to reason that "what seemed like cruelty [to a human] might,in the long run, be kindness." 12The more subtle life- and- death cases, such as assistance in the voluntary euthanasia of a fatally ill or injuredperson to gain immediate access to organs that would save several other lives, might fall well outside arobot's appreciation. Thus, the first law would require a robot to protect the threatened human, unless it wasable to judge the steps taken to be the least harmful strategy. The practical solution to such difficult moralquestions would be to keep robots out of the operating theater. 22The problem underlying all of these issues is that most probabilities used as input to normative decisionmodels are not objective; rather, they are estimates of probability based on human (or robot) judgment. Theextent to which judgment is central to robotic behavior is summed up in the cynical rephrasing of the first lawby the major (human) character in the four novels: "A robot must not hurt a human being, unless he can thinkof a way to prove it is for the human being's ultimate good after all." 19http://www.rogerclarke.com/SOS/Asimov.htmlPage 9 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AM* The sheer complexityTo cope with the judgmental element in robot decision making, Asimov's later novels introduced a furthercomplication: "On......[worlds other than Earth], . . . the Third Law is distinctly stronger in comparison to theSecond Law. . . . An order for self- destruction would be questioned and there would have to be a trulylegitimate reason for it to be carried through - a clear and present danger." 16 And again, "Harm through anactive deed outweighs, in general, harm through passivity - all things being reasonably equal. . . . [A robot is]always to choose truth over nontruth, if the harm is roughly equal in both directions. In general, that is."16The laws are not absolutes, and their force varies with the individual machine's programming, thecircumstances, the robot's previous instructions, and its experience. To cope with the inevitable logicalcomplexities, a human would require not only a predisposition to rigorous reasoning, and a considerableeducation, but also a great deal of concentration and composure. (Alternatively, of course, the human mayfind it easier to defer to a robot suitably equipped for fuzzy- reasoning- based judgment.)The strategies as well as the environmental variables involve complexity. "You must not think . . . thatrobotic response is a simple yes or no, up or down, in or out. ... There is the matter of speed of response."16In some cases (for example, when a human must be physically restrained), the degree of strength to beapplied must also be chosen.* The scope for dilemma and deadlockA deadlock problem was the key feature of the short story in which Asimov first introduced the laws. Heconstructed the type of stand- off commonly referred to as the "Buridan's ass" problem. It involved a balancebetween a strong third- law self- protection tendency, causing the robot to try to avoid a source of danger, anda weak second- law order to approach that danger. "The conflict between the various rules is [meant to be]ironed out by the different positronic potentials in the brain," but in this case the robot "follows a circlearound [the source of danger], staying on the locus of all points of ... equilibrium." 5Deadlock is also possible within a single law. An example under the first law would be two humansthreatened with equal danger and the robot unable to contrive a strategy to protect one without sacrificing theother. Under the second law, two humans might give contradictory orders of equivalent force. The laternovels address this question with greater sophistication:What was troubling the robot was what roboticists called an equipotential of contradiction onthe second level. Obedience was the Second Law and [the robot] was suffering from two roughlyequal and contradictory orders. Robot- block was what the general population called it or, morefrequently, roblock for short . . . [or] `mental freeze- out.' No matter how subtle and intricate abrain might be, there is always some way of setting up a contradiction. This is a fundamentaltruth of mathematics. 16Clearly, robots subject to such laws need to be programmed to recognize deadlock and either choosearbitrarily among the alternative strategies or arbitrarily modify an arbitrarily chosen strategy variable (say,move a short distance in any direction) and reevaluate the situation: "If A and not- A are precisely equalmisery- producers according to his judgment, he chooses one or the other in a completely unpredictable wayand then follows that unquestioningly. He does not go into mental freeze- out."16http://www.rogerclarke.com/SOS/Asimov.htmlPage 10 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMhad the laws imposed in precisely the manner intended; andwas at all times subject to them - that is, they could not be overridden or modified.It is important to know how malprogramming and modification of the laws' implementation in a robot(whether intentional or unintentional) can he prevented, detected, and dealt with.In an early short story, robots were "rescuing" humans whose work required short periods of relativelyharmless exposure to gamma radiation. Officials obtained robots with the first law modified so that they wereincapable of injuring a human but under no compulsion to prevent one from coming to harm. This clearlyundermined the remaining part of the first law, since, for example, a robot could drop a heavy weight towarda human, knowing that it would be fast enough and strong enough to catch it before it harmed the person.However, once gravity had taken over, the robot would be free to ignore the danger. 25 Thus, a partialimplementation was shown to be risky, and the importance of robot audit underlined. Other risks includetrapdoors, Trojan horses, and similar devices in the robot's programming.A further imponderable is the effect of hostile environments and stress on the reliability and robustness ofrobots' performance in accordance with the laws. In one short story, it transpires that "The Machine ThatWon the War" had been receiving only limited and poor- quality data as a result of enemy action against itsreceptors and had been processing it unreliably because of a shortage of experienced maintenance staff. Eachof the responsible managers had, in the interests of national morale, suppressed that information, even fromone another, and had separately and independently "introduced a number of necessary biases" and "adjusted"the processing parameters in accordance with intuition. The executive director, even though unaware of theadjustments, had placed little reliance on the machine's output, preferring to carry out his responsibility tomankind by exercising his own judgment. 27A major issue in military applications generally 28 is the impossibility of contriving effective compliance testsfor complex systems subject to hostile and competitive environments. Asimov points out that the difficultiesof assuring compliance will be compounded by the design and manufacture of robots by other robots. 22* Robot autonomySometimes humans may delegate control to a robot and find themselves unable to regain it, at least in aparticular context. One reason is that to avoid deadlock, a robot must be capable of making arbitrarydecisions. Another is that the laws embody an explicit ability for a robot to disobey an instruction, by virtueof the overriding first law.In an early Asimov short story, a robot "knows he can keep [the energy beam] more stable than we [humans]can, since he insists he's the superior being, so he must keep us out of the control room [in accordance withthe first law]." 29 The same scenario forms the basis of one of the most vivid episodes in science fiction,HAL's attempt to wrest control of the spacecraft from Bowman in 2001: A Space Odyssey. Robot autonomyis also reflected in a lighter moment in one of Asimov's later novels, when a character says to his companion,"For now I must leave you. The ship is coasting in for a landing, and I must stare intelligently at the computerthat controls it, or no one will believe I am the captain." 14In extreme cases, robot behavior will involve subterfuge, as the machine determines that the human, for his orhttp://www.rogerclarke.com/SOS/Asimov.htmlPage 12 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMWith their fictional "positronic" brains imprinted with the mandate to (in order of priority) prevent harm tohumans, obey their human masters, and protect themselves, Asimov's robots had to deal with greatcomplexity. In a given situation, a robot might be unable to satisfy the demands of two equally powerfulmandates and go into "mental freezeout." Semantics is also a problem. As demonstrated in Part 1 of thisarticle (Computer, December 1993, pp. 53- 61), language is much more than a set of literal meanings andAsimov showed us that a machine trying to distinguish, for example, who or what is human may encountermany difficulties that humans themselves handle easily and intuitively. Thus, robots must have sufficientcapabilities for judgment - capabilities that can cause them to frustrate the intentions of their masters when, ina robot's judgment, a higher order law applies.As information technology evolves and machines begin to design and build other machines, the issue ofhuman control gains greater significance. In time. human values tend to change; the rules reflecting thesevalues, and embedded in existing robotic devices. may need to be modified. But if they are implicit ratherthan explicit, with their effects scattered widely across a system, they may not be easily replaceable. Asimovhimself discovered many contradictions and eventually revised the Laws of Robotics.Asimov's 1985 revised Laws of RoboticsThe Zeroth lawAfter introducing the original three laws, Asimov detected. as early as 1950, a need to extend the first law,which protected individual humans, so that it would protect humanity as a whole. Thus, his calculatingmachines "have the good of humanity at heart through the overwhelming force of the First Law of Robotics" 1(emphasis added). In 1985 he developed this idea further by postulating a "zeroth" law that placed humanity'sinterests above those of any individual while retaining a high value on individual human life. 2 The revised setof laws is shown in the sidebar.Asimov pointed out that under a strict interpretation of the first law, a robot would protect a person even ifthe survival of humanity as a whole was placed at risk. Possible threats include annihilation by an alien ormutant human race, or by a deadly virus. Even when a robot's own powers of reasoning led it to conclude thatmankind as a whole was doomed if it refused to act, it was nevertheless constrained: "I sense the oncoming ofcatastrophe . . . [but} I can only follow the Laws." 2In Asimov's fiction the robots are tested by circumstances and must seriously consider whether they can harma human to save humanity. The turning point comes when the robots appreciate that the laws are indirectlymodifiable by roboticists through the definitions programmed into each robot: "If the Laws of Robotics, eventhe First Law, are not absolutes, and if human beings can modify them, might it not be that perhaps, underproper conditions, we ourselves might mod - " 2 Although the robots are prevented by imminent "roblock"(robot block, or deadlock) from even completing the sentence, the groundwork has been laid.Later, when a robot perceives a clear and urgent threat to mankind, it concludes, "Humanity as a whole ismore important than a single human being. There is a law that is greater than the First Law: `A robot may notinjure humanity, or through inaction, allow humanity to come to harm." 2Defining "humanity"http://www.rogerclarke.com/SOS/Asimov.htmlPage 14 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMorders given it by superordinate robots except where such orders would conflict with a higher order law."Such a law would fall between the second and third laws.Furthermore, subordinate robots should protect their superordinate robot. This could be implemented as anextension or corollary to the third law; that is, to protect itself, a robot would have to protect another robot onwhich it depends. Indeed, a subordinate robot may need to be capable of sacrificing itself to protect its robotoverseer. Thus, an additional law superior to the third law but inferior to orders from either a human or arobot overseer seems appropriate: "A robot must protect the existence of a superordinate robot as long assuch protection does not conflict with a higher order law."The wording of such laws should allow for nesting, since robot overseers may report to higher level robots. Itwould also be necessary to determine the form of the superordinate relationships:a tree, in which each robot has precisely one immediate overseer, whether robot or human;a constrained network, in which each robot may have several overseers but restrictions determinewho may act as an overseer; oran unconstrained network, in which each robot may have any number of other robots or persons asoverseers.This issue of a command structure is far from trivial, since it is central to democratic processes that no singleentity shall have ultimate authority. Rather, the most senior entity in any decision- making hierarchy must besubject to review and override by some other entity, exemplified by the balance of power in the threebranches of government and the authority of the ballot box. Successful, long- lived systems involve checksand balances in a lattice rather than a mere tree structure. Of course, the structures and processes of humanorganizations may prove inappropriate for robotic organization. In any case, additional laws of some kindwould be essential to regulate relationships among robots.The sidebar shows an extended set of laws, one that incorporates the additional laws postulated in thissection. Even this set would not alway's ensure appropriate robotic behavior. However, it does reflect theimplicit laws that emerge in Asimov's fiction while demonstrating that any realistic set of design principleswould have to be considerably more complex than Asimov's 1940 or 1985 laws. This additional complexitywould inevitably exacerbate the problems identified earlier in this article and create new ones.The Meta-LawAn Extended Set of the Laws of RoboticsA robot may not act unless its actions are subject to the Laws of RoboticsLaw ZeroA robot may not injure humanity, or, through inaction, allow humanity to come to harmLaw OneA robot may not injure a human being, or, through inaction, allow a human being to come to harm, unlesshttp://www.rogerclarke.com/SOS/Asimov.htmlPage 19 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMThe Laws of Robotics designate no particular class of humans (not even a robot's owner) as more deservingof protection or obedience than another. A human might establish such a relationship by command, but thelaws give such a command no special status: another human could therefore countermand it. In short, thelaws reflect the humanistic and egalitarian principles that theoretically underlie most democratic nations.The laws therefore stand in stark contrast to our conventional notions about an information technologyartifact, whose owner is implicitly assumed to be its primary beneficiary. An organization shapes anapplication's design and use for its own benefit. Admittedly, during the last decade users have been givengreater consideration in terms of both the human- machine interface and participation in system development.But that trend has been justified by the better returns the organization can get from its information technologyinvestment rather than by any recognition that users are stakeholders with a legitimate voice in decisionmaking. The interests of other affected parties are even less likely to be reflected.In this era of powerful information technology, professional bodies of information technologists need toconsider:identification of stakeholders and how they are affected;prior consultation with stakeholders;quality assurance standards for design, manufacture, use, and maintenance;liability for harm resulting from either malfunction or use in conformance with the designer'sintentions; andcomplaint- handling and dispute- resolution procedures.Once any resulting standards reach a degree of maturity, legislatures in the many hundreds of legaljurisdictions throughout the world would probably have to devise enforcement procedures.The interests of people affected by modern information technology applications have been gainingrecognition. For example, consumer representatives are now being involved in the statement of userrequirements and the establishment of the regulatory environment for consumer electronic- funds- transfersystems. This participation may extend to the logical design of such systems. Other examples are trade- unionnegotiations with employers regarding technology- enforced change, and the publication of software qualityassurancestandards.For large- scale applications of information technology, governments have been called upon to applyprocedures like those commonly used in major industrial and social projects. Thus, commitment might haveto be deferred pending dissemination and public discussion of independent environmental or social impactstatements. Although organizations that use information technology might see this as interventionism,decision making and approval for major information technology applications may nevertheless become morewidely representative.Closed- system versus open- system thinkingComputer- based systems no longer comprise independent machines each serving a single location. Themarriage of computing with telecommunications has produced multicomponent systems designed to supportall elements of a widely dispersed organization. Integration hasn't been simply geographic, however. Thepractice of information systems has matured since the early years when existing manual systems wereautomated largely without procedural change. Developers now seek payback via the rationalization ofhttp://www.rogerclarke.com/SOS/Asimov.htmlPage 21 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMexisting systems and varying degrees of integration among previously separate functions. With the advent ofstrategic and interorganizational systems, economies are being sought at the level of industry sectors, andfunctional integration increasingly occurs across corporate boundaries.Although programmers can no longer regard the machine as an almost entirely closed system with tightlycircumscribed sensory and motor capabilities, many habits of closed- system thinking remain. When systemshave multiple components, linkages to other systems, and sophisticated sensory and motor capabilities, thescope needed for understanding and resolving problems is much broader than for a mere hardware/softwaremachine. Human activities in particular must be perceived as part of the system. This applies to manualprocedures within systems (such as reading dials on control panels), human activities on the fringes ofsystems (such as decision making based on computer- collated and - displayed information), and the securityof the user's environment (automated teller machines, for example). The focus must broaden from meretechnology to technology in use.General systems thinking leads information technologists to recognize that relativity and change must heaccommodated. Today, an artifact may be applied in multiple cultures where language, religion, laws, andcustoms differ. Over time, the original context may change. For example, models for a criminal justicesystem - one based on punishment and another based on redemption - may alternately dominate socialthinking. Therefore, complex systems must be capable of adaptation.Blind acceptance of technological and other imperativesContemporary utilitarian society seldom challenges the presumption that what can be done should be done.Although this technological imperative is less pervasive than people generally think, societies neverthelesstend to follow where their technological capabilities lead. Related tendencies include the economicimperative (what can be done more efficiently should be) and the marketing imperative (any effectivedemand should be met). An additional tendency might be called the "information imperative," the dominanceof administrative efficiency, information richness, and rational decision making. However, the collection ofpersonal data has become so pervasive that citizens and employees have begun to object.The greater a technology's potential to promote change, the more carefully a society should consider thedesirability of each application. Complementary measures that may be needed to ameliorate its negativeeffects should also be considered. This is a major theme of Asimov's stories, as he explores the hidden effectsof technology. The potential impact of information technology is so great that it would be inexcusable forprofessionals to succumb blindly to the economic, marketing, information, technological, and otherimperatives. Application software professionals can no longer treat the implications of informationtechnology as someone else's problem but must consider them as part of the project. 15Human acceptance of robotsIn Asimov's stories, humans develop affection for robots, particularly humaniform robots. In his very firstshort story, a little girl is too closely attached to Robbie the Robot for her parents' liking.' 12 In another earlystory, a woman starved for affection from her husband and sensitively assisted by a humanoid robot toincrease her self confidence entertains thoughts approaching love toward it/him. 16http://www.rogerclarke.com/SOS/Asimov.htmlPage 22 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMNonhumaniforms, such as conventional industrial robots and large, highly dispersed robotic systems (such aswarehouse managers. ATMs, and EFT/POS systems) seem less likely to elicit such warmth. Yet severalstudies have found a surprising degree of identification by humans with computers. 17,18 Thus, some hithertoexclusively human characteristics are being associated with computer systems that don't even exhibit typicalrobotic capabilities.Users must be continually reminded that the capabilities of hardware/software components are limited:they contain many inherent assumptions;they are not flexible enough to cope with all of the manifold exceptions that inevitably arise; andthey do not adapt to changes in their environment;authority is not vested in hardware/ software components but rather in the individuals who use them.Educational institutions and staff training programs must identify these limitations; yet even this is notsufficient: The human- machine interface must reflect them. Systems must be designed so that users arerequired to continually exercise their own expertise, and system output should not be phrased in a way thatimplies unwarranted authority. These objectives challenge the conventional outlook of system designers.Human opposition to robotsRobots are agents of change and therefore potentially upsetting to those with vested interests. Of all themachines so far invented or conceived of, robots represent the most direct challenge to humans. Vociferousand even violent campaigns against robotics should not be surprising. Beyond concerns of self interest is thepossibility that some humans could be revulsed by robots, particularly those with humanoid characteristics.Some opponents may be mollified as robotic behavior becomes more tactful. Another tenable argument isthat by creating and deploying artifacts that are in some ways superior. humans degrade themselves.System designers must anticipate a variety of negative reactions against their creations from different groupsof stakeholders. Much will depend on the number and power of the people who feel threatened - and on thescope of the change they anticipate. If, as Asimov speculates, 9 a robot- based economy develops withoutequitable adjustments, the backlash could be considerable.Such a rejection could involve powerful institutions as well as individuals. In one Asimov story, the USDepartment of Defense suppresses a project intended to produce the perfect robot- soldier. It reasons that thedegree of discretion and autonomy needed for battlefield performance would tend to make robots rebelliousin other circumstances (particularly during peace time) and unprepared to suffer their commanders' foolishdecisions. 19 At a more basic level, product lines and markets might be threatened, and hence the profits andeven the survival of corporations. Although even very powerful cartels might not be able to impede roboticsfor very long, its development could nevertheless be delayed or altered. Information technologists need torecognize the negative perceptions of various stakeholders and manage both system design and projectpolitics accordingly.The structuredness of decision makingFor five decades there has been little doubt that computers hold significant computational advantages overhttp://www.rogerclarke.com/SOS/Asimov.htmlPage 23 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMhumans. However, the merits of machine decision making remain in dispute. Some decision processes arehighly structured and can be resolved using known algorithms operating on defined data items with definedinterrelationships. Most structured decisions are candidates for automation, subject, of course, to economicconstraints. The advantages of machines must also be balanced against risks. The choice to automate must bemade carefully because the automated decision process (algorithm, problem description. problem- domaindescription, or analysis of empirical data) may later prove to be inappropriate for a particular type ofdecision. Also, humans involved as data providers, data communicators, or decision implementers may notperform rationally because of poor training, poor performance under pressure, or willfulness.Unstructured decision making remains the preserve of humans for one or more of the following reasons:humans have not yet worked out a suitable way to program (or teach) a machine how to make that classof decision;some relevant data cannot be communicated to the machine;"fuzzy" or "open- textured" concepts or constructs are involved;such decisions involve judgments that system participants feel should not be made by machines onbehalf of humans.One important type of unstructured decision is problem diagnosis. As Asimov described the problem,"How..... can we send a robot to find a flaw in a mechanism when we cannot possibly give precise orders,since we know nothing about the flaw ourselves'? `Find out what's wrong' is not an order you can give to arobot; only to a man." 20 Knowledge- based technology has since been applied to problem diagnosis, butAsimov's insight retains its validity: A problem may be linguistic rather than technical, requiring commonsense, not domain knowledge. Elsewhere, Asimov calls robots "logical but not reasonable" and tells ofhousehold robots removing important evidence from a murder scene because a human did not think to orderthem to preserve it. 9The literature of decision support systems recognizes an intermediate case, semistructured decision making.Humans are assigned the decision task, and systems are designed to provide support for gathering andstructuring potentially relevant data and for modeling and experimenting with alternative strategies. Throughcontinual progress in science and technology, previously unstructured decisions are reduced to semistructuredor structured decisions. The choice of which decisions to automate is therefore provisional, pending furtheradvances in the relevant area of knowledge. Conversely, because of environmental or cultural change,structured decisions may not remain so. For example, a family of viruses might mutate so rapidly that thereference data within diagnostic support systems is outstripped and even the logic becomes dangerouslyinadequate.Delegating to a machine any kind of decision that is less than fully structured invites errors and mishaps. Ofcourse. human decision- makers routinely make mistakes too. One reason for humans' retaining responsibilityfor unstructured decision making is rational: Appropriately educated and trained humans may make moreright decisions and/or fewer seriously wrong decisions than a machine. Using common sense, humans canrecognize when conventional approaches and criteria do not apply, and they can introduce conscious valuejudgments. Perhaps a more important reason is the arational preference of humans to submit to the judgmentsof their peers rather than of machines: If someone is going to make a mistake costly to me, better for it to bean understandably incompetent human like myself than a mysteriously incompetent machine. 8Because robot and human capabilities differ, for the foreseeable future at least, each will have specifichttp://www.rogerclarke.com/SOS/Asimov.htmlPage 24 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMcomparative advantages. Information technologists must delineate the relationship between robots and peopleby applying the concept of decision structuredness to blend computer- based and human elementsadvantageously. The goal should be to achieve complementary intelligence rather than to continue pursuingthe chimera of unneeded artificial intelligence. As Wyndham put it in 1932: "Surely man and machine arenatural complements: They assist one another." 21Risk managementWhether or not subjected to intrinsic laws or design guidelines, robotics embodies risks to property as well asto humans. These risks must be managed; appropriate forms of risk avoidance and diminution need to beapplied, and regimes for fallback, recovery, and retribution must be established.Controls are needed to ensure that intrinsic laws, if any, are operational at all times and that guidelines fordesign, development, testing, use, and maintenance are applied. Second- order control mechanisms areneeded to audit first- order control mechanisms. Furthermore, those bearing legal responsibility for harmarising from the use of robotics must be clearly identified. Courtroom litigation may determine the actualamount of liability, but assigning legal responsibilities in advance will ensure that participants take due care.In most of Asimov's robot stories, robots are owned by the manufacturer even while in the possession ofindividual humans or corporations. Hence legal responsibility for harm arising from robot noncompliancewith the laws can be assigned with relative ease. In most real- world jurisdictions, however, there areenormous uncertainties, substantial gaps in protective coverage, high costs, and long delays.Each jurisdiction, consistent with its own product liability philosophy, needs to determine who should bearthe various risks. The law must be sufficiently clear so that debilitating legal battles do not leave injuredparties without recourse or sap the industry of its energy. Information technologists need to communicate tolegislators the importance of revising and extending the laws that assign liability for harm arising from theuse of information technology.Enhancements to codes of ethicsAssociations of information technology professionals, such as the lEEE Computer Society, the Associationfor Computing Machinery, the British Computer Society, and the Australian Computer Society, areconcerned with professional standards, and these standards almost always include a code of ethics. Suchcodes aren't intended so much to establish standards as to express standards that already exist informally.Nonetheless, they provide guidance concerning how professionals should perform their work, and there issignificant literature in the area.The issues raised in this article suggest that existing codes of ethics need to be reexamined in the light ofdeveloping technology. Codes generally fail to reflect the potential effects of computer- enhanced machinesand the inadequacy of existing managerial, institutional, and legal processes for coping with inherent risks.Information technology professionals need to stimulate and inform debate on the issues. Along with robotics.many other technologies deserve consideration. Such an endeavor would mean reassessing professionalism inthe light of fundamental works on ethical aspects of technology.http://www.rogerclarke.com/SOS/Asimov.htmlPage 25 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMAsimov's Laws of Robotics have been a very successful literary device. Perhaps ironically, or perhapsbecause it was artistically appropriate, the sum of Asimov's stories disprove the contention that he beganwith: It is not possible to reliably constrain the behavior of robots by devising and applying a set of rules.The freedom of fiction enabled Asimov to project the laws into many future scenarios; in so doing, heuncovered issues that will probably arise someday in real- world situations. Many aspects of the lawsdiscussed in this article are likely to be weaknesses in any robotic code of conduct. Contemporaryapplications of information technology such as CAD/CAM, EFT/POS, warehousing systems, and trafficcontrol are already exhibiting robotic characteristics. The difficulties identified are therefore directly andimmediately relevant to information technology professionals.Increased complexity means new sources of risk, since each activity depends directly on the effectiveinteraction of many artifacts. Complex systems are prone to component failures and malfunctions, and tointermodule inconsistencies and misunderstandings. Thus, new forms of backup, problem diagnosis, interimoperation, and recovery are needed. Tolerance and flexibility in design must replace the primacy of shorttermobjectives such as programming productivity. If information technologists do not respond to thechallenges posed by robotic systems. as investigated in Asimov's stories, information technology artifactswill be poorly suited for real- world applications. They may be used in ways not intended by their designers,or simply be rejected as incompatible with the individuals and organizations they were meant to serve.Isaac Asimov, 1920-1992Born near Smolensk in Russia, Isaac Asimov came to the United States with his parents three years later. Hegrew up in Brooklyn, becoming a US citizen at the age of eight. He earned bachelor's, master's, and doctoraldegrees in chemistry from Columbia University and qualified as an instructor in biochemistry at BostonUniversity School of medicine, where he taught for many years and performed research in nucleic acid.As a child, Asimov had begun reading the science fiction stories on the racks in his family's candy store, andthose early years of vicarious visits to strange worlds had filled him with an undying desire to write his ownadventure tales. He sold his first short story in 1938 and after wartime service as a chemist and a short hitchin the Army, he focused increasingly on his writing.Asimov was among the most prolific of authors, publishing hundreds of books on various subjects anddozens of short stories. His Laws of Robotics underlie four of his full-length novels as well as many of hisshort stories. The World Science Fiction Convention bestowed Hugo Awards on Asimov in nearly everycategory of science fiction, and his short story "Nightfall" is often referred to as the best science fiction storyever written. The scientific authority behind his writing gave his stories a feeling of authenticity, and hiswork undoubtedly did much to popularize science for the reading public.References to Part 11. I. Asimov, The Rest of the Robots (a collection of short stories originally published between 1941 and1957), Grafton Books, London, 1968.2. N. Frude, The Robot Heritage, Century Publishing. London. 1984.http://www.rogerclarke.com/SOS/Asimov.htmlPage 26 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AM3. I. Asimov, I, Robot (a collection of short stories originally published between 1940 and 1950), GraftonBooks, London, 1968.4. I. Asimov, P.S. Warrick, and M.H. Greenberg, eds., Machines That Think, Holt, Rinehart. and Wilson,London. 1983.5. I. Asimov, "Runaround" (originally published in 1942), reprinted in Reference 3, pp. 33- 51.6. L. Del Rey, "Though Dreamers Die" (originally published in 1944). reprinted in Reference 4, pp. 153- 174.7. I. Asimov, "Evidence" (originally published in 1946). reprinted in Reference 3. pp. 159- 182.8. H.M. Geduld and R. Gottesman. eds.. Robots, Robots, Robots, New York Graphic Soc., Boston. 1978.9. P.B. Scott. The Robotics Revolution: The Complete Guide. Blackwell, Oxford. 1984.10. I. Asimov, Robot Dreams (a collection of short stories originally published between 1947 and 1986),Victor Gollancz, London. 1989.11. A. Chandor, ed., The Penguin Dictionary of Computers, 3rd ed.. Penguin, London, 1985.12. I. Asimov, "The Bicentennial Man" (originally published in 1976), reprinted in Reference 4, pp. 519-561. Expanded into I. Asimov and R. Silverberg. The Positronic Man, Victor Gollancz, London, 1992.13. A.C. Clarke and S. Kubrick, 2001: A Space Odyssey, Grafton Books. London, 1968.14. I. Asimov, Robots and Empire, Grafton Books, London, 1985.15. I. Asimov, "Risk" (originally published in 1955), reprinted in Reference 1. pp.122- 155.16. I. Asimov, The Robots of Dawn, Grafton Books, London, 1983.17. I. Asimov, "Liar!" (originally published in 1941), reprinted in Reference 3, pp.92- 109.18. I. Asimov, "That Thou Art Mindful of Him" (originally published in 1974), reprinted in The BicentennialMan, Panther Books, London, 1978, pp. 79- 107.19. I. Asimov. The Caves of Steel (originally published in 1954). Grafton Books. London. 1958.20. T. Winograd and F. Flores. Understanding Computers and Cognition. Ablex. Norwood, N.J., 1986.21. I. Asimov. "Robbie" (originally published as "Strange Playfellow" in 1940). reprinted in Reference 3. pp.13-32.22. I. Asimov, The Naked Sun (originally published in 1957), Grafton Books. London, 1960.23. I. Asimov. "The Evitable Conflict" (originally published in 1950). reprinted in Reference 3, pp. 183- 706.24. I. Asimov. "The Tercentenary Incident" (originally published in 1976). reprinted in The Bicentennialhttp://www.rogerclarke.com/SOS/Asimov.htmlPage 27 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMMan, Panther Books, London, 1978, pp. 229- 247.25. I. Asimov, "Little Lost Robot" (originally published in 1947). reprinted in Reference 3, pp. 110- 136.26. I. Asimov, "Robot Dreams," first published in Reference 10, pp. 51- 58.27. I. Asimov, "The Machine That Won the War" (originally published in 1961), reprinted in Reference 10.pp. 191- 197.28. D. Bellin and G. Chapman. eds., Computers in Battle: Will They Work? Harcourt Brace Jovanovich,Boston, 1987.29. I. Asimov, "Reason" (originally published in 1941), reprinted in Reference 3, pp. 52- 70.References to Part 21. I. Asimov, "The Evitable Conflict" (originally' published in 1950), reprinted in I. Asimov, I Robot, GraftonBooks. London. 1968. pp. l83- 206.2. I. Asimov, Robots and Empire, Grafton Books. London. 1985.3. I. Asimov, "The Bicentennial Man" (originally published in 1976). reprinted in I. Asimov, P.S. Warrick,and M.H. Greenberg, eds., Machines That Think. Holt. Rinehart, and Wilson, 1983, pp 519- 561.4. I. Asimov, The Robots of Dawn, Grafton Books. London, 1983.5. I. Asimov, "Jokester" (originally' published in 1956), reprinted in 1. Asimov, Robot Dreams, VictorGollancz, London. 1989 pp 278- ~94.6. D. Adams. The Hitchhikers Guide to the Galaxy, Harmony Books. New York. 1979.7. A.C. Clarke. Rendezvous with Rama, Victor Gollancz, London. 1973.8. J. Weizenbaum. Computer Power and Human Reason, W.H. Freeman. San Francisco, 1976.9. I. Asimov, The Naked Sun, (originally' published in 1957). Grafton Books. London. 1960.10. I. Asimov, "Lenny" (originally published in 1958), reprinted in I. Asimov. The Rest of the Robots.Grafton Books, London. 1968, pp. 158- 177.11. H. Harrison. "War With the Robots" (originally published in 1962), reprinted in I, Asimov, P.S. Warrick,and M.H. Greenberg, eds., Machines That Think, Holt, Rinehart, and Wilson, 1983, pp.357- 379.12. I. Asimov, "Robbie" (originally published as "Strange Playfellow" in 1940). reprinted in I. Asimov, I,Robot. Grafton Books. London, 1968, pp. 13- 32.13. A.E. Van Vogt, "Fulfillment" (originally published in 1951). reprinted in I. Asimov, P.S. Warrick, andM.H. Greenberg. eds., Machines That Think, Holt, Rinehart, and Wilson, 1983, pp.175- 205.http://www.rogerclarke.com/SOS/Asimov.htmlPage 28 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AM14. I. Asimov. "Feminine Intuition" (originally published in 1969), reprinted in I. Asimov, The BicentennialMan, Panther Books, London, 1978, pp. 15- 41.15. R.A. Clarke, "Economic, Legal, and Social Implications of Information Technology," MIS Q,uarterly,Vol 17 No. 4, Dec. 1988, pp. 517- 519.16. I. Asimov, "Satisfaction Guaranteed" (originally published in 1951), reprinted in I. Asimov, The Rest ofthe Robots, Grafton Books, London, 1968, pp.102- 120.17. J. Weizenbaum, "Eliza," Comm. ACM, Vol. 9, No. 1, Jan. 1966, pp. 36- 45.18. S. Turkle, The Second Self' Computers and the Human Spirit, Simon & Schuster, New York, 1984.19. A. Budrys, "First to Serve" (originally published in 1954), reprinted in I. Asimov, M.H. Greenberg, andC.G. Waugh, eds., Robots, Signet, New York, 1989, pp. 227- 244.20. I. Asimov, "Risk" (originally published in 1955), reprinted in I. Asimov, The Rest of the Robots, GraftonBooks, London, 1968, pp. 122- 155.21. J. Wyndham, "The Lost Machine" (originally published in1932), reprinted in A. Wells, ed., The Best ofJohn Wyndham, Sphere Books, London, 1973, pp. 13- 36, and in I. Asimov, P.S. Warrick, and M.H.Greenberg, eds.,Machines That Think, Holt, Rinehart, and Wilson, 1983, pp. 29- 49.Author AffiliationsRoger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in theCyberspace Law & Policy Centre at the University of N.S.W., a Visiting Professor in the E-CommerceProgramme at the University of Hong Kong, and a Visiting Professor in the Department of Computer Scienceat the Australian National University.PersonaliaPhotographsAccessStatisticsThe content and infrastructure for these community servicepages are provided by Roger Clarke through his consultancycompany, Xamax.From the site's beginnings in August 1994 until February2009, the infrastructure was provided by the AustralianNational University. During that time, the site accumulatedclose to 30 million hits. It passed 40 million by the end of2012.Sponsored by Bunhybee Grasslands, the extended ClarkeFamily, Knights of the Spatchcock and their drummerXamaxConsultancy PtyLtdACN: 002 360 45678 Sidaway St,Chapman ACT2611 AUSTRALIATel: +61 2 62886916http://www.rogerclarke.com/SOS/Asimov.htmlPage 29 of 30

Roger Clarke's 'Asimov's Laws of Robotics'2013-09-03 10:33 AMCreated: 16 May 1997 - Last Amended: 16 May 1997 by Roger Clarke - Site Last Verified: 15 February 2009This document is at www.rogerclarke.com/SOS/Asimov.htmlMail to Webmaster - © Xamax Consultancy Pty Ltd, 1995-2013 - Privacy Policyhttp://www.rogerclarke.com/SOS/Asimov.htmlPage 30 of 30