A Framework for Evaluating Early-Stage Human - of Marcus Hutter

More documents

Recommendations

Info

• Task and Domain Generality: How well does a system (or architecture) support behavior across a wide range of tasks and domains? Concerns about task and domain generality are one of the primary factors that distinguish research in HLI from much of the other research in AI. This requires measures of diversity of tasks and domains, which are currently lacking. Given the primacy of generality, it is not surprising that many other abstract metrics address aspects of behavior and system construction that are related to generality. • Expressivity: What kinds or range of knowledge can an HLI system accept and use to influence behavior? This relates to generality because restrictions on expressiveness can, in turn, restrict whether a system can successfully pursue a task in a domain. For example, systems that only support propositional representations will have difficulty reasoning about problems that are inherently relational. • Robustness: How does speed or quality of solutions change as a task is perturbed or some knowledge is removed or added? One can also measure robustness of an architecture – how behavior changes as an aspect of the architecture is degraded – but this is rarely considered an important feature of HLI systems. Instead, the interest lies in how well the system can respond to partial or incomplete knowledge, incorrect knowledge, and changes in a task that require some mapping of existing knowledge to a novel situation. • Instructability: How well can a system accept knowledge from another agent? Instructability emphasizes acquiring new skills and knowledge, as well as acquiring new tasks. Finer-grain measures of instructability include the language needed for instruction, the breadth of behavior that can be taught, and the types of interactions supported, such as whether the instructor is in control, whether the agent is in control, or whether dynamic passing of control occurs during instruction. • Taskability: To what extent can a system accept and/or generate, understand, and start on a new task? Taskability is related to instructability, but focuses working on new tasks. Humans are inherently taskable and retaskable, being able to attempt new tasks without requiring a external programmer that understands its internal representations. Humans also generate new tasks on their own. In contrast, most current systems only pursue the tasks and subtasks with which they were originally programmed and cannot dynamically extend the tasks they pursue. • Explainability: Can the system explain what it has learned or experienced, or why it is carrying out some behavior? Humans do not have “complete” explanability – the ability to provide justifications for all decisions leading up to external behavior – so this capability is a matter of degree. 95 Conclusion It is obvious that developing human-level intelligence is a huge challenge. However, important parts of that scientific and engineering enterprise are the methods and practices for evaluating the systems as they are developed. In this paper, we present some of the primary challenges that arise in evaluation that distinguish it from research on more specialized aspects of artificial intelligence. We also attempt to characterize the types of scientific claims that arise in research on HLI, distinguishing different classes of claims that can be made at the system level, and then further analyzing the independent and dependent variables of those claims. Having clear, explicit claims has always been a critical part of scientific progress, and we encourage researchers to be more explicit in the claims of the theories and systems they develop. This not only helps ourselves in designing appropriate experiments, it also makes it much easier for other researchers to evaluate the contribution of a piece of work. In addition to being specific about claims, the field also needs shared notions of methodologies and metrics associated with evaluating those claims. The abstract metrics enumerated here suggest some ways in which HLI researchers can begin to better distinguish these systems from more traditional AI systems. However, much work remains to identify methods for measuring and evaluating these capabilities. The next step for this effort is to explore tasks and environments that can be shared across the community. Given the broad goals of HLI research, we need multiple testbeds that support environments in which many tasks can be pursued, and which include tools for performing experimentation and evaluation. Working on common problems will simplify cross-system evaluation and collaboration, both important steps toward developing human-level intelligence. Acknowledgments This paper is based on discussions at a workshop on evaluation of human-level intelligence, held at the University of Michigan in October, 2008. Participants included Joscha Bach, Paul Bello, Nick Cassimatis, Ken Forbus, Ben Goertzel, John Laird, Christian Lebiere, Pat Langley, Robert Mariner, Stacey Marsella, Matthias Scheultz, Satinder Singh, and Robert Wray. The workshop was funded under grant N00014-08-1-1214 from ONR. References Anderson, J. R. & Lebiere, C. L. 2003. The Newell test for a theory of cognition. Behavioral & Brain Science 26, 587-637. Cohen, P. R., 1995. Empirical methods for artificial intelligence, Cambridge, MA: MIT Press. Cohen. P.R. 2005. If not Turing’s test, then what? AI Magazine, Winter, 26; 61-68.
Doorenbos, R. B. 1994. Combining left and right unlinking for matching a large number of learned rules. In Proceedings of the Twelfth National Conference on Artificial Intelligence. Gluck, K. A. & Pew, R.W. 2005. Modeling human behavior with integrated cognitive architectures: comparison, evaluation, and validation LEA/Routledge. Jones, R. M., & Wray, R. E. 2006. Comparative analysis of frameworks for knowledge-intensive intelligent agents. AI Magazine, 27, 57-70. Laird, J. E., 2008. Extending the Soar cognitive architecture. In Proceedings of the First Conference on Artificial General Intelligence. Langley, P., & Choi, D., 2006. A unified cognitive architecture for physical agents. In Proceedings of the Twenty-First National Conference on Artificial Intelligence. Boston: AAAI Press. Langley, P., Laird, J. E., & Rogers, S., in press. Cognitive architectures: Research issues and challenges. Cognitive Systems Research. Langley, P., & Messina, E. 2004. Experimental studies of integrated cognitive systems. Proceedings of the Performance Metrics for Intelligent Systems Workshop. Gaithersburg, MD. Legg , S. & Hutter, M. 2007. Universal Intelligence: A Definition of Machine Intelligence, Minds and Machines, 17:4, 391-444. Newell, A., Simon, H. A., 1976. Computer science as empirical inquiry: Symbols and search. Communications of the ACM, 19 Turing, A., 1950. Computing Machinery and Intelligence. Mind, 59; 433–460. Wray, R. E., & Laird, J. E. 2003. An architectural approach to consistency in hierarchical execution. Journal of Artificial Intelligence Research. 19; 355-398. Wray, R. E., & Lebiere, C. 2007. Metrics for Cognitive Architecture Evaluation. In Proceedings of the AAAI-07 Workshop on Evaluating Architectures for Intelligence, Vancouver, B. C. 96
Page 2 and 3:
Ben Goertzel, Pascal Hitzler, Marcu
Page 4 and 5:
Artificial General Intelligence Vol
Page 6:
Preface Artificial General Intellig
Page 9 and 10:
Organizing Committee Tsvi Achler U.
Page 11 and 12:
A formal framework for the symbol g
Page 13 and 14:
Importing Space-time Concepts Into
Page 15 and 16:
one structure, everything else adap
Page 17 and 18:
generalize is essential to understa
Page 19 and 20:
First Application The above framewo
Page 21 and 22:
problem solving, use of knowledge,
Page 23 and 24:
Of particular note is the separatio
Page 25 and 26:
Conclusions and Future Work While p
Page 27 and 28:
Conceptual Spaces The conceptual ar
Page 29 and 30:
perceptions and actions. The lingui
Page 31 and 32:
means of the knowledge stored in th
Page 33 and 34:
grams given some (syntactically res
Page 35 and 36:
For example, let reverse([]) = [] r
Page 37 and 38:
Igor2 produced solutions with auxil
Page 39 and 40:
evolve neural net modules as quickl
Page 41 and 42:
ain”, consisting of some dozen or
Page 43 and 44:
Population Chr NListPtr m Chrm is a
Page 45 and 46:
and shaping, we feel that exploring
Page 47 and 48:
Table 2 reviews the key capabilitie
Page 49 and 50:
One way to achieve this goal would
Page 51 and 52:
question. Our approach of staying a
Page 53 and 54:
H0 δB(H0) δF(H0) H1 δB(H1) δF(H
Page 55 and 56:
Discussion and future work Testing
Page 57 and 58: Types of Reasoning Corresponding Fo
Page 59 and 60: Integrating Different Types of Reas
Page 61 and 62: good overview of analogy models can
Page 63 and 64: ��
Page 65 and 66: � ��
Page 67 and 68: ��
Page 69 and 70: A Unified Framework for IP Conditio
Page 71 and 72: negative evidence when added to a r
Page 73 and 74: ing strategy. Due to GOLEM’s rand
Page 75 and 76: any state index, n∈IN the current
Page 77 and 78: is the best observation summary, wh
Page 79 and 80: to a code for the rewards only, whi
Page 81 and 82: have exponentially many states (2O(
Page 83 and 84: where ˆσ 2 =Loss( ˆw)/n. Given
Page 85 and 86: • The cost function can be improv
Page 87 and 88: decides to introduce inflation or d
Page 89 and 90: € € The diffusion matrix is the
Page 91 and 92: tuning may be done adaptively by te
Page 93 and 94: Of course, Harnad’s argument is n
Page 95 and 96: By exploring variations of Harnad
Page 97 and 98: Discussion The representational sys
Page 99 and 100: the cognitive map is less of a map
Page 101 and 102: models of cognition except in cases
Page 103 and 104: 7(b), we can see that the straight
Page 105 and 106: eaction times, error rates, or exac
Page 107: comparing behavior with and without
Page 111 and 112: egion, we leave the type of object
Page 113 and 114: extract features in support of reco
Page 115 and 116: with a minimum score of -1.0. The
Page 117 and 118: the NASA Human Error Modeling compa
Page 119 and 120: whether their models are capable of
Page 121 and 122: Incorporating Planning and Reasonin
Page 123 and 124: see an unclaimed ten-dollar bill al
Page 125 and 126: swering user queries) and the state
Page 127 and 128: Program Representation for General
Page 129 and 130: distribution P corresponding to the
Page 131 and 132: with all Ei replaced by the unbound
Page 133 and 134: Consciousness in Human and Machine:
Page 135 and 136: at the sensory receptors, is pre-pr
Page 137 and 138: The second choice is to take a deta
Page 139 and 140: Hebbian Constraint on the Resolutio
Page 141 and 142: The Case of Inputs that Change by T
Page 143 and 144: This route is relevant if the noise
Page 145 and 146: 1. Oculus Info. Inc. Toronto, Ont.
Page 147 and 148: Our implemented Turing judge used a
Page 149 and 150: 2.7; Human vs. JabberWacky t(157.6)
Page 151 and 152: UnsupervisedSegmentationofAudioSpee
Page 153 and 154: FinallyweusedthediscreteFourierTran
Page 155 and 156: Secondly,thereexistsmorethanonelogi
Page 157 and 158: Parsing PCFG within a General Proba
Page 159 and 160:
known in advance, although many imp
Page 161 and 162:
Figure 2: Shows the underlying weig
Page 163 and 164:
Self-Programming: Operationalizing
Page 165 and 166:
expressivity. More over, self-progr
Page 167 and 168:
existence of a distance function ov
Page 169 and 170:
Bootstrap Dialog: A Conversational
Page 171 and 172:
elation is typically a verb. In the
Page 173 and 174:
node RDF proposition N1 N2 N3 N4 N5
Page 175 and 176:
Analytical Inductive Programming as
Page 177 and 178:
Problem domain: puttable(x) PRE: cl
Page 179 and 180:
eq Hanoi(0, Src, Aux, Dst, S) = mov
Page 181 and 182:
Human and Machine Understanding Of
Page 183 and 184:
case orderings t correspond to the
Page 185 and 186:
in (1) received a different denotat
Page 187 and 188:
Abstract This paper analyzes the di
Page 189 and 190:
Having grounded meaning: In an inte
Page 191 and 192:
to experience” can be argued to b
Page 193 and 194:
Abstract Case-by-case Problem Solvi
Page 195 and 196:
First, since NARS is designed in th
Page 197 and 198:
typical response is to find such an
Page 199 and 200:
What Is Artificial General Intellig
Page 201 and 202:
Finally, the facts that both seem t
Page 203 and 204:
differ. NARS uses Narsese, the fair
Page 205 and 206:
Integrating Action and Reasoning th
Page 207 and 208:
states, with the condition that the
Page 209 and 210:
action model. The system is able to
Page 211 and 212:
Neuroscience and AI Share the Same
Page 213 and 214:
Relevance Based Planning: Why Its a
Page 215 and 216:
General Intelligence and Hypercompu
Page 217 and 218:
To appear, AGI-09 1 Stimulus proces
Page 219 and 220:
Distribution of Environments in For
Page 221 and 222:
The Importance of Being Neural-Symb
Page 223 and 224:
Improving the Believability of Non-
Page 225 and 226:
Understanding the Brain’s Emergen
Page 227 and 228:
Abstract The challenge of creating
Page 229 and 230:
Importing Space-time Concepts Into
Page 231 and 232:
HELEN: Using Brain Regions and Mech
Page 233 and 234:
Holistic Intelligence: Transversal
Page 235 and 236:
Achieving Artificial General Intell
Page 237:
Achler, Tsvi . . . . . . . . . . .
show all

A Framework for Evaluating Early-Stage Human - of Marcus Hutter

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?