Evaluation of learning in computer based education using log ...

Session 12a3EVALUATION OF LEARNING IN COMPUTER BASED EDUCATIONUSING LOG SYSTEMSMartti Rahkila and Matti KarjalainenHelsinki University of TechnologyLaboratory of Acoustics and Audio Signal ProcessingI! 0. Box 3000FIN-U201 5 HUT FinlandMartti.Rahkila@hut.fi, Matti.Karjalainen~hut:.fihttp://www.acouaticsl.hut.fi/ABSTRACTEvaluation is llifliculr but importantpart of leuvtiing to bathmchcrs nad siudents. This paper discusses the idcn of usinglog systerns to collect information ira Compiirer Baaed Education(CBE}. The log information can be aiinlyzed and usedtogether or cvcn instead of traditional evnliintiota measures.Furtlierniore, in terms of self-study CBE mnterinl. loggingcan he beraeficially applied to interactively control Icnniirig.As n special case of interest, methods fur ming WWW logsystenn in learliing evaluation as well as advanced structuresfor interactive WWW-based edimuion are discussed1. INTRODUCTIONthwc cases arc illustrated with examples from CBE applications.As a special casc of interest, WWW and its log systemare discussed. All current WWW servers include a loggingmechanism and this information could be used for cvaluatingWWW-based education. A standard WWW server logsystem is thoroughly discussed and evaluated what can andwhat can’t be analyzed from that data. Methods such as userauthentication or HTTP cookie tcchniques and their use forthe identification of users are examined. Furthermore, it isshown how logging can be utilized for interactive user controlin an advanced WWW-based education system.2. EVALUATION OF LEARNINGThe evaluation of learning is without a doubt of interest toevery teacher. However, it is vcry difficult to develop any objectiveor technical measures to it. After all, learning evaluationcontains also philosophical aspects such as what learninginherently is. Still, teachers have throughout years developedand used some criteria to estimate thcir work results.The traditional methods include examinations and fecdbackforms. However, Computer Based Education (CBE) allowsanothcr possibility, logging, that is, gathering informationand analyzing user’s actions, to be used together withor eveninstead of tho traditional methods.This paper discusses the pros and cons of the traditionalmethods and presents the concept of logging from cducationalpoint of view. Three majar uses for logging are presented:1) using logs (files) as a teacher’s reference to evaluatelearning process, 2) using logging (memory) to interactivelyguide users in multimedia, self-study CBE applications,and 3) using logs (files and memory) to give users apossibility to examine their own progrcss and automaticallygenerate reports in multimerlin-based, self-study CBE. All-The evaluation of learning is a difficult but extremely importantpart of developing courses and teaching methods. Themotivating effect is obvious. But equally or even mnrc importantthan it is to a teachor, it is to a studcnt to evaluatc andmeasure their progress. Only by feedback students are ableto understand the quantity or quality of their learning. However,the question of learning evaluation is far from beingsimple: How to measure if a student has learned somethingand if so, what would it be? At the vcry end, the questionis not of finding a suitable criteria but B philosophical one:What is learning?A wide variety of methods for evaluation, assessment,comparison etc. of learning has been presented in the literature,for both come and curriculum level. It is not withinthe scope of this paper to cavcr these nil, but two commoncourse-level mcthods are discussed.Perhaps the most traditional metliud of evaluating learningat coursc level is an examination. In this method, the teachercreates a test and students take it. After that, the teacher evaluatesanswers RS correct, incorrect or something in between0-7803-564.7-8/99/$10.000 1999 IEEENovember 10 - 13,1999 San Junn, Puerto Rico29th ASEEILEEE Froiitiers ill Education Conference12a3-16

Session 12a3find finally describes theresults as numbers. Thc student canthen coinpare hislhcr results to ail ubsolute scalo or to resultsof others and thus evaluate their perforrnancc. The teacher,on thc other Iiand, tnkes the avcrage results and uses themfor an estimate of the overall learning of the students hkingthe examination. But can we really say that these resultsdescribe Icarning? If the test wils cleverly designcd, probablyyes, at least to some extent, but in general this methodis far too dependent of the test itself, the course, the goals.the material etc. Often thc situation is that the teacher has toadjust the overall scale leaving nothing but a hunch of trueevaluation of learning. Still, thcre is no need to abandon cxaminationswithout carefully considerating it first. Despitetheir problems, examinations are vcry often tlic only practicalalternative.Another commonly uscd method with courses is a feedbackform. Students are givcn R form and they are expectedto use it to evaluate their own learning and/or course withsome kind of predefined scale or by short written answers. Itis customary that when a form is used together with an examination,they do not affect course grading and usually thereis a possibility to ft~lfill and return the form imonymously.Unfortunately, oftcn the feedback forms have been designedonly from the viewpoint of A teacher and the benefit of fulfillingonc remains unclear for a student. There are exceptions,of course, and a well designed feedback form can give bothstudents and teachers equally uscfiil information far learningcvnluation.There are also morc advanced methods such as course assignments,laboratory experiments etc. that give a studcnt npossibility lo apply thc things that thcy have learned. Thecourse assignments ccrtainly give a mare accurate estimateof the leaning, but for tcnchers, it is sometimes impossible10 organize such in practice. Combinations of the methodsdescribed can also be beneficial, see for example [2].3. THE CONCEPT OF LOGGINGIn cornpatcr science, logging is normally considered as savinguser or debug information to a file. It is il very much standardtechnique to 1) calculate the usagc of the software and2) tracc back possible problem situations. In client-serverapplicstinns, logging is a very solid and natural part of thesystem, hccnuse much of the log information tins to bc exchangedbetween the client and the scrver anyway. Thcrefore,for examplo all common Internet services like WWW,ftp, cmail etc. include a log systcm.Hnwcvcr, in this paper, the concept OF logging should beunderstood in a more general way: as exchanging and storinginformation. Thc information can then be saved into afile or only kept in B memory while an ~pplication is used,depending on what it is used far. What information is in thissense logged also depcnds on a particular application wenthough common parameters defioitcly cxist. Thus thc conceptof lugging should actually be thought of as includinguser related mefadata with the original information.It should be noted that the concept of logghgmay includerestrictions by law. In some countries and in certain type ofservices or applications, logging is regarded obligatory bylaws and in somc countries, even the same ones, the lawsprohibit any logging that may expose or relate to persons’trite identity.Therefore, whenever a log system that authenticates usersis used, the authentication mechanism should not relate toperson‘s trire identity but primarily identify and distinguishthe users from each other and secondarily identify they RSindividual persons. In the fieldof education this in fact couldbe a problem, think of for example electronic examinations.Still, quitc often it is only desired to distinguish the studentsas individual students rather than true, individual persons inlegal sence.4. LOGS AS TEAC€Il?.R’S REFERENCEIn 1994- 1995 a tritorial lcvcl CBE application, “Introduclionlu Signal P[-uucsssilig” [ 11 was developed at the IldsiiikiUniversity of Technology, Laboratory of Acoustics and AudioSignal Processing. The application is a multimediaself-study package. but it has been used as a part of thecourse “Fundamentals of Acoustics I” which is the veryfirst acoustics coiirse for students. “Introduction to SignalProcessing” is an Apple Macintosh application and studentsused the program in a dedicated workstation located in thelaboratory.The application consists of “pages” (windows) with eitherstatic information, interactive demonstrations Or questions,and “chapters”, collections of pages that cover a certain topic. The studcnts were allowed to go through thcsetopics in any order they wished, but there was also a defaultpath that could be followed. From thc very beginninguT thr: development project it was clear that informatiou wasneeded ta cvsluate llie SLICCCSS :ind (isage of this CBE npplication,From tcachec’s point of view, the questions of interestincluded:4 How much time would the students spend with it?How much material would they go through?4 In which order would the studcnts study subtopics?Are them pagesitopics that students would spend n lotmorc time on than nthcrs?Are tliere pogos/topics that students would skip’?What is the aver:igc time students spend on the application or page’?0-7803- 5643 - 8/99/$10.00 @ 1999 1 E EUNovernhcr 10 - 13,1993 San Jtinii, Puerto Rico29th ASEWIEEE Frontiers in Ediicatioii Conference12 a3 -17

Session 12a3I Would the students take the default path or would theyuse tho possibility to study the topics in same other order?In order to get answers to thesc questions, a log systemwas implemented into the application. The following informationwas logged:login: course name, username, studentnumber, date, time0 page open: symbol name of the page,universal timepage close: symbol name of the page,universal timeIsgout: timeTlic information WRS saved as text and later imported toa sprcadsheet program for detailed analysis. After using theapplication, students were also cncouraged to fill in a feedbackform that contained questions related to both the contentsand the implementation.The use oC“In1wduction tu Sigiial Prucessirig” was esstiitinllyonly a small part of a regular course that included itnexamination as the primary course evaluation method. Thcapplication was not intended to replace any other course orcoiirse material but to provide additional material to the regularacoustics course. Thc CBE-material was not includedin the course examination and that is why the evaluation hadto be based on some other method. Therefore, in ordcr toestimate learning with the CBE-application, the logs wereanalyzed and two criteria chosen as learning measures: theoverall time a student spent with the program and the amountof pages fitudent browsed. These criteria were given minimumvalues based on thecontents. Students who spent moretime and went through more pages than the minimum values,were given additional points in the exatnination (5 YO of the~naxi~riuiri). Wedbwk him (paper) weir. usrd as a backupmethod to ensure log evaluation results especially in possiblcproblem situations.The CBE-application being a separate, self-study cxperimentwithout any relation to previous or parallel matcrialprevents a thorough, compardve analysis to be included.However, the overnll results from the first group of students(approx. 80) are as follows: A comparison of the resultsof the fccdback form analysis and the log analysis clearlyshowed that students who really concentrnted and spent timewith the CBE-application. actually learned somcthing fromit and that the chosen criteria would be suitable for learningevaluation. Additional log analysis results like averagetime spent on a page sliowerl that students spent more timeon pages that contained interactive material but otherwise nomajor dil‘Cerellces betwccn he puges could bt: Fuurid. However,this kind of result was rather expected in the originaldesign goals for this CBE-application.5. LOGS AS IN’1’EKACTIVE; GUIDESThe “Introduction to Signal Processing” contained also anotherlogging mechanism: it kept the information of visitedpages in memory. Because the p ap were divided into tcipicspecificgroups (chapters), it was pnssible CO keep track oftho chapters and autfimatically suggest users with topics thatthcy hod not covered yet. Furthermore, thc visited pageswere also attached tu navigation history thus allowing siudentsto refer to any page they had already visited.In sclf-study, multimedia CBE, such behavior is very usefulbecause it allows the application itself to suggest the studentwhnt topics hckhe should study next and, furthe.rmore,relationships betwecn topics to be in control of the CBE devcloper,that is, the teacher.“Introduction to Signal Proccssing” utilized this behaviorby containing an irilerictivt: expuiment that students wereallowed to explore only after all topics had becn covered.Thc cxperiment itself combined information fmm the othertopics. In general, this kind of control is somewhat essentialin self-study CBE to make sure students have the necessarybackground information before moving on to mme advancedmaterial. It is also uscful in case of electronic examinations.6. LOGS AS STUDENT’S REPI3RHENCEIf a CBE-application contains lots of material or the materialconsists of different topics related to each other, it is necessaryto give studcnt a possibility to estimate how much thcyhave done alredy and how much they still have to do in ordertn cnvcr dl or certain amount of the material. One usefilltoul for that is the previously mentioned navigation history,but “Introduction to Signal Processing” contained also anothertool for this: the status window.After logging in, thc studcnt could, at any stagc. choose“Status” from a menu. The stiltus window then presented infnrmationsuch as Ihc topics already covered, the topics todo and total time spent so far. It could have also includedcomplex estjmates like a time estimate how long it wouldtake For a student to study everything (based [m the average“spced”) etc. but such were not implcmcnied in “Introductiontn Signal Processing”.From the point or view of self-evaluatinn of leatning, thestatus tool is very useful and nt its best quite accurate, dependingof course 011 the particular CBEnpplication.Another interesting idea is tn use logging for automoticallygencrating reports, An cxample of this type of use wasincluded in B “Psychoacoustics” CBE-package [I] I The applicationcontained theory and demonstrations of psychoacuustjcalphmurIieiia in the limn dlistening tests. The studentscould for instance merrsurc their own frequency maskingcurves by performing a series of listening tests. The au-0.7803-5643-8/99/$10,00 @ 1999 IEEENovember 10 - 13,1999 San Juan, Puerto Rico29th ASEEflEEE Frontiers in Education Confcrcncc12a3- 18

Session 12a3tomatically generated rcport in this case was student's ownfrequency masking curve, which helshe could easily comparewith theoretical results.The automatic report generating is a useful concept inevery CBE application that involves measurements (real orsimulated) or testlqucstion-like interactivity.The studcnts can also be given access to their own logfiles afterwards. Analyzing the logs by themselves wouldeasily point out issues and topics thnt were not thoroughlyexamined and cstimate what topics should bc given furthernotice. Basically, the analysis could be similar to the oneteachers pcrfnrrn, but in this case, it would only evaluate thelearning process of an individual student.7. WWW LOG SYSTEMIn the last few years, the WWW (World Wide Web) has becomeperhaps the most important tool for CBE. From thevery beginning, WWW scrvcrs (httpd) havc included a loggingmcchanism as well. Like many other Internet services,the httpd log system has been designed mainly for "hit-rate"type statistical analysis or debugging purposes, but in hctit is one of the most flexible and well-designed log systemsaround.To be precise, the WWW log systcm is a sum of threecomponents: 1) the server software, 2) the client softwareand 3) the HI'l'P protocol. The information that is written tothc log depends on thc server softwarc and also the paramctersthnt are sent by the browser. 'I'he information exchangeis done withiti thc c~pabilities of the HTTP protocol [4].A typical httpd log cuntains (example takcn from thedocumentation of one of thc most popular WWW-servers.apache (http://ww. apache. orq/)):host ident authuscr date request stntns byteswhcrc the tokens archost The fully~qiialifiecl domain name of the client, orits IP number if the name is not avoilnble.idcnt IT IdentityCheck is enabled and the client machineruns itlentcl; then this is the identity informationrcported by the client.authiiser If the rcqucst was for n password protectedducumcnt, then this is thc userid used in thc request.date The date and time of the request, in fonnntdaylmonthlyemhour: minute:second zonerequest The request line (url) from the client, enclosedin double quotes (").status The thrcc digit status codc rcturnerl to the clicnt.bytes The number of bytcs in the object returned to theclient, not including any hcaders.1fa token docs not have a value then it is represented by ilhyphen (-1.The above is called Common Log Format (CLF) and theinFormation is written to so called TransferLog. Apacheserver also has a possibility to use customized log formatswith a number of additional keywords. Howcvcr, the CLF assuch is adequate for most purposes except for two particulartokens, Refercr and User-Agent. The Refercr is the url tuthe page from where the user came from (via HTML-link)and User-Agent, an id-string of the browscr software sendingthe request. Apache server can bo configured to storethese directly to TransferLog with custom log commands"W{Refcrcr}i" and "%{User-Agent}i". Errors are usuallywritten to a separate ErrorLog with similar informntion tpossibly scime deiails of the error itself.The log systems in other WWW-servers vary both informat and in customisability but thc differences are quitesmall.On a busy server the log files easily become very big andthus difficult to handle. Hence there is a number of specializcdWWW log analysis tools available, for instance nndog(http://www.statslab.cam.ac.uk/"sretl/analog/)or WUmp (http://www.boutell .com/wusage/). Themost advanced ones are very fast and customiznble in termsof analysis paramctcrs and input or output format, butusually they provide only basic statistical measiires such ashit rates etc.8. LEARNING EVALUATION WITH WWW LOGSYSTEMFrom the point of view of learning evaluation, it is importantto notice that the basic loggingmakes it somewhat irnpossibleto tracc back individual users' requcsts. A standardsolution is to use authentication to restrict access tothe contents. Most WWW servers and brnwsers supportauthentication and in its simplest form, uscr ;iuthenticationis easy to implcmcnt. Usually nuthenticatinn involves alsoaccount management, preferably as automatic as possible,which is not simple to implement (not being very difficultcither, though). A gouood management system would allowstudents (or anyone) to register on-line and create the necessaryaccount immediately as well as methods for taking careof changing password or removing account. If user authenticationis used, it is possiblc to do off-line evniuation andannlysis similar io one presented in section 4.Another disadvantage of the basic WWW log systcrn isthat the information is writtcn only on B request basis. loput it in other words: it doesn't remember anything. In orderto use logging for purposes such as interactive guidanceor on-line evnluation, morc advanced techniques arc needed.Two basic solutions are available: 1) producing thc contentNovember 10 I 13,1999 San Juan, Puerto Rico29th ASEWIEEE Fronticrs in Education Confcrciice12a3-19

Session 12a3dynamically (on the server) so that appropriate metadata ciinbe included as for examplc hidden input fields etc. and 2)I-ITTP Cookies [5] that are small information fragments tmvdingwith I-ITTP headers.The former method, producing contents dynamically supersedesthe latter, cookies, in various aspects. Perhaps themost important one is reliability; not all browsers supportcookies at all and, even if they do, the user can deliberatelydeny their use. Cookies are also very limited in their cnpnbilitiesof carrying information, for instance, the size of acookie can not exceed 1024 bytes. Still, cookies are verycommon and used especially for tracking timing or users'paths when browsing through a WWW site.Producing the contents dynamically allows the developer(teacher) tu have all the log information available anytimcand furthermore, to specify what metadata and how it is indudedin the real contents. There arc drawbacks too: it ismuch more difficult for a devclopcr to create dynamic contentsthan static WWW pilges and it always takcs more computationalresources from thc server.A widc variety of implementation techniques can beusd. CGt (Common Gateway Interface), apachc moduletechniques(mod-pd. modphp, etc.), Java Servlets and soan. Also, in order to manage large amount of data, that isCBE-material as WWW pages, a ditilbase of some kind isnceded too. SQL or text databases seem to be the most popularchoices at the moment.It should be noticed that, in principle, client-side idcractivitytechniques such as Java Applets or scripting languagesdo not provide log information from the processing itself.This is due to the fact that the WWW log system is based oninteraction betwccnclient and server and client-side processingbasically does not involve server for other than providingthe code itself.9. EXAMPLE ANALYSISConsider the following (sirnplificd) log data:IP1IP1I P2IP1IP2IP1IP1I P2IPluserluserluser2userluser2userluserluserauserlt imelt ime2time3t irne4time5time6time7time8time9urll codeurl2 codeurZl codeur13 codeur12 codeur14 codeurLs codeur37 codeur16 codesizesizesizesizesizesizesizesizesizeFrom that data, imrnidiately can be calculated:reEOref1ref0ref2ref1ref3ref4ref2ref 5What pages have userl and user2 visited?What path have they followed?brwsrbrwsrbrwsrbrwsrbrwsrbrwsrbrwsrbrwsrbrwss+ What is the amount of lime they hnve spent onpagclpages'! (see notc below)e Have they connected from local network, i.e., used forcxample computers provided by schoolluniversity (1Paddress)?What WWW-browser software have they used?As themselves, the results arc not very descriptive, but ifthese measures and iirls are mapped with respect to thc CBEmaterial they present, they can be of great value in learningevaluation. For example, we can observe that userl hasstudied 6 pages (topics), spent certain amount of time on apagelpages, has followed il linear path, has used a certainWWW-browser and cornpiiter coniiectcd to university localnetwork. User2 on the other hand has studied 3 pagcs incertain timc, has jiimped to a ncw iopic from pagc 2 and hasused university modem connections.Furthermore, if there is an interactive task on page 4 thatrequires applying information from pagcs 1-3, it would indicatethat user1 has indeed managed to overcome the taskand thus leanid that topic.It shnuld be noticed that because the WWW log-systemis based on requcst processing, the timing is not accuratein certain cascs. For instance. users may jump to somc otherwebsite by typing in thc address directly, later returning backto the original page and the jump would not be seen in thelogs. However, in many cascs il is reasonable to assume thatusers study educational material with concentration.Another important Feature of the WWW log-system is thatthe logging is always performed on the particular WWWserver containing the material. This means that if materialis distributed over a range of servers, thc log data needs tnbc combined before analysis or advanced solutions like redirectingneeds to be used.The final notice regrading this example is that if the CBEmaterial only contains basic or reference material, but notany interactivity or tasks that require applying information,the log-based evaluation gives metadata type informationthat is most useful when combined with other evalnationmcthods. On the other hand, in case of sclf-study, interactiveWeb-based education, the log analysis can provide avery good picture of the student's leaiming process and thusbe used ils an evaluation mensure by itself.10. ADVANCED WWW STRUCTUXE'i'o USC logging for truc interactive guidance, basically everysingle web page lias to be turned into a call for r programin the server. Onc solution (see Figurc 1) for this is lo usea User Agent to filter traffic between client and server [3],(Note: User Agent here is complctcly different from the onementioned in section 7).0-7803-5643-8/99/$10.00 @ 1999 IEEENovcrnber 10 - 13,1999 San Juan, Puerto Rico29th ASEE/lEEE Frontiers in Education Confercncc12~13 -20

Session 12a3Server I ClimtIIIIest in learning evaluation. Naturally, these would be casedependentparameters and certainly inorc difficult to implcment.Besides cvaluation of learning, logging is R very usefulconcept in other related areas as well. An example could bcpsychological testing, where it is sometimes cven mare iinportantto know how a person takes the test than whai hdsheanswered. Creating such test in an electronic form ratherthan traditional paper version, and including logging in thesyslem, would result in an cxtended set of material and, perhaps,increased accuracy for psychological evaluation.12. REFERENCESFigure 1: An advanced WWW architectureIn this structure, a User Agent represents the real user andtakes carc of all the requests made to thc server. It respondsto these requests with appropriate data according to spccihcations made by the developer. Thus itrepresents the teacherits well and the nppropriate term in this case would be n DUMbleAgeng.There are a numbcr of ways to implement such architecture,but these are very sophisticated and far beyond thescope of this paper. In principlc, this type of structure allowslogging to be applied in both off-line evaluation and on-lineinteractive control.13. CONCLUSIONSCompared to traditional methods, log-based evaluntion hascertain advantages. Particularly, logs will include real informationof the learning p~~occss itself, which is quite impossibleto expose with traditional methods like written examinationsnr assessment forms. The concept of logging alsosupports the cugnifive-conslr!tctive theory of learning I cl],prnvided that the CBE material itself was constructed in thismanner.Logging also adds ncw possibilities io student selfevaluation,which in university lcvel studies and studentorientedlearniiig is very importnnt. Furthermorc, log-basedevaluation includes a strong dependency un the contents,same as cxaminations and Fccdback forms. When used incombination with cxaminntions or fecdback forms, loggingcan, however, provide a more accurate evaluation OF learning.Furlhermore, loggingdoes not havc to be restrictcd to userand transaction relatcd data but morc content orientcd parameterscould be used as well. For example content descriptinnsor content lcvcl would be parameters of inter-[I] Karjalainen M., Rahkila M., “Learning signal processingconcepts and psychoacoustics in the QuickSig DSPenvironment”, in Pmc. 1995 ?EEE Irrt. Con$ Acoust.,Speech, nnd Proc. (ICASSP’H), Detroit, Michigan,USA, vol2, pp. 1125-1128, May 9-12,1995.http://www.acoustics.hut.fi/”mara/publica~ions/icassp95/[2] Wiezel, A., “Measuring thc success of virtual tutoring”,in Proc. 1998 IEEE Frontiers in Education(FIE’98), Tempo, Arizona, USA, Nov 12-15,1978.http://€airway.ecn.purdue.edu/-€ie/~ieg~/papera/l357.pdf[3] Rahkila M., Karjalainen M., “An Experimental Architecturefor Interactive Web-based DSP education,” inProc. I998 IEEE hit. Cot$ Acotrst., Speech, and Six.Proc. (ICASSP’98). Seattle, Washington, USA, May12-15,1998. pp. 1857-1860,http://www.acoust;ics,hut.fi/-mara/publications/icasspga/[4] Fielding, R., et al., “Hypertcxt Transfer Protocol -HTTP/1.1”, RFC 2068, January 1997.ftp://ftp.isi.edu/in-notes/rfc206E.txt[5] Kristol, D., Mnntulli, L., “HTTF Statc ManagementMechanism”, EFC 2109, February 1997.ftp://ftp.isi.edu/in-notes/rfc2109.txt[6] Bruner, 1. S., “Tnwrrd n Theory of Instruction”, IhrvardUniversity Press, Cambridge, MA, USA, 1966.November 10 - 13,1999 San Juan, Puerto Rico29th ASEEIIKEE Frontiers in Education Conferencc12n3-21