Author Guide for Preparing Your JISE Paper - IFIP Working Group 2.13

courses students would have completed prior to the ISEcourse are:• Semester 1: C language (Basic Constructs)• Semester 2: Advance C language, UNIX• Semester 3: C++, Logic and FunctionalProgramming• Semester 4: PROLOG, CompilersDuring semesters 1 to 4 students would have acquired certainsoftware development skills (liu, 2005) which may be vitalto the software testing aspect of the ISE course and theimplementation of our framework. These skills are:• Writing small programs (usually in C language) astheir programming assignments• Developing software in teams and collaborating insmall-scale software projects.In the ISE course we try to make students understand thedifference between testing the small programs they write forthemselves in class and as assignments and the testing oflarge scale software products that they might deal with whenthey graduate. The teaching and learning context focuses onthe identification of software faults and failures, unit andintegration testing, function and performance testing, writingand execution of test plans/cases, etc.Two lecturers were involved in the pilot study. One wasresponsible for scheduling F/OSS activities, the other actingas an adviser. Students at their previous semesters havealready been taught programming, so coding is not a focalpoint of the ISE course. Instead, focus is placed on otheractivities such as software testing.The framework shown in Figure 1 is in three phases. Eachphase describes a context in which students get involved inF/OSS projects activities. Their involvement was basic.Students select a project and download and use the software.Any problems they encounter in the use of the software arereported to the project's community for action. Their maintasks were to find and report bugs in their respectiveprojects. These tasks may take the form of functional,usability, or smoke testing. In what follows, we discuss eachphase in turn.Figure 1. F/OSS Framework for teaching softwareengineering courses, (Sowe et al. 2006a; pp.262).2.1 Phase 1Phase 1 was a preparatory stage in which the lecturerscheduled classroom activities and guided the students intheir project selection. We discussed with the 150 students inthe ISE course about involving them in software testing inF/OSS projects. Fifteen students volunteered to take part inthe pilot program, but only thirteen students completed theexercise. The F/OSS development process is different fromtraditional software development that students are taught intheir CS courses. Thus, it’s vital at this phase that studentsare introduced to F/OSS. Our introductory lectures were onthe following topics:• What is F/OSS? This section covered the F/OSSdevelopment process, activities in projects, the rightsvarious licenses (e.g. GNU/GPL) grants the user ofF/OSS, etc.• F/OSS communities: Formation, structures andmembers’ roles. We discussed communities in theLinux, Apache, and Debian projects.• Communication: We discussed etiquettes of forums,mailing lists (moderated and un-moderated), andInternet Relay Chats (IRCs).• Collaborative platforms: We introduced the studentsto CVS, Tinderbox, Bugzilla, bug tracking systems(BTS) and how to browse bug databases.At the end of the introductory lectures the students wereguided to explore sourceforge.net, a repository of F/OSSprojects. This session was intended to give the students a feelas to the category of F/OSS projects available on theInternet. At the end of the exploratory process the studentsselected their projects. In choosing a project, the studentsfollowed these F/OSS projects selection criteria.• Operating system/platform (Linux, Windows, etc).Students may choose projects which run on platformthey are most comfortable with.• Size of ownership/developers. According to the Bazaarmodel (Raymond, 1999), we expect a project with more“eyeballs” to have higher software developmentactivity. Therefore, we encouraged the students to selectprojects with three or more developers.• Development status (Alpha, Beta, Mature, etc). Weencouraged the students to use the alpha and betareleases. These versions of the software are released tothe F/OSS community for debugging andimplementations of functionalities. Much projectactivity is centered on these versions. The mature andstable releases are not likely to generate muchdiscussion in which students can contribute becausemany of the critical bugs may have been removed.• Programming language (C, C++, etc). If students areto take part in coding activities, they should chooseprogramming languages they are most comfortablewith. Coding was desirable but not necessary task forthis pilot study.• Extensive collaboration in lists/forums. Most projectactivities take place in forums and lists. So it’simportant that students choose projects with activeforums. This is mostly the case with projects that arehosted at sourceforge.net but also having their own websites.Each student was asked to prepare a report on his/herselected project for class presentation. In their presentationseach student gave a brief history of his project and listed theproject's characteristics based on the F/OSS projectsselection criteria.

2.2 Phase 2During this period the students learned how to register intheir projects, use bug tracking systems, and browse andreport bugs. Each student sent his/her project name and logindetails to the lecturer. These details were used to trackstudents’ activities in their projects. Every time a studentsubmitted a bug, he/she notified the lecturer. Students wereasked to continuously login to check the status of theirsubmission. They could work in their projects anytime andanywhere they felt like. The students implemented thetesting strategy shown in Figure 2. They applied testingtechniques such as smoke tests, functional tests, usabilitytests, etc.• Like, dislike, and their future plans (if any) in theproject selected.3. GRADING AND EVALUATION APPROACHAt the end of the pilot study we evaluated the students basedon the presentations they made in class, their participation intheir respective projects, and their testing activities.Furthermore, we conducted two online surveys in order tocapture the students’ opinions and experiences in testing inF/OSS projects.The presentations of the students are available at ourresearch group’s ftp (see Endnote 1). The 16 projects thestudents tested in are shown in Table 1.Table 1. Projects students tested in.Figure 2. F/OSS Testing Strategy.As shown in Figure 2, the students downloaded the softwareto be tested and applied various software testing techniques.This may result in the discovery of bugs (design faults,improvements to be incorporated into the next release, etc)which are then logged into the project's bug database usingstandard bug reporting procedure and tools (e.g. BugTracking Systems). Where a student was not able to find abug, he/she ran more tests on the software or selectedanother project to continue testing.In the fifth week the students were asked to make anotherclass presentation. In presenting their experiences, thestudents discussed the types of bugs they found, how thebugs were found, what they thought caused the bugs, howthey reported the bugs, what responses, if any, were receivedfrom other participants, and any other problems theyencountered.2.3 Phase 3At the end of the pilot study the students were sent a slidepresentation template and asked to make a final fifteenminutes presentation. The layout of the presentation was asfollows:• Particulars (Course title, real name, email, and studentid).• Project details (name, login id, website, brief history,and screen shots).• List of Testing Activities (number of bugs found (bfn),bugs reported (brp), bugs fixed (bfx), and number ofreplies (rep) received). Students should give the URL ofthe variables brp, bfx, and rep.The online surveys and their respective URLs are inEndnotes 2 and 3. The students were graded and the marksawarded as their coursework (50% of their final grade),written exams accounts for the other 50%. The grading wasdone as follows:• Class presentation (10%). 3 points for each of thepresentations made in Phases 1 and 2. And 4 points forthe final presentation in Phase 3.• Project participation (12%). Measured by the numberof emails we exchanged with the student about hisproject.• Working with testing tools (13%). How a student usedand understood the bug tracking system or bug databasein his project.• Testing activity (TA) 15%. Measured by fourvariables; (bfn), (brp), (bfx), and (rep).In what follows we discuss the results of the students testingactivities and their responses to the questions in our surveys.3.1 Students Testing ActivitiesResults of students testing activities are described in terms ofthe four variables (bfn, brp, bfx, and rep). Table 2 shows asimple summary of students testing activities.bfn brp bfx repN Valid 13 13 13 13Missing 0 0 0 0Mean 5.538 5.231 1.150 3.310Median 4.000 4.000 1.000 3.000Std. Deviation 3.017 2.743 1.281 2.175Range 8.0 9.0 3 7Minimum 3.0 2.0 0 1Maximum 11.0 11.0 3 8Sum 72.0 68.0 15 43Table 2. Descriptive statistics of students testing activities

In total, the 13 students tested in 16 F/OSS projects, found72 bugs, reported 68, fixed 15, and received 43 replies fromthe F/OSS communities in their projects. The mean values ofbugs found and reported per student were 5.54 and 5.23,respectively. These figures show that the students reportedslightly less bugs than they found, because some of the bugsthey found were already reported. Even though the studentsperformed well in finding and reporting bugs in theirprojects, they did not do well in fixing bugs (mean=1.15).This is because they were not required to do any coding inthis part of their course.Number1210864bfn brp bfx repin your project?Was the process of reporting bugs easy? Q4 Q6Would you prefer to do other courses in Q9 Q8Open Source software?Did you read and understand the bugs Q10 Q11others reported?Are you considering participating inyou project after you graduate?Q20 Q14Table 3. Questions common to both surveysComparing the responses of the students to the commonquestions, we were able to make the following conclusions:ST1Q1 vs ST2Q3: All 11 students responded "Yes" in bothsurveys. This means that throughout the pilot study studentsenjoyed software testing in their projects.201 2 3 4 5 6 7 8 9 10 11 12 13StudentsFigure 3. Stacked Bar charts showing how the studentsperform in each variable.As shown in Figure 3, students #2, #3, #4, #5, #7, #011 and#13 have low to moderate performance in all of theiractivities. Students #1, #6, #8, #9, #10, and #12 haveperformance only in three variables (bfn, brp, rep), withstudent #6 performing much better than the rest of thisgroup.Count876543210Easy to get a projectYes NoSurvey 1 Survey 2SurveysFigure 4. ST1Q2 vs ST2Q183.2 Students Opinions and Experiences: SurveysWe invited the students via email to complete twoanonymous online surveys. The surveys were designed andpublished on the department’s website using PHPSurveyor.The surveys items were meant to measure• the pedagogical value of the study,• students' opinions and experiences about softwaretesting in F/OSS projects, and• how much the students have benefited by contributingand learning from testing in F/OSS projects.Survey one was conducted in week 6 and consisted of 21items. We called it an intervention survey because it allowedus to intervene early and focus attention on difficultiesstudents were having (e.g. ease of finding a project, processof reporting bugs). Survey two was conducted in week 13and consisted of 19 items. The response rate for both surveyswas 84.62% (11 out of 13 students). Seven items (Table 3)from survey one were repeated in survey two so that wecould compare students' responses to the common questionsand see how their motivation and perception has changedovertime.VariablesItems/QuestionsSurvey1Survey2Do you enjoy software testing in F/OSS Q1 Q3projects?Did you find it easy to get a project to Q2 Q18participate in?Was it easy to find bugs in the software Q3 Q5As shown in figure 4, in both surveys the students expressedthat it was not easy to find a project to participate in. Thiswas surprising because we were expecting the students tofind projects very easily within the myriad of F/OSS projectsat sourceforge.net.Counts76543210Easy to find bugsYesSurvey 1 Survey 2SurveysFigure 5. ST1Q3 vs ST2Q5:In both surveys the answers are the same. Most of thestudents found it difficult to find bugs in their software. Thiswas the case at the beginning. But as they became familiarwith the software, the rate of finding bugs increasedgradually.No

NoST2Q15: How often do you login to check your bug reports?Please choose *all* that applyEverydayEvery 2 daysOnce a weekMostly during the weekendsSometimes at Internet CafesMostly at homeWhen am at the UniversityAt a friend's placeST2Q16: Was the people in your project friendly?YesNoST2Q17: Did you have any communication problem duringyour testing?YesNoST2Q18: Did you find it easy to get a project to participatein?YesNoST2Q19: Did you have experience with software similar tothe one in your project?YesNo9. REFERENCESAlzamil, Z. (2005), “Towards an effective softwareengineering course project.” Proceedings of the 27thinternational Conference on Software Engineering, ICSE'05. ACM Press, pp. 631-632.Barahona, J. M., Tebb, C. and Dimitrova, V. (2005),“Transferring Libre Software development Practices to theProduction of Educational Resources: the EdukalibreProject.” First International Conference on Open SourceSystems, Genova, Italy.Carrington, D. and Kim, S. (2003), “Teaching SoftwareEngineering Design with Open Source Software.” 33rdASEE/IEEE Frontiers in Education Conference, Nov. 5-8,Boulder, CO.Faber, B. D. (2002), “Educational models and open source:resisting the proprietary university.” Proceedings of the 20thAnnual international Conference on ComputerDocumentation, SIGDOC '02, ACM Press, pp. 31-38.“FLOSSCom” (2006). Retrieved Oct. 20, 2006, fromhttp://www.flosscom.net/“Flosscom.net Kick-off meeting”, November, 2006http://flosscom.net/index.php?option=com_content&task=view&id=23&Itemid=40German, M. D. (2005), “Experience teaching a graduatecourse in Open Source Software Engineering.” Proceedingsof the first International Conference on Open SourceSystems. Genova, pp.326-328.Hans, V. (2005), “Some Myths of Software EngineeringEducation.” Proceedings of the 27th internationalConference on Software Engineering, ICSE '05. ACM Press,pp. 621-622.Hippel, E., and Krogh, G. (2003), “Open Source Softwareand the "Private-Collective" Innovation Model: Issues forOrganization Science.”Organization Science,Vol. 14, pp.209-223.IEEE/ACM Joint Task Force on Computing Curricula,Software Engineering 2004 Curriculum Guidelines forUndergraduate Degree Programs in Software Engineering.Retrieved November 27, 2005, fromhttp://sites.computer.org/ccse/SE2004Volume.pdfLiu, C. (2005), “Enriching software engineering courses withservice-learning projects and the open- source approach.”Proceedings of the 27th international Conference onSoftware Engineering, ICSE '05. ACM Press, pp. 613-614.Mantis Bug Tracker (2006), Retrieved April 27, 2006, fromhttp://www.mantisbt.org/Megias, D., Serra, J., Macau, R. (2005), “An InternationalMaster Programme in Free Software in the European HigherEducation Space.” Proceedings of the first InternationalConference on Open Source Systems. Genova, pp.349-352.SQO-OSS Workshop, “Open Source Software Workshop”(2006), Open Source Software: Research Communities andIndustries. Thessaloniki, 20 December, 2006, available at:http://www.sqo-oss.eu/events/public-workshop/Ozel, B., Cilingir, B., Erkan, K. 2006 (Eds.) Towards OpenSource Software Adoption. OSS 2006 tOSSad workshopproceedings, Como, Italy, pp.79-88PHP Surveyor (2006), Retrieved April 27, 2006, fromhttp://www.phpsurveyor.org/index.phpRaymond, S. E. (1999), The Cathedral and the Bazaar.Musings on Linux and Open Source by an AccidentalRevolutionary. O'Reilly, Sebastopol, USA.Sowe, S.K., Karoulis, A., Stamelos, I., Bleris. G.L. (2004),“Free/Open Source Software Learning Community andWeb-Based Technologies”. IEEE Learning TechnologyNewsletter, Vol. 6(1), 2004. pp 26-29.Sowe, S .K., Stamelos, I., Deligiannis, I. (2006a), “AFramework for Teaching Software Testing using F/OSSMethodology.” In Damiani, E., Fitzerald, B., Scacchi, W.,Scott, M., Succi, G.,(Eds.), (2006). IFIP InternationalFederation for Information Processing, Open SourceSystems, Vol. 203, (Boston: Springer), pp. 261-266

Sowe, S.K., Stamelos, I., Angelis, L. (2006b), “An EmpiricalApproach to Evaluate Students Participation in Free/OpenSource Software Projects”. IADIS International Conferenceon Cognition and Exploratory Learning in Digital AgeCELDA 2006, 8-10 December, 2006, Barcelona, Spain,pp.304-308Sourceforge.net (2006), Retrieved April 27, 2006, fromhttp://sourceforge.net/Pfleeger, L. S. 1998. Software Engineering. Theory andPractice. Prentice Hall, pp. 278-392.Lakhani K. R, and von Hippel E. (2003), “How open sourcesoftware works: "free" user-to-user assistance”. Researchpolicy 32 (6), pp. 923-943Holtgrewe U. (2004), “Articulating the speed(s) of theInternet: The case of Open Source/Free Software. Time &Society 13 (1): 129-146 mar 2004Sowe, S.K., Angelis, L., Stamelos, I. (2006c), “IdentifyingKnowledge Brokers that Yield Software EngineeringKnowledge in OSS Projects”, Information and SoftwareTechnology, Vol. 48, 2006, pp. 1025-1033