ComTest = TDD + D + D, Testing in Introductory Level Programming TDD + design (D) + documentation/examples (D) Vesa Lappalainen, Jonne Itkonen, Tomi Karppinen
Agenda i.jyu.fi/comtest Problem and motivation Basic idea and usage of ComTest tool Examples (demonstration/video) Assumed benefits Lessons learned (initial evaluation) Future plans
Problem and motivation 1/2 Many have tried to introduce TDD and unit level testing to early programming courses (see references of the paper) Experiences vary: some report success (e.g., Janzen & Saiedian 2006, Wellington et al. 2005), but some identify an increased cognitive and technical load for students (e.g., Keefe et al. 2006, Marrero & Settle 2005) Our research motivation is based on the problem of increased cognitive and technical load
Problem and motivation 2/2 The problem was addressed with a testing tool JUnit was the first choice, but turned out problematic due to increased technical load CS2 students demonstrated a clear resistance to test writing with JUnit A more "transparent" tool was needed, hence ComTest We have collected experiences from more than twenty of our course offerings (3*CS1/Java, 7*CS1/C#, 7*CS2/Java, 5*CS2/Java summer) with ComTest tool, and report initial lessons learned
Basic idea and usage of ComTest ComTest tests are written using a simple macro language (for Java, C#, C++) Tests are written into JavaDoc/Doxygen comments to also get documents and examples ComTest are automatically translated into JUnit/NUnit tests for execution (one can also mix JUnit/NUnit and ComTest) Eclipse/Visual Studio plug-in is used to facilitate test writing and execution
Example 1 JUnit: assertequals(false, isleapyear(1900)); ComTest: isleapyear(1900) === false see example of simple function without any side effects Link to video» LeapYearReady.java» test/leapyearreadytest.java» doc/example/leapyearready.html one can also use syntax: isleapyear(1900) => false
Example of student written test * @example * <pre name="test"> * karkausvuosi(0) === true; * karkausvuosi(4) === true; * karkausvuosi(400) === true; * karkausvuosi(1000) === false; * karkausvuosi(1900) === false; * karkausvuosi(2000) === true; * karkausvuosi(3000) === false; * karkausvuosi(1805) === false; * karkausvuosi(1988) === true; * </pre> "I would never get the leapyear done without ComTest" In C++ Course: "Is there no ComTest for C++"
Example 2 Example from function with side effects: Link to video TakeCharsReady.java test/takecharsreadytest.java doc/example/takecharsready.html It is also possible to use tabular syntax to make complicated tests more readable: * $s $n $letters $result $left * ---------------------------------------------------------------------------- * "Mieleni minun tekevi" 3 "aeiouyåäö" "iee" "Mlni minun tekevi" * --- 3 "xyz" "" "Mlni minun tekevi" * --- 4 "klmn" "Mlnm" "i inun tekevi" * --- 4 "e" "ee" "i inun tkvi" Compare: https://trac.cc.jyu.fi/projects/comtest/wiki/vertailuerottelija
Benefits ComTest tests are shorter and easier to write than their JUnit counterparts Tests in JavaDoc comments keep the tests close to the implementations the cognitive load in browsing between tests and implementations diminishes ComTest is a tool for designing classes and methods, particularly interfaces Tests inside JavaDoc comments serve as examples of method usage
LESSONS LEARNED 1 CS2 N Tool users AVG n of tests ComTest JUnit ComTest JUnit 2007 93 - s=57,t=150-2.6 2008 95 s=75,t=258 s=23, t=69 3.4 3.0 2009 99 s=80,t=264 s=15, t=37 3.3 2.85 s=nr of students written tests, t=nr of tests Table 1: The number of ComTest and JUnit users and their test writing in a CS2 weekly assignment more students wrote tests using ComTest they wrote more tests compared to JUnit users
LESSONS LEARNED 2 Open questionnaire: Mitä on ohjelmointi? Mikä on tärkeää ohjelmoinnissa? Mitä taitoja ohjelmoinnissa tarvitaan? Kerro, mistä asioista... Pidät? Et pidä? Millä tavalla käsityksesi ohjelmoinnista on muuttunut suorittamiesi kurssien aikana? Open questionnaire Nr of students with %test% Before CS1-2013 7/218 After CS1/Before CS2 40/207 After CS2-2014 83/138 Positiivista: Testeistä tuleva vihreä viiva tuottaa hyvän mielen raskaan yritys-erehdys-näpyttelyn jälkeen. Aiemmin ajatteli vain koodin kirjoittamista, mutta ohjelmointiin kuuluu myös testaaminen. Huolellinen suunnittelu, testaus ja näiden kautta ylläpidettävän, toimivan ja selkeän koodin kirjoittaminen. Testit kiinnittivät monesti huomiota ohjelmissa oleviin ongelmiin, jotka eivät selvinneet ohjelman toimiessa varsin hyvin rajatussa tehtävänannossa. Testit myös auttoivat miettimään, kuinka laajentaa ohjelmien käytettävyyttä parametrien muuttuessa. Haasteita: Se ei ole kivaa, kun itse ohjelma on lyhyt ja testit vievät ohjelmaa enemmän tilaa ja aikaa. Minusta kääntäjän virheilmoitus kertoi enemmän kuin excpected but was Tykkäsin enemmän itse muutella arvoja ja testailla tuleeko oikeita tuloksia.
Challenges GUI-testing Only few (Swing) examples in CS2, not obligatory Understanding that tests are also for later use as are also comments Visual Studio installation (CS1) Too small examples
Future plans Improvements to plug-in (auto-complete, syntax check) Quick Check (random testing) Mutation testing (test quality?) Test protected and private methods To get other users also A content analysis and a phenomenographic study on student conceptions of programming: how well the curriculum supports different areas of expertize, including testing => improving teaching with ComTest (work-in-progress)
Questions?