Written summary f peer testing activities Tested grup: Xylphne Tested system: Htelzn htel bking system Time: 18.2.2007 Lcatin: Maari Duratin: 16.00-20.00 Tester grup: Game On! Slutins Testers: Ism Lehmus, Antti Pihamaa, Tapi Auvinen Authr f this reprt: Ism Lehmus Peer testing prcedures The peer testing was executed by tw primary means f testing: Usability tests and Explratry testing using SBTM. Usability test prcedures All three testers were asked t execute heuristic evaluatin f the htel bking system. Instructins were t g thrugh all views f the system and investigate all heuristics n ne view at a time and then mve t the next view. All findings were written t papers which were given t Jhn Dannberg. Explratry testing The explratry testing was executed by the testers fllwing SBTM practices. The testers were given free access f the system and advised by Xylphne t test the system cmpletely but specifically n the flw f the system. While testing, the testers filled in a test charter t dcument their findings. A cpy f the test charter can be fund in appendix A. Summary f peer testing results All in all the system is currently in usable cnditin. It is able t prvide the functins that are needed but there still are quite a few annying defects in the system. These defects were mainly usability prblems and missing input validatin but als sme real bugs were detected. The detected bugs were reprted t the Xylphne s Bugzilla as instructed. Usability testing summary As nted in the previus chapter, mst prblems seem t be in the usability f the system. The usability prblems were detected using heuristic evaluatin and fund prblems with
the descriptin and a pssible slutin were written t paper which was given t Jhn Dannberg. Mst f fund prblems were nly minr esthetic glitches but als few bigger usability prblems were fund. Sessin based testing summary Cverage All aspects f the htel bking system were tested. The testers fcused n the flw f the system. Nt all pssible paths were tested because the number f cuntries and cities was t large. All the main cmpnents and functinalities were tested. Belw is the list f views that were tested. Main screen Searching fr a htel/rm Lking at htel inf Making a reservatin Searching / cancelling a reservatin Rugh evaluatin f quality Main screen Seems k Searching fr a htel/rm Seems fine but a scrll bar fr drp dwn menus wuld be nice. Als it s impssible t knw which city is which if there are many cities with the same name in ne cuntry. Lking at htel inf N extra inf n htel available. If user ges back frm viewing different htels available and searches with different criteria, the ld results may still be printed n the result screen althugh it is nt pssible t select them anymre. Making a reservatin This view has few serius defects. It desn t really have any input validatin. Fr instance the telephne number can be anything, even letters. Als the errr messages shuld be mre clear as they currently d nt tell which infrmatin was incrrect when trying t make a reservatin. Searching/cancelling a reservatin Searching fr a reservatin seems a bit awkward as it requires bth reservatin number and the the name f the persn wh made the reservatin. Is there a real reasn fr this r wuld either ne be sufficient?
Imprvement issues Sme imprvement issues nt recrded as bugs were fund: - N shrtcut t find a htel that is knwn befrehand - The clr shrtcut at the bttm f the page als as links s that they can be accessed withut actually using the clr buttns n the remte cntrl
Appendices Appendice A: Test charter Appendice B: Test charters f testers
Appendix A: Test charter 1. Tested areas Please test the fllwing functins: Make new reservatins in different htels, areas and different times Make nte f all made reservatins: First name, last name and cnfirmatin number. This is in rder t be able t delete the reservatins later. Search fr reservatins Delete reservatins 2. Gal and fcus The primary gal is t assess the verall flw f the system Is the system cnvenient t use? Are all needed inf included fr making, searching and deleting reservatins? Secndary gals are: Find bugs and minr glitches Prvide input n future develpment pprtunities 3. Apprach The nly tls yu need are ur prgram and the XleTView emulatr as described abve Please make sure all required functinalities are cvered in enugh detail. A suggestin f time divisin between functinalities is 3 hurs making reservatins ½ hurs searching fr reservatins ½ hurs deleting reservatins
Appendix B: Test charters f testers 4. Explratin lg SESSION START TIME: 2007-02-18 18:15 TESTER: Ism Lehmus ENVIRONMENT: XleTView + Debian 4.1 Task breakdwn DURATION (hurs): 1 TEST DESIGN AND EXECUTION (percent): 75% BUG INVESTIGATION AND REPORTING (percent): 20% SESSION SETUP (percent): 5% CHARTER VS. OPPORTUNITY (percent/percent): 30/70 4.2 Data files N data files fr testing are prvided 4.3 Test ntes Brwsing thrugh the cities n the reservatins screen. Scandinavian letters (such as Å, Ä and Ö) are displayed wrng. Als the names f the cities are nt always clear. Fr instance in United States there are many cities with the same name. If ne first checks htels in ne place (Washingtn, United States fr instance) and then ges back and checks htels in a smaller place (say, Palau), there still is sme f the htels in Washingtn mentined in the list. N shrtcuts t find the wanted city. User must scrll thrugh a mnstrus list f cities. Accepts anything as user's telephne number. 4.4 Bugs Please use bugzilla fr bug reprting as described belw
4.5 Issues Any bservatins, issues, new feature requests and questins that came up during testing but were nt reprted as bugs. 4. Explratin lg SESSION START TIME: 2007-02-18 18:00 TESTER: Antti Pihamaa ENVIRONMENT: Emulatr + Linux ubuntu 4.1 Task breakdwn DURATION (hurs): TEST DESIGN AND EXECUTION (percent): BUG INVESTIGATION AND REPORTING (percent): SESSION SETUP (percent): CHARTER VS. OPPORTUNITY (percent/percent): 4.2 Data files N data files fr testing are prvided 4.3 Test ntes (BUG): syötetty bugzillaan (HE): raprtitu heuristisessa arviinnissa Aletaan luda uutta varausta. Valitaan aika ja paikka Valitaan htelli, screenin tsikk ei näy kknaan. (BUG) Valitaan hunetta. Rm sarakkeen sisällön merkitys ei avaudu (HE).
Rate sarakkeen sisällössä kdin BAR1 merkitys ei avaudu (HE). Rate sarakkeen sisältö ei näy kknaan (BUG). Rate/date ja ttal sarakkeiden yksikkö puuttuu (HE). Help ei saatavilla (HE). Htel inf ei saatavilla (BUG). Varaustiedt. Alatsikk ei näy kknaisuudessaan (HE). Htellin site tyhjä (HE). Päivämäärän esitysfrmaatti epäselvä (päivämäärän sisäinen ja päivämäärien välinen ertusmerkki sama) (HE). Rm type arv sama kuin edellisellä sivulla rm (tsikn käyttö ehkä hieman epäknsistentti). Arvn merkitys epäselvä (HE). Prize yksikkö puuttuu (HE). Cancellatin plicy, sisältö ei mahdu ruutuun kknaan (HE). Repay plicy erillään muista (BUG). Input bking infrmatin Expiratin date letusarv 1 (vältä virheitä esitäyttämällä järkevästi) (HE) Card number ei mahdu näkyviin (HE) Errr: ilmeisesti inputissa vikaa mutta virheilmitus ei kerr mitä (HE). Cmmitted DEMO8207055: AAAAA, BBB Ohjeteksti ei mahdu kknaan ruudulle. (BUG) Js tästä valitsee back ja taas nex, tulee virheilmitus. Js ei le järkevää päästä takaisin ei sen pitäisi lla mahdllistakaan. (BUG)
Linkki takaisin päävalikkn siirtymiseksi lisi myös kiva ja luultavasti npeampi (ainakin näppäimistöllä) kuin värikdi? Search reservatins, etsitään tehty varaus Reservatin inf Htellin nimi ja site tyhjä (BUG) Laskun lppusumma ilman yksikköä ja neljän desimaalin tarkkuudella (BUG). Oletustimintna cancel liian esillä ja herkästi vahingssa valittavissa -> turvattmuuden tunnetta vaikka varmistuksen takana. (BUG) Ei nrmaalia linkkiä takaisin alkuun siirtymiseksi. (BUG) Peruutetaan varaus. Mikä n cnfirmatin message. Tälle käyttäjälle tuntematn termi. (BUG) Cancel id: CNCL8207055 Uusi varaus (HZF test htel). Antti Antti Valitse hune. Rate sarakkeen sisältö en näy kknaan (HE). Syötetään varaustiedt linkki next humaamatn, pelkkä teksti, ympäröivä laatikk tai jku muu navigaatita kuvaava visuaalinen indikaati selkeyttäisi. (BUG) DEMO3247666 Cancelidaan varaus. k. 4.4 Bugs Please use bugzilla fr bug reprting as described belw
4.5 Issues Any bservatins, issues, new feature requests and questins that came up during testing but were nt reprted as bugs.
4. Explratin lg SESSION START TIME: 2007-02-18 18:00 TESTER: Tapi Auvinen ENVIRONMENT: xletview + Linux 4.1 Task breakdwn DURATION (hurs): 1 h TEST DESIGN AND EXECUTION (percent): 100% BUG INVESTIGATION AND REPORTING (percent): 100% SESSION SETUP (percent): 100% CHARTER VS. OPPORTUNITY (percent/percent): 50%/60% 4.2 Data files N data files fr testing are prvided 4.3 Test ntes Test ntes that describe what was dne, and hw. Detailed enugh t be able t use in briefing the test sessin with ther persns. Detailed enugh t be able t reprduce failures. Creating a new reservatin Setting arrival date that has already passed -> an errr message is shwn Searching duble rm -> HZF test -> Standard rm Entering a very lng name -> Leads t an errr Entering letters t phne number: accepted Trying t cancel the reservatin Searching takes a very lng time Reservatin cancelled Trying t find a reservatin with the wrng reservatin number -> leads t an errr message Trying t find a reservatin with smebdy elses name and a randm reservatin number -> leads t an errr 4.4 Bugs Please use bugzilla fr bug reprting as described belw 4.5 Issues Any bservatins, issues, new feature requests and questins that came up during testing but were nt reprted as bugs.