Screenshot

arc42

© This document uses material from the arc42 architecture template, freely available at http://github.com/arc42.
arc42 is open source and provided under the Creative Commons Sharealike 4.0 license. It comes without any guarantee. Use on your own risk. arc42 and its structure by Dr. Peter Hruschka and Dr. Gernot Starke. Asciidoc version initiated by Markus Schärtel and Jürgen Krey, completed and maintained by Ralf Müller and Gernot Starke.

Note
RDM: I used the arc42 template for the documentation of this plugin since it just helps a lot in writing documentation. It is not a perfect example for arc42, since the plugin is just a small project and does not need much architecture. See http://www.arc42.de for better examples :-)

1. Introduction and Goals

Geb — together with Spock — is a great combination if you need to create functional tests for your Grails application. But the output is basically just the information that the tests succeeded — at least when there was no failure :-)
Out of the box, it does not report on what has been tested. If you want to get details about the tests, you are expected to read the sources.

A not so well known feature of Geb is the report feature:
whenever you use the command report "some text", geb tries to create a screenshot and save the current DOM as HTML.

This plugin crawls through the generated screenshots and creates an HTML report which displays preview images as a film strip at the top of the report-page and the test-result on the main part of the window. When you click on an image, it will be displayed in full size, if you click on the name of a report image, the corresponding HTML file will be displayed. Without the proper path to referenced files, this view will look broken, but it is still useful for debugging. When you click on the small star in front of the name, the link will take you to the recorded original URL of this test — so if your project is running, you will be taken to the right page on your web server and you can manually verify what happened.

The aim of this plugin is not only to help a tester in developing Geb/Spock tests and see what went wrong, but also to create a report for your customer. With all those screenshots you can easily get an graphical overview of the application and see if all pages display correctly.

1.1. Quickstart

There is not much to do in order to use the plugin. Just configure it in your BuildConfig.groovy

test ':film-strip:0.1.4'

and add the folowing line to your gebConfig.groovy:

reportingListener = new grails.plugin.filmstrip.FilmStripReportingListener()

If your geb reportDir is the standard one, the report will be generated after running grails test-app.

You will find the generated report in target/test-reports/geb

There is a fork of the Gails-Pet-Clinic sample application on https://github.com/rdmueller/grails-petclinic which contains two branches (film-strip-plugin-2.2 and film-strip-plugin-2.4) with Spock/Geb tests and the film strip plugin installed. See the readme on how to execute the tests.

1.2. Requirements Overview

ID Priority Description

R1

high

the system has to produce an easy to navigate visual report containing all screenshots reported by geb

R2

medium

the system has to be easy to use. Best would be install and run with some reasonable default configs

R3

medium

the system should be configurable in terms of html output and css style. Some kind of jsp template should be available

R4

low

no magic strings. All parameters like output directory should be configurable

1.3. Quality Goals

Type Description

performance

since tests should run fast, performance is important. Report should be generated in less than 2 seconds

modifiability

every Grails developer should be able to modify the plugin in order to fit his needs

availability

plugin should be available through central plugin repository and as download

security

once installed, the plugin must not connect to the internet

testability

it would be nice if there where automated tests

usability

install and run with reasonable default configs. See [R2]

1.4. Stakeholders

These are the stakeholders I (Ralf D. Müller) had in mind when I started to code this plugin and I am open for discussions :-)

Role/Name Goal/Boundaries Expected Participation and Contribution

Developer

wants to proof to BA that her software is covered with automated tests

Need direct feedback from other developers through issue tracking or direct mail

Business Analyst (BA)

wants to specify tests in natural language and wants to be able to read test reports

Hope for indirect feedback through developers

Contractor

wants to proof to client that his software is thourougly tested and uses screenshot to have a proof for the look and feel at a certain point in time

Hope for indirect feedback through developers

Designer

wants to see that his design is implemented the right way and may want to compare screens

Hope for indirect feedback through developers

1.4.1. Personas

The following section will describe some fictional stakeholders. The intention is to add some more to the stakeholder description above and to allow you to easily switch into the role of a stakeholder.

Melanie Schmitt

Melanie works as a freelancer in the role of a developer. She is used to write automated tests for her code. She would now like to use the test results as some kind of acceptance report. When she delivers a release, she wants to have a report ready which can be signed of by the customer as soon as he accepts the delivered features.
She also made the experience that sometimes customers report a bug where it is not clear if it is really a bug or works as designed. Through a history of proper test reports with screenshots, she hopes to be able to easily prove that it’s not a bug but a change the customer has to pay for (because it was this way all the time).
Those two features will help Melanie to be recognized as professional developer and optimize her effort/Value ratio.

Peter Miller

Peter is a business analyst who works together with a team of developers on larger projects. He would like to specify automated tests but has not enough technical background to write test scripts. He also does not like to have a development environment up and running in order to run the tests by himself but he would like to have a good looking report which helps him to check if the tests have been executed in the intended way.
The goal for Peter is to write tests in natural language and hand them over to the developers. They will then turn them into executable tests and generate tests reports for each new release. The reports are handed back to Peter for an acceptance sign-off.

Henry Ford

Henry is a developer who works together with Peter. Henry needs a way to add readable code to the natural language test description (received from Peter) and combine both in a report. When Henry runs his tests, he can watch his web browser execute the tests and he sees if something looks wrong. But he also would like to have those screens as screenshots included in his report. In case that Peter complains about a specific test in the generated report, the report should include enough detail for Henry to figure out what code has been executed for the test in order to easily see what’s going on.

Steve Bauer

Steve is a designer who needs to make sure that his design guidelines are followed by the developers. He doesn’t care about the functionality of the application, only the screens. So he needs a report where he can flip through all screens and see if he can find a design flaw.

2. Architecture Constraints

Since this plugin should be available to a wide range of developers, the plugin should be compatible with a wide range of Grails versions (min. 2.2.4). The currently not released Grails 3.0 is out of scope but will be in scope as soon as it is released.

Since the plugin should be based on stable and released libraries, 0.7 is the targeted Version of Spock.

The plugin has to work on windows and *nix and it should work with all webdrivers which take screenshots, but it will only be tested against phantomJs.

Since maps are easy to handle in groovy and grails, the generated data model should not be a complex java object, but a simple map.

Whenever the map has to be serialized, JSON should be used as format with pretty print turned on. This ensures that a developer is able to analyze the raw data.

3. System Scope and Context

The main scope of the project is to extract a model from the data generated by geb through the execution of grails run-app functional. In order to visualize the model, the plugin will also generate some HTML to browse through the data.

Currently, data generated by spock is out of scope, but it is planned to take this in scope for the next iteration.

3.1. External Interfaces

Since the plugin should be easy to handle, it should use some automated way to gather the data.

In order to do this, the following interfaces to Grails and Geb have been identified:

Event System

The plugin will hook into the reporting events in order to setup the collection of data (eventTestPhasesStart) and to generate the reports (eventTestProduceReports)

Geb ReportingListener

The ReportingListener has been identified as the right interface to Geb. It will notify the plugin every time the report command has been executed and will provide the needed data. Currently, no automated way has been identified to add the plugin’s own ReportingListener to the gebConfig in an automated way, so the developer has to add it by herself.

4. Solution Strategy

The plugin uses the ReportingListener provided by geb to collect the needed data. Since geb can only provide an onReport-method, it is hard to collect the data in memory (there is no event fired by Geb which tells the plugin that all reports have been collected and data can be persisted). So the solution is to write the data from these report snapshots to a file as JSON strings.

Later, when Grails notifies the plugin that all tests have been executed, the plugin will read those report snapshots and remove redundant information by generating a better structured map which is also stored as JSON in another report file (in case someone else wants to base some functionality on this data).

In the next step, the plugin takes the created data structure and feeds it as model into an HTML-Report-Template. Currently, this "template" is implemented as xml.MarkupBuilder but it will be soon replaced by a jsp like template which can be easily provided by the project the plugin is used in.

5. Design Decisions

5.1. Data Collection

5.1.1. Problem

In order to create a report, we have to collect the data we want to report on.

5.1.2. Alternatives and Solution/Decision

The first version of the FilmStrip-Plugin crawled through the generated screenshots and tried to extract all data from the image name and its path.

The current version uses the Geb ReportingListener to collect the data. The ReportListener gets some more information than the one available in the file system like the URL and name of page object.

Since there is only an onReport event and no onReportFinished and because of the data provided, the data collection is a two step process:

  1. extract the needed data from the onReport event

  2. transform the result into a better structured map without redundancy

The data from step one is collected in a JSON formatted report file and read in step two from this file.

The number together with the label of the test and report is made available as one String, joined together with dashed (-). So if the labels contain a dash themself, the labels will not be extracted the right way. To solve this, the geb.report.ReporterSupport got monkey patches through meta-programming.

5.2. Monkey Patching

5.2.1. Problem

As described in the previous design decision, the plugin uses a monkey patch for geb.report.ReporterSupport

Monkey patches are considered as bad practice.

5.2.2. Alternatives and Solution/Decision

One alternative to the monkey patch is to derive a new Spec-class from GebReportingSpec (proposed by Marcin Erdmann - thanx for the idea!). This would be a clean solution with the drawback that all specs have to be rewritten to use this Spec-class.

Currently, the plugin will still use the monkey patch until is causes problems ;-)

5.3. Fix for Packages

5.3.1. Problem

The report should link to the HTML test reports in order to show some more details (link behind the spec name). But the reports are prefixed with a number. If no packages are used to group the tests, the number is in line with the order of test execution, but if packages are used, I couldn’t find out how they are numbered.

5.3.2. Alternatives and Solution/Decision

One way to solve this could have been to dive deep down into the report generation and find out how the files are numbered. The effort would have been to high at the moment.

The solution which is now implemented is to crawl through the existing html report files, find the one with the proper name in the proper folder and extract the prefixed number. Since the reports are not available when eventTestProduceReports is fired, the code was moved to the eventTestPhasesEnd.

6. Glossary

This is mainly a glossary for Spock terms and how I do understand their meaning. Please correct it if I am wrong - RDM

Term Synonym Description

Spec

Specification

A test specification which groups some tests together

Feature

Test

A spec specifies features which have to be tested. Mostly exchangeable with "test", but some features need more than one test…

Feature Run

Iteration

When you define a data driven Spock test, each execution is called a feature run

Report

Snapshot

The geb command to save the current browser page as image and HTML source is called report and hence the created snapshot is often called a report

Report Group

a report group is a Geb concept. Spock automatically sets the report group to the name of the current Feature/Test and Geb will store all reports in a folder corresponding to this name.

Appendix A: References

Similiar Projects

In this section, references to other projects like this are collected. So if this project does not exactly fit your need, maybe one of the other does.

canoo WebTest

Canoo WebTest is a great tool for functional tests and it provides the best reports I know. Screenshots are taken automatically, each time a button or link is clicked. Those screenshots are available as film strip like the one in this plugin.

Unfortunately, it seems to be a bit outdated. It does not work with Spock since it is a test framework by its own and the Gails Plugin seems not to work with current versions of Grails

Appendix B: Example Reports

Currently there is only one example which is based on the Pet Clinic sample application.

Screenshot

the modified grails pet-clinic project to generate above report can be found at

Appendix C: Contributors

  • Marcin Erdmann - thoughts about monkey patch

  • Burth Beckwith - cleanup of plugin code

  • Peter Ledbrook - tipps on how to fix the plugin for grails 2.4.4

  • Tobias Kraft - support for Grails 2.1.*

Contributing

Contributions are welcome. The quality and form does not matter. If you don’t have much time, just drop me a note (ralf.d.mueller at gmail.com) or add an issue.

If you have more time and want to contribute to the code, just fork the project and create a pull request when ever you contribution is ready.

The easiest way to work on the project is to fork both, this repository and the film strip fork of the petclinic project. Place them on the same level and the petclinic project will already be configured to reference the film strip plugin locally - this avoids having to build the plugin whenever you modify it.

Currently, there are no automated tests. So you are also not required to create tests :-).

Appendix D: Changelog

Version Description

0.1.1

Initial Version

0.1.2

-

0.1.3

support for packages

0.1.4

support for Grails 2.1

Open Issues

In order to work offline on this project, all open issues are exported to this document whenever it is updated. But if you want to open a new issue, just add it to the github issues located at https://github.com/rdmueller/grails-filmStrip/issues . The ultimate goal is to sync these offline issues with the github ones.

ID State Description

FS-21

open

upload the petclinic report to github.io and add it to the examples section

FS-19

open

create the same functionality as gradle project, later plugin

FS-18

open

compare created arc42 documentation with docs of the 10 best plugins

FS-16

open

It would be nice to find an automated way to add the FileStripReportingListener to the gebConfig

FS-15

open

Want to have a button with which I can flip through all screens

FS-13

open

End-Reports are currently not shown. The intended behaviour is that they are only not shown when the content is the same as of the report before. This can be achieved by comparing the reported HTML

FS-10

open

add some profile "fotos" to each persona. There is this cool avatar genrator which might be used.

FS-9

open

Create a Demo-Report

FS-8

open

Report is also generated if only unit or integration tests are executed

FS-7

open

location of the report folder

FS-6

open

TESTS-TestSuites.xml