|
© This document uses material from the arc42 architecture
template, freely available at http://github.com/arc42. |
|
Note
|
RDM: I used the arc42 template for the documentation of this plugin since it just helps a lot in writing documentation. It is not a perfect example for arc42, since the plugin is just a small project and does not need much architecture. See http://www.arc42.de for better examples :-) |
1. Introduction and Goals
Geb — together with Spock — is a great combination if you need to create functional tests for your Grails application.
But the output is basically just the information that the tests succeeded — at least when there was no failure :-)
Out of the box, it does not report on what has been tested. If you want to get details about the tests, you are
expected to read the sources.
A not so well known feature of Geb is the report feature:
whenever you use the command report "some text", geb tries to create a screenshot and save the current DOM as HTML.
This plugin crawls through the generated screenshots and creates an HTML report which displays preview images as a film strip at the top of the report-page and the test-result on the main part of the window. When you click on an image, it will be displayed in full size, if you click on the name of a report image, the corresponding HTML file will be displayed. Without the proper path to referenced files, this view will look broken, but it is still useful for debugging. When you click on the small star in front of the name, the link will take you to the recorded original URL of this test — so if your project is running, you will be taken to the right page on your web server and you can manually verify what happened.
The aim of this plugin is not only to help a tester in developing Geb/Spock tests and see what went wrong, but also to create a report for your customer. With all those screenshots you can easily get an graphical overview of the application and see if all pages display correctly.
1.1. Quickstart
There is not much to do in order to use the plugin. Just configure it in your BuildConfig.groovy
test ':film-strip:0.1.4'
and add the folowing line to your gebConfig.groovy:
reportingListener = new grails.plugin.filmstrip.FilmStripReportingListener()
If your geb reportDir is the standard one, the report will be generated after running grails test-app.
You will find the generated report in target/test-reports/geb
There is a fork of the Gails-Pet-Clinic sample application on https://github.com/rdmueller/grails-petclinic which contains two branches (film-strip-plugin-2.2 and film-strip-plugin-2.4) with Spock/Geb tests and the film strip plugin installed. See the readme on how to execute the tests.
1.2. Requirements Overview
| ID | Priority | Description |
|---|---|---|
high |
the system has to produce an easy to navigate visual report containing all screenshots reported by geb |
|
medium |
the system has to be easy to use. Best would be install and run with some reasonable default configs |
|
medium |
the system should be configurable in terms of html output and css style. Some kind of jsp template should be available |
|
low |
no magic strings. All parameters like output directory should be configurable |
1.3. Quality Goals
| Type | Description |
|---|---|
performance |
since tests should run fast, performance is important. Report should be generated in less than 2 seconds |
modifiability |
every Grails developer should be able to modify the plugin in order to fit his needs |
availability |
plugin should be available through central plugin repository and as download |
security |
once installed, the plugin must not connect to the internet |
testability |
it would be nice if there where automated tests |
usability |
install and run with reasonable default configs. See [R2] |
1.4. Stakeholders
These are the stakeholders I (Ralf D. Müller) had in mind when I started to code this plugin and I am open for discussions :-)
| Role/Name | Goal/Boundaries | Expected Participation and Contribution |
|---|---|---|
Developer |
wants to proof to BA that her software is covered with automated tests |
Need direct feedback from other developers through issue tracking or direct mail |
Business Analyst (BA) |
wants to specify tests in natural language and wants to be able to read test reports |
Hope for indirect feedback through developers |
Contractor |
wants to proof to client that his software is thourougly tested and uses screenshot to have a proof for the look and feel at a certain point in time |
Hope for indirect feedback through developers |
Designer |
wants to see that his design is implemented the right way and may want to compare screens |
Hope for indirect feedback through developers |
1.4.1. Personas
The following section will describe some fictional stakeholders. The intention is to add some more to the stakeholder description above and to allow you to easily switch into the role of a stakeholder.
- Melanie Schmitt
-
Melanie works as a freelancer in the role of a developer. She is used to write automated tests for her code. She would now like to use the test results as some kind of acceptance report. When she delivers a release, she wants to have a report ready which can be signed of by the customer as soon as he accepts the delivered features.
She also made the experience that sometimes customers report a bug where it is not clear if it is really a bug or works as designed. Through a history of proper test reports with screenshots, she hopes to be able to easily prove that it’s not a bug but a change the customer has to pay for (because it was this way all the time).
Those two features will help Melanie to be recognized as professional developer and optimize her effort/Value ratio. - Peter Miller
-
Peter is a business analyst who works together with a team of developers on larger projects. He would like to specify automated tests but has not enough technical background to write test scripts. He also does not like to have a development environment up and running in order to run the tests by himself but he would like to have a good looking report which helps him to check if the tests have been executed in the intended way.
The goal for Peter is to write tests in natural language and hand them over to the developers. They will then turn them into executable tests and generate tests reports for each new release. The reports are handed back to Peter for an acceptance sign-off. - Henry Ford
-
Henry is a developer who works together with Peter. Henry needs a way to add readable code to the natural language test description (received from Peter) and combine both in a report. When Henry runs his tests, he can watch his web browser execute the tests and he sees if something looks wrong. But he also would like to have those screens as screenshots included in his report. In case that Peter complains about a specific test in the generated report, the report should include enough detail for Henry to figure out what code has been executed for the test in order to easily see what’s going on.
- Steve Bauer
-
Steve is a designer who needs to make sure that his design guidelines are followed by the developers. He doesn’t care about the functionality of the application, only the screens. So he needs a report where he can flip through all screens and see if he can find a design flaw.
2. Architecture Constraints
Since this plugin should be available to a wide range of developers, the plugin should be compatible with a wide range of Grails versions (min. 2.2.4). The currently not released Grails 3.0 is out of scope but will be in scope as soon as it is released.
Since the plugin should be based on stable and released libraries, 0.7 is the targeted Version of Spock.
The plugin has to work on windows and *nix and it should work with all webdrivers which take screenshots, but it will only be tested against phantomJs.
Since maps are easy to handle in groovy and grails, the generated data model should not be a complex java object, but a simple map.
Whenever the map has to be serialized, JSON should be used as format with pretty print turned on. This ensures that a developer is able to analyze the raw data.
3. System Scope and Context
The main scope of the project is to extract a model from the data generated by geb
through the execution of grails run-app functional. In order to visualize the
model, the plugin will also generate some HTML to browse through the data.
Currently, data generated by spock is out of scope, but it is planned to take this in scope for the next iteration.
3.1. External Interfaces
Since the plugin should be easy to handle, it should use some automated way to gather the data.
In order to do this, the following interfaces to Grails and Geb have been identified:
- Event System
-
The plugin will hook into the reporting events in order to setup the collection of data (
eventTestPhasesStart) and to generate the reports (eventTestProduceReports) - Geb
ReportingListener -
The ReportingListener has been identified as the right interface to Geb. It will notify the plugin every time the
reportcommand has been executed and will provide the needed data. Currently, no automated way has been identified to add the plugin’s ownReportingListenerto thegebConfigin an automated way, so the developer has to add it by herself.
4. Solution Strategy
The plugin uses the ReportingListener provided by geb to collect the needed data.
Since geb can only provide an onReport-method, it is hard to collect the data
in memory (there is no event fired by Geb which tells the plugin that all reports
have been collected and data can be persisted). So the solution is to write the
data from these report snapshots to a file as JSON strings.
Later, when Grails notifies the plugin that all tests have been executed, the plugin will read those report snapshots and remove redundant information by generating a better structured map which is also stored as JSON in another report file (in case someone else wants to base some functionality on this data).
In the next step, the plugin takes the created data structure and feeds it as model
into an HTML-Report-Template. Currently, this "template" is implemented as
xml.MarkupBuilder but it will be soon replaced by a jsp like template which can
be easily provided by the project the plugin is used in.
5. Design Decisions
5.1. Data Collection
5.1.1. Problem
In order to create a report, we have to collect the data we want to report on.
5.1.2. Alternatives and Solution/Decision
The first version of the FilmStrip-Plugin crawled through the generated screenshots and tried to extract all data from the image name and its path.
The current version uses the Geb ReportingListener to collect the data. The ReportListener gets some more information than the one available in the file system like the URL and name of page object.
Since there is only an onReport event and no onReportFinished and because of the data provided, the data collection is a two step process:
-
extract the needed data from the
onReportevent -
transform the result into a better structured map without redundancy
The data from step one is collected in a JSON formatted report file and read in step two from this file.
The number together with the label of the test and report is made available as one String, joined together with dashed (-). So if the labels contain a dash themself, the labels will not be extracted the right way. To solve this, the geb.report.ReporterSupport got monkey patches through meta-programming.
5.2. Monkey Patching
5.2.1. Problem
As described in the previous design decision, the plugin uses a monkey patch for geb.report.ReporterSupport
Monkey patches are considered as bad practice.
5.2.2. Alternatives and Solution/Decision
One alternative to the monkey patch is to derive a new Spec-class from GebReportingSpec (proposed by Marcin Erdmann - thanx for the idea!).
This would be a clean solution with the drawback that all specs have to be rewritten to use this Spec-class.
Currently, the plugin will still use the monkey patch until is causes problems ;-)
5.3. Fix for Packages
5.3.1. Problem
The report should link to the HTML test reports in order to show some more details (link behind the spec name). But the reports are prefixed with a number. If no packages are used to group the tests, the number is in line with the order of test execution, but if packages are used, I couldn’t find out how they are numbered.
5.3.2. Alternatives and Solution/Decision
One way to solve this could have been to dive deep down into the report generation and find out how the files are numbered. The effort would have been to high at the moment.
The solution which is now implemented is to crawl through the existing html report files,
find the one with the proper name in the proper folder and extract the prefixed number.
Since the reports are not available when eventTestProduceReports is fired, the code was
moved to the eventTestPhasesEnd.
6. Glossary
This is mainly a glossary for Spock terms and how I do understand their meaning. Please correct it if I am wrong - RDM
| Term | Synonym | Description |
|---|---|---|
Spec |
Specification |
A test specification which groups some tests together |
Feature |
Test |
A spec specifies features which have to be tested. Mostly exchangeable with "test", but some features need more than one test… |
Feature Run |
Iteration |
When you define a data driven Spock test, each execution is called a feature run |
Report |
Snapshot |
The geb command to save the current browser page as
image and HTML source is called |
Report Group |
a report group is a Geb concept. Spock automatically sets the report group to the name of the current Feature/Test and Geb will store all reports in a folder corresponding to this name. |
Appendix A: References
- Geb and the Book of Geb
-
http://www.gebish.org/ http://www.gebish.org/manual/current/
- Geb Plugin
- Spock
- Spock Plugin
- Pet Clinic Sample Application
- GhostDriver
- PhantomJS
- arc42
-
http://arc42.org (german) http://arc42.de/en (english)
Similiar Projects
In this section, references to other projects like this are collected. So if this project does not exactly fit your need, maybe one of the other does.
canoo WebTest
- Grails Plugin
- Website
Canoo WebTest is a great tool for functional tests and it provides the best reports I know. Screenshots are taken automatically, each time a button or link is clicked. Those screenshots are available as film strip like the one in this plugin.
Unfortunately, it seems to be a bit outdated. It does not work with Spock since it is a test framework by its own and the Gails Plugin seems not to work with current versions of Grails
Appendix B: Example Reports
Currently there is only one example which is based on the Pet Clinic sample application.
the modified grails pet-clinic project to generate above report can be found at
Appendix C: Contributors
-
Marcin Erdmann - thoughts about monkey patch
-
Burth Beckwith - cleanup of plugin code
-
Peter Ledbrook - tipps on how to fix the plugin for grails 2.4.4
-
Tobias Kraft - support for Grails 2.1.*
Contributing
Contributions are welcome. The quality and form does not matter. If you don’t have much time, just drop me a note (ralf.d.mueller at gmail.com) or add an issue.
If you have more time and want to contribute to the code, just fork the project and create a pull request when ever you contribution is ready.
The easiest way to work on the project is to fork both, this repository and the film strip fork of the petclinic project. Place them on the same level and the petclinic project will already be configured to reference the film strip plugin locally - this avoids having to build the plugin whenever you modify it.
Currently, there are no automated tests. So you are also not required to create tests :-).
Appendix D: Changelog
| Version | Description |
|---|---|
0.1.1 |
Initial Version |
0.1.2 |
- |
0.1.3 |
support for packages |
0.1.4 |
support for Grails 2.1 |
Open Issues
In order to work offline on this project, all open issues are exported to this document whenever it is updated. But if you want to open a new issue, just add it to the github issues located at https://github.com/rdmueller/grails-filmStrip/issues . The ultimate goal is to sync these offline issues with the github ones.
| ID | State | Description |
|---|---|---|
open |
upload the petclinic report to github.io and add it to the examples section |
|
open |
create the same functionality as gradle project, later plugin |
|
open |
compare created arc42 documentation with docs of the 10 best plugins |
|
open |
It would be nice to find an automated way to add the FileStripReportingListener to the gebConfig |
|
open |
Want to have a button with which I can flip through all screens |
|
open |
End-Reports are currently not shown. The intended behaviour is that they are only not shown when the content is the same as of the report before. This can be achieved by comparing the reported HTML |
|
open |
add some profile "fotos" to each persona. There is this cool avatar genrator which might be used. |
|
open |
Create a Demo-Report |
|
open |
Report is also generated if only unit or integration tests are executed |
|
open |
location of the report folder |
|
open |
TESTS-TestSuites.xml |