Jenkins is a popular and extensible open-source platform for managing continuous integration builds. You can use Rigor’s Optimization plugin for Jenkins to test your website performance as a part of your Jenkins build process.
This article assumes you already have Jenkins installed and you’re familiar with how to manage Jenkins freestyle projects. You can learn more about getting started with Jenkins on the Jenkins Website.
How to Set Up the Rigor Optimization Plugin for Jenkins
The Rigor Optimization plugin is freely available for download and can be installed via the Jenkins plugin manager. From your Jenkins installation homepage, select Manage Jenkins and then Manage Plugins. Then select the Available tab and search for Rigor Optimization. Once you locate the plugin, install through the interface.
For more details about this plugin, check out the plugin wiki on the Jenkins site.
Add a New Build Step
Once the plugin is installed, select your Jenkins freestyle project from the root page and click Configure. Scroll down to the Build section and click Add build step. Find the option labeled Test website performance using Rigor Optimization.
Select this and you will see a custom build step to use Rigor Optimization.
Configure Your API Key
You will need to grant API access to Jenkins to call into the Rigor Optimization API to initiate performance snapshots as part of the build. To do so, first, locate your API key in your Rigor Optimization settings.
Note the API key must belong to a user with *User* level permissions or higher. (e.g. not a read-only user).
In Jenkins click Add next to the API key setting of the build step, then select Rigor API Key under the Kind dropdown.
Enter your credentials and then select them on the return screen.
Configure the Performance Tests
When this build step executes, Jenkins will call into the Rigor Optimization API to initiate new performance snapshots for one or more performance tests. So the next step of our set-up is to identify which existing performance tests we want to execute.
To locate these, navigate to the View Test Results page of the Rigor Optimization application, and locate one or more tests to run with each build. To find the Performance Test ID, hover over the i icon and locate the Performance Test ID in the popup.
Add this to the Performance Test IDs field in your Jenkins configuration and repeat for any other tests you would like to launch. You can separate multiple test IDs with commas.
Then click the Test Connection button to confirm your API key and Test IDs are valid.
If you see errors, double-check that your API key was pasted correctly, the account has User level or higher permissions, and the listed Performance Test IDs were properly listed.
You are now ready to go! When you save the build step, the next time your build runs it will launch new snapshots for the configured tests as part of the build. Over time you will see historical records of how your website performance changed after each change was committed.
How to Customize Build Failures
You may want to optionally fail the build if certain performance problems are detected. This next section will help you further configure different parameters for failing the build.
First, check the option Fail build based on test results and a number of new options will be displayed.
If one or more of these settings are enabled, the build will wait until all initiated performance snapshots complete, then test the results of each against these parameters. If any one of these parameters fails for any one of the configured tests, the build will be marked as failed.
Let’s walk through each option.
Fail if below this performance score
This refers to the overall performance score assigned to each snapshot when it completes. This is perhaps the easiest metric to track. Assign a minimum value you would like to see (ideally based on your existing historical results), and if any snapshots from this build score under that number, the build will be marked as failed.
Fail if above this number of critical defects
Like the name implies, if any snapshot has more than this number of critical defects reported, the build will fail. What’s interesting here is you can configure which defects get marked as Critical using the Rigor Optimization Customizations feature.
For example, you can assign specific thresholds to specific defect types, and re-assign the priority of those defects to Critical when those defects flag. This could be useful for alerting you when images above a certain size are introduced, or lossless optimization savings are beyond a certain value, for example.
Fail if defect IDs found
This field takes a comma separated list of defect IDs that, if found, will fail the build. For example, perhaps you want to fail the build if any compression defects are introduced.
To locate the defect IDs, visit any defect page in the Rigor Optimization application (or in the Rigor Optimization Performance Knowledge base and look for the ID label towards the top of the defect description:
Timeout waiting for test completion
If one or more metrics are configured in the Fail build based on test results section, the build will wait for all snapshots to complete before proceeding to the next build step. Depending on the tests configured, this could delay your build from 30 seconds to several minutes. This timeout value allows you to set an upper limit on how long to wait. If the timeout occurs waiting for completion of the snapshots, the build step terminates and proceeds to the next step. This may or may not fail the Jenkins build based on the next setting.
Fail if tests encounter errors
This controls how the Jenkins build will be marked if any Rigor performance tests encountered errors (including a timeout, per the above section). If true, the build will always be marked as failed if errors are encountered. This is appropriate if you want immediate visibility into potential configuration errors. If false, the build will not be marked as failed if test errors occur. This option is less disruptive to your build process but may mask configuration errors and hence puts the onus on you to monitor your logs from time to time.
Linking Builds to Snapshots
Whenever a Rigor performance test is run from a Jenkins build, that test is tagged inside Rigor Optimization with the name of the Jenkins build project that executed the test. For example, the tests below were run from the ZoompfCore Jenkins project.
In addition, each snapshot executed is also tagged with the project name, build number and build pass/fail details. For example:
this association allows you to quickly spot which tests and snapshots belong to which Jenkins builds, and quickly filter your results in Rigor Optimization for any errors that may have occurred.
Debugging Build Failures
The Rigor Optimization plugin logs verbose details of all performance tests run in the console output of the Jenkins build. If a build failure occurs (or even if it succeeds), simply view the Console Output for that build number and scroll down to the Rigor Optimization section. You should see something like this:
If failures occur, verbose details of what failed and why can be found in the log.