Rigor’s Changelog shows logged changes that we can reference for reporting on changes to performance before and after events. We can use custom date range filters to view the Performance History of checks before and after changes.
Here are some things to consider when planning to analyze performance changes before and after a deploy or change to your site:
- Know exactly when to expect a major change to a site? Schedule a blackout period. A blackout period helps keep checks from running and triggering alerts while changes are being pushed to a live site.
- Changing something on the fly? Use a Control Group and the Rigor API to make an annotation on any checks that will be affected by the change.
- Make any needed changes to Rigor check steps or configured settings for checks monitoring the site where changes will take place.
- Decide how you will measure success.
Reporting On Change In Average Response Time
On 1 February 2016, we scheduled a change to move the Rigor Knowledge Base from our Help Desk’s CMS to a site that we manage ourselves and host on Github pages.
We had a pre-existing Rigor Real Browser check configured to monitor the performance of visiting help.rigor.com and clicking on the first article link named Rigor KB (homepage > article). The check did not run, capture data, or alert while we were making changes over a scheduled period of time.
We can use the custom date range view to see the performance of the check for a week before and after the change. The blue vertical line represents changes made to the check. We updated the check steps to use selectors that matched the HTML and CSS of our new site.
After we made changes to the site and the Rigor check configuration we resumed the check and began collecting data again.
Comparing one week before and one week after the change we can see that our changes reduced the average response time of the user flow from 3 seconds to about 750 ms.
The average response time of the check over a week was 3 seconds with the slowest response time equalling 30 seconds.
After changing to the new site the average response time of the check over a week was 752 ms (2.2 seconds faster) with the slowest response time equalling 3 seconds (27 seconds faster).