Articles in this section

How Do I Configure Alert Webhooks?

Avatar
Splunk Synthetics
  • Updated
Follow

We can use Alert Webhooks to notify an HTTP endpoint when a Real Browser check or Uptime check fails. This endpoint could be a third-party alerting service, a company chat application, or even a custom API. To use Alert Webhooks, we need to:

  1. Create a new Alert Webhook
  2. Edit a check to notify the webhook

We can select from the following pre-configured services:
cna8fSA7_400x400.jpg Splunk On-Call

datadoglogo.pngDatadog

hipchat.png HipChat

opsgenie.png OpsGenie

pagerduty.png PagerDuty

slack.png Slack

servicenow.png ServiceNow

webhook.png Configure a webhook to send to a custom endpoint

Send alerts to Splunk On-Call

Splunk Synthetic Monitoring verifies applications are performing well and alerts you to user experience problems before they impact your customers.  Integrating Splunk Synthetic Monitoring with Splunk On-Call allows you to send alert notifications into the Splunk On-Call timeline.

You must be an administrator in Splunk Synthetic Monitoring and a Global or Alert Admin in Splunk On-Call to establish this integration.


In Splunk On-Call

In Splunk On-Call, select Integrations >> Splunk Synthetic Monitoring

If the integration has not yet been enabled, click the “Enable Integration” button to generate your endpoint URL.  Copy this URL to your clipboard.

In Splunk Synthetic Monitoring (Rigor)

From the Rigor Monitoring web portal, select the 3 dot menu next to your profile icon in the upper-righthand corner and select Alert Webhooks.

mceclip0.png

Click the + New button to create a new webhook and select Splunk On-Call as the type.

mceclip1.png
Give the webhook a name and specify your trigger or triggers.  In the “send request to” box for each trigger, replace the default

https://alert.victorops.com/integrations/<<YOUR_VICTOROPS_REST_ENDPOINT

with the URL you’d previously copied.

On each trigger, also replace the trailing $routing_key value with the Splunk On-Call routing key you’d like this webhook to hit.  For more information on routing keys, see this article.

You’re welcome to leave the payloads at their default values, or if you find it more appropriate, replace the default variables with ones more catered towards your use case.

Once finished, click the Test button to test the integration out.  This will send an alert with your specified payloads into the Splunk On-Call timeline.

Lastly, hit the Create button at the bottom of the screen to save your Alert Webhook.  Splunk Synthetic Monitoring will now send a notification to Splunk On-Call whenever the trigger conditions are met.

Send alerts to Datadog

    1.  Create a new Alert Webhook under Admin > Alert Webhooks
    2. Select the pre-built webhook for Datadogdatadogscreenshotbigger.png
    3. Select the ‘Triggers’ you want to activate. There are two types: ‘When failed’ and ‘When back online’

      Datadog-Triggers.png

    4. Insert your Datadog API key into the Trigger endpoint URLsDatadog-API-Key-e1519151216847.png
    5. OPTIONAL: If you want, you can customize the JSON data sent in the notification via the ‘Payload’ section. This allows you to change the text and alert format to something that better suits your workflow.Datadog-JSON-Payload.png
    6. Once you have saved and named your webhook, it can be added to any Rigor monitoring check via the check notifications page in the check edit menu. From that point forward, your Datadog event’s stream will receive a POST request with the specified JSON payload any time your check fails. Example failures would look like this:Datadog-events-dashboard-failure.png

Send alerts to HipChat

  1. Log in to HipChat and go to ‘Rooms’

    hipchat_select_room.png

  2. Select the room to receive Rigor alert notifications and click ‘Integrations’ in the left menu

  3. Click Create on the ‘Build Your Own!’ option

    hipchat_create_integration.png

  4. Name the integration and click Create

  5. Copy the integration URL and click Save

    hipchat_copy_url.png

  6. In Rigor, go to Admin > Alert Webhooks > + New and create a HipChat Alert Webhook

  7. Paste the HipChat integration URL copied in step 5 in the ‘Webhook URL’ field for each trigger

    hipchat_paste_url.png

  8. (optional) Customize the JSON data to be sent to HipChat
  9. Save the webhook

  10. Edit a Rigor check and set it to notify the webhook

    hipchat_edit_check.png

With the webhook set-up, Rigor alerts will post notifications to HipChat:

hipchat_notification.png

Send alerts to OpsGenie

  1. Log in to OpsGenie and go to ‘Integrations’

  2. Find the Rigor integration and click Add (or click here)

    opsgenie_add_integration.png

  3. Copy the ‘API Key’ in the integration settings

    opsgenie_api_key.png

  4. In Rigor, go to Admin > Alert Webhooks > + New and create an OpsGenie Alert Webhook

  5. Paste the OpsGenie API key copied in step 3 in the ‘apiKey’ field in the ‘Payload’ section for each trigger

    opsgenie_paste_api_key.png

  6. (optional) Customize the JSON data to be sent to OpsGenie
  7. Save the webhook

  8. Edit a Rigor check and set it to notify the webhook

    opsgenie_edit_check.png

With the webhook set-up, Rigor alerts will trigger alerts in OpsGenie:

opsgenie_opened.png

When the check comes back online, the associated OpsGenie alert will get closed:

opsgenie_closed.png

For more, see the OpsGenie integration documentation.

Send alerts to PagerDuty

  1. Log in to PagerDuty and go to Configuration > Services

    add_pagerduty_service.png

  2. Click Add Services, name the Service, and select ‘Rigor’ as the integration type

    create_pagerduty_service.png

  3. Click Add Service

  4. Copy the ‘Service API Key’ of the new PagerDuty Service

    pagerduty_service_api_key.png

  5. In Rigor, go to Admin > Alert Webhooks > + New and create a PagerDuty Alert Webhook

  6. Paste the PagerDuty Service API Key copied in step 4 in the ‘service_key’ field in the ‘Payload’ section for each trigger

    pagerduty_service_key.gif

  7. (optional) Customize the JSON data to be sent to PagerDuty
  8. Save the webhook

  9. Edit a Rigor check and set it to notify the webhook

    edit_check_notify_pagerduty.png

With the webhook set-up, Rigor alerts will trigger incidents in PagerDuty:

pagerduty_triggered.png

When the check comes back online, the associated PagerDuty incident will get resolved:

pagerduty_resolved.png

Send alerts to Slack*

*You need to have a paid, 'Standard' Slack plan to integrate as a webhook.

  1. Log in to Slack and go to ‘Integrations’

    slack_incoming_webhook.png

  2. Click the ‘Incoming WebHooks’ option (or click here)

  3. Select the Slack channel to receive alerts and click Add Incoming WebHooks Integration

    slack_select_channel.png

  4. Copy the ‘Webhook URL’ for the new Incoming WebHook

    slack_webhook_url.png

  5. In Rigor, go to Admin > Alert Webhooks > +New > [Slack]

  6. Paste the Webhook URL copied in step 4 in the ‘Webhook URL’ field for each trigger

    slack_paste_url.png

  7. (optional) Customize the JSON data to be sent to Slack
  8. Save the webhook

  9. Edit a Rigor check and set it to notify the webhook

    slack_edit_check.png

With the webhook set-up, Rigor alerts will post notifications to Slack:

slack_notification.png

 

 

Send Alerts to ServiceNow

  1. Create a new Alert Webhook under Admin > Alert Webhooks
    Screenshot_2018-10-31_09.30.04.png
  2. Select the icon for ServiceNow
    2.png
  3. Provide the credentials for the ServiceNow user that will be used to create an incident. Click "Generate Header" button to generate the encoded authorization header that will be used with this webhook. Note: Make sure the ServiceNow user has the permissions to create, modify and close incidents.
    3.png
  4. Select the 'Triggers' you want to activate. There are two types: 'When failed' and 'When back online'. 'When failed' will open a new incident in ServiceNow when a check fails and 'When back online' will close that same incident when that check resumes functioning. Note: you should not use the 'When back online' trigger without also using a 'When failed' trigger.
    4.png
  5. Find the base URL of your ServiceNow instance. In the new webhook form, edit the URLs for each trigger's endpoint, replacing "<<YOUR_SERVICENOW_BASE_URL>>" with the base URL for your ServiceNow instance.
    5.png
  6. OPTIONAL: If you want, you can customize the JSON data sent in the notification via the 'Payload' section. This allows you to change the text and alert format to something that better suits your workflow. Note: When customizing a 'When back online' trigger, the state and close_code fields should not be changed.
  7. Once you have saved and named your webhook, it can be added to any Rigor monitoring check via the check notifications page in the check edit menu. From that point forward, new ServiceNow incidents will be created with the specified details any time a check fails. In turn, the incident will be closed automatically when the check is back online.

 

 

Send alerts to a custom endpoint

  1. In Rigor, go to Admin > Alert Webhooks > + New and create a Custom Alert Webhook
  2. (optional) Add Request Headers
  3. In the ‘Webhook URL’ field for each trigger, provide the endpoint to receive Rigor alertsImage_2019-07-26_at_9.38.58_AM_2.png
  4. (optional) Customize the JSON data to be sent in the webhook
  5. Test the webhook (check API endpoint to verify results)
  6. Save the webhook

  7. Edit a Rigor check and set it to notify the webhook

    custom_edit_check.png

With the webhook set-up, Rigor alerts will trigger POST requests to the specified endpoints.

Related articles