How do I send notifications to Splunk On-Call?

Splunk Synthetic Monitoring verifies applications are performing well and alerts you to user experience problems before they impact your customers.  Integrating Splunk Synthetic Monitoring with Splunk On-Call allows you to send alert notifications into the Splunk On-Call timeline.

You must be an administrator in Splunk Synthetic Monitoring and a Global or Alert Admin in Splunk On-Call to establish this integration.

Enable the integration in Splunk On-Call

1. In Splunk On-Call, select Integrations > Splunk Synthetic Monitoring
2. If the integration has not yet been enabled, click the “Enable Integration” button to generate your endpoint URL.  Copy this URL to your clipboard.

Create a new Alert Webhook in Splunk Synthetic Monitoring (Rigor)

1. In Splunk Synthetic Monitoring, click on the three-dot Admin Tools menu and select Alert Webhooks.
   
2. Click + New to create a new webhook and select Splunk On-Call as the type.
   
3. Give the webhook a name and specify your trigger or triggers.  In the “send request to” box for each trigger, replace the default  https://alert.victorops.com/integrations/<<YOUR_VICTOROPS_REST_ENDPOINT>>  with the URL you’d previously copied.
   
4. On each trigger, also replace the trailing $routing_key value with the Splunk On-Call routing key you’d like this webhook to hit. For more information on routing keys, see this article.
5. (optional) Click JSON to open the JSON request body for the API request that will be sent upon the trigger event. Edit the values in the payload as needed and then click Save.
6. Click the Test button to test the integration.  This will send an alert with your specified payloads into the Splunk On-Call timeline.
7. Click Create to save your Alert Webhook. 

Add your webhook to a check 

You need to add your webhook to each check you want to send notifications from. To add your webhook to a check, do the following:

1. Navigate to a check in Splunk Synthetic Monitoring, click the gear icon, and select Edit from the dropdown.
2. Click the Notifications tab.
3. Under Notify, select your desired notification trigger. 
4. Under Recipients, select your new webhook from the dropdown. 
5. Adjust the additional settings to your preferences. 
6. Click Save.

Splunk Synthetic Monitoring will now send a POST request to the Splunk On-Call endpoint whenever the trigger conditions for each check to which you added the webhook are met.