WebHook API

This API is exposed to allow you the ability to auto-import data into your application whenever a new batch of data becomes available.


Registering a webhook

When you are creating a new data source, you can indicate a Webhook URL. The URL you provide will need to accept HTTP POST requests

Webhook url


The Webhook Request

A HTTP POST request is made to your application each time a new batch of data collection gets completed. Below is an example of a sample payload request for the Hacknews data source

POST HTTP/1.1
Content-Type: application/json

{
  "krake_name":     "Hacker News - News Headlines",
  "krake_handle":   "n3901_f3e4cbce5dccaf886b0b6f02b4d24509eses",
  "event_name":     "complete",
  "batch_time_sec": 1526515200,
  "batch_time_str": "2018-05-17 00:00:00",
  "data_url_json": "http://cache.getdata.io/n3901_f3e4cbce5dccaf886b0b6f02b4d24509eses/1526515200_page_1.json",
  "data_url_csv": "http://cache.getdata.io/n3901_f3e4cbce5dccaf886b0b6f02b4d24509eses/1526515200_all.csv"
}

Attributes in the payload
krake_name String The name of your data source
krake_handle String The handle of your data source
event_name String The event that triggered off this webhook. Types include:
  • complete - when crawling was successfully completed
  • aborted - when crawling was aborted by the user
batch_time_sec Integer The Unix time stamp for when this batch of crawling started
batch_time_str String The human readable time stamp for when this batch of crawling started
data_url_json String The URL to the cache to download the first batch of results in JSON format
data_url_csv String The URL to the cache to download the entire batch of results in CSV format

Testing Your Webhook

Once you have indicated your Webhook URL and a batch of data is available for download, you can do a trial run to test the webhook integration.

To do so, you will need to click on the corresponding test button of your data source

Webhook test


Next Steps

To find out more about how GetData.IO works