WebHook API

This API allows you to auto-import data into your application whenever fresh ones become available.

Registering a webhook

To register a webhook, you will need to provide a URL that accepts HTTP POST requests. You can provide it when you are creating or updating a data source.

The Webhook Request

A HTTP POST request is sent to the registered URL each time fresh data becomes available. This is how the body of the request looks like for a data source 

POST HTTP/1.1
Content-Type: application/json

{
  "data_source_name":       "Hacker News - News Headlines",
  "data_source_unique_id":  "n3901_f3e4cbce5dccaf886b0b6f02b4d24509eses",
  "event_name":             "complete",
  "batch_time_unix":         1526515200,
  "batch_time_iso":         "1969-12-31T18:46:40-08:00",
  "data_url_json":          "http://cache.getdata.io/n3901_f3e4cbce5dccaf886b0b6f02b4d24509eses/1526515200_page_1.json",
  "data_url_csv":           "http://cache.getdata.io/n3901_f3e4cbce5dccaf886b0b6f02b4d24509eses/1526515200_all.csv"
}

Attributes in the payload
data_source_name String Your data source's name
data_source_unique_id String Your data source unique identification key
event_name String The event that triggered this webhook. Types include:
  • complete - when crawling was successfully completed
  • aborted - when crawling was aborted by the user
batch_time_unix Integer The Unix time stamp when this batch of data was gathered
batch_time_iso String The ISO format time stamp when this batch of data was gathered
data_url_json String The URL to the cache to download the first batch of results in JSON format
data_url_csv String The URL to the cache to download the entire batch of results in CSV format

Testing Your Webhook

You can test your integration with your registered webhook when a batch of data is available. To do so, simply click on test button of your data source

Next Steps

To find out more about how GetData.IO works