Farsight DNSDB API Documentation

Table of Contents

  1. Introduction
  2. Audience
  3. Getting Started
  4. Authentication
  5. Lookup methods
  6. Result formats
  7. Service limits
  8. Response codes
  9. Example scripts
  10. Example results
  11. Changes

Introduction

DNSDB is a database that stores and indexes both the passive DNS data available via Farsight Security's Security Information Exchange as well as the authoritative DNS data that various zone operators make available. DNSDB makes it easy to search for individual DNS RRsets and provides additional metadata for search results such as first seen and last seen timestamps as well as the DNS bailiwick associated with an RRset. DNSDB also has the ability to perform inverse or rdata searches.

This document describes how to make bulk, automated DNSDB queries via the HTTP API.

Audience

This document is intended for programmers who want to write applications that can interact with the DNSDB API using JSON and HTTP.

Getting Started

To request an API key, please see the Farsight Security service application form.

Authentication

The DNSDB API is provided over an encrypted HTTPS transport over the Internet at the following URL:

https://api.dnsdb.info/

Authentication is performed by providing an API key (a long string of hexadecimal digits) in a special HTTP request header, X-API-Key. For example, if your API key is d41d8cd98f00b204e9800998ecf8427e, then the following HTTP header should be added to the HTTP request:

X-API-Key: d41d8cd98f00b204e9800998ecf8427e

If the X-API-Key header is not present, or the provided API key is not valid, a 403 Forbidden response will be returned to the HTTP client.

Lookup methods

All DNSDB lookup requests are rooted in URL paths under the /lookup hierarchy. There are two separate lookup methods, "rrset" and "rdata", which are accessed via the URL paths /lookup/rrset and /lookup/rdata respectively.

There is a built-in limit to the number of results that are returned via these lookup methods. The default limit is set at 10,000. This limit can be raised or lowered by setting the "limit" query parameter. E.g., appending "?limit=20000" to the URL path will set the response limit to 20,000 results.

You may filter results by time using the "time_first_before," "time_first_after," "time_last_before," and "time_last_after" query parameters. These parameters expect a UTC timestamp with seconds granularity or a relative time in seconds (preceded by -).

rrset lookups

The "rrset" lookup queries DNSDB's RRset index, which supports "forward" lookups based on the owner name of an RRset.

An owner name with a leading asterisk and label separator, (i.e., "*.") will perform a wildcard search for any RRsets whose owner names end with the given domain name. An owner name with a trailing label separator and asterisk (i.e., ".*") will perform a wildcard search for any RRsets whose owner names start with the given label(s). Note that the latter type of query is somewhat more expensive.

rrset lookups can optionally filter based on RRtype and/or bailiwick.

The results of an rrset lookup return zero or more RRsets, along with the following metadata for each result:

Item Description
count The number of times the RRset was observed via passive DNS replication.
bailiwick The "bailiwick" of an RRset in DNSDB observed via passive DNS replication is the closest enclosing zone delegated to a nameserver which served the RRset.

The "bailiwick" of an RRset in DNSDB observed in a zone file is simply the name of the zone containing the RRset.
first seen A UTC timestamp with seconds granularity indicating the first time an RRset was seen in the given bailiwick via passive DNS replication.
last seen A UTC timestamp with seconds granularity indicating the last time an RRset was seen in the given bailwick via passive DNS replication.
first seen in zone file A UTC timestamp with seconds granularity indicating the first time an RRset was seen in the given bailiwick via zone file import.
last seen in zone file A UTC timestamp with seconds granularity indicating the last time an RRset was seen in the given bailiwick via zone file import.

An rrset search result may be missing either the pair of (first seen, last seen) timestamps from passive DNS replication or from zone file import, but at least one pair of timestamps will always be present. This may change if a fundamentally new data source is introduced in the future, but there will always be at least one timestamp pair associated with an RRset.

rrset lookups follow the URL path scheme:

/lookup/rrset/name/OWNER_NAME/RRTYPE/BAILIWICK

with placeholders OWNER_NAME, RRTYPE, and BAILIWICK. Only OWNER_NAME is required; RRTYPE and BAILIWICK are optional. RRtype ANY may be specified for RRTYPE in order to perform bailiwick filtering without also filtering on a particular RRtype. OWNER_NAME and BAILIWICK are DNS names specified in DNS presentation format. RRTYPE is specified as a DNS RRtype mnemonic.

The RRtype ANY is modified somewhat from its usual meaning. A DNSDB lookup for RRtype ANY will match any RRtype except the DNSSEC-related RRtypes: DS, RRSIG, NSEC, DNSKEY, NSEC3, NSEC3PARAM, and DLV. A new pseudo-mnemonic ANY-DNSSEC has been introduced that will return only those records matching the aforementioned RRtypes.

rdata lookups

The "rdata" lookup queries DNSDB's Rdata index, which supports "inverse" lookups based on Rdata record values. In contrast to the rrset lookup method, rdata lookups return only individual resource records and not full resource record sets, and lack bailiwick metadata. An rrset lookup on the owner name reported via an rdata lookup must be performed to retrieve the full RRset and bailiwick

rdata lookups follow the URL path scheme:

/lookup/rdata/TYPE/VALUE/RRTYPE

with placeholders TYPE, VALUE, and RRTYPE. TYPE and VALUE are required; RRTYPE optionally filters the results by RRtype in the same manner as the rrset lookup.

Placeholder TYPE specifies how VALUE is interpreted. TYPE may be one of the following:

Type Description
name The VALUE is a DNS domain name in presentation format, or a left-hand (".example.com") or right-hand ("www.example.") wildcard domain name. Note that left-hand wildcard queries are somewhat more expensive than right-hand wildcard queries.
ip The VALUE is one of an IPv4 address, an IPv6 address, an IPv4 network with prefix length, or an IPv6 network with prefix length. If a network lookup is being performed, the delimiter between network address and prefix length is a single comma (",") character rather than the usual slash ("/") character to avoid clashing with the HTTP URI path name separator.
raw The VALUE is an even number of hexadecimal digits specifying a raw octet string.

Result formats

The DNSDB API supports two result formats: the ad hoc "text" format which is reminiscent of the DNS master file format, and the "json" format which returns one result per line encoded in JSON format. The two formats use the exact same URL scheme. A specific result format can be selected by specifying an Accept header in the HTTP request.

The text format is the default format if no Accept header is specified, or if the Accept header specifies the text/plain type. The JSON format is selected by requesting the application/json type.

The rrset text result format is reminiscent of the DNS master file format. Each result is an RRset separated by two newline characters and is annotated with the bailiwick and timestamp information described above, prefixed with leading ;; characters. At the end of the result document is a footer prefixed with leading ;;; characters. Currently the footer only notes how many results were found and how long it took to query the database. Note that multiple RRsets with the same owner name and RRtype can appear in the result document.

The rdata text result format is similar, except that full RRsets are not returned (only resource records), no metadata is provided, and individual results are separated by a single newline rather than two.

The rrset JSON result format is a document containing one JSON-encoded result object per line. Each result object is an associative array with the following keys.

rrest results

Key Description
rrname The owner name of the RRset in DNS presentation format.
rrtype The resource record type of the RRset, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number.
rdata An array of one or more Rdata values. The Rdata values are converted to the standard presentation format based on the rrtype value. If the encoder lacks a type-specific presentation format for the RRset's rrtype, then the RFC 3597 generic Rdata encoding will be used.
bailiwick The "bailiwick" metadata value described in the section above.
count The "count" metadata value described in the section above.
time_first, time_last UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via passive DNS replication. Will not be present if the RRset was only observed via zone file import.
zone_time_first, zone_time_last UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via zone file import. Will not be present if the RRset was only observed via passive DNS replication.

rdata results

Key Description
rrname The owner name of the resource record in DNS presentation format.
rrtype The resource record type of the resource record, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number.
rdata The record data value. The Rdata value is converted to the standard presentation format based on the rrtype value. If the encoder lacks a type-specific presentation format for the resource record's type, then the RFC 3597 generic Rdata encoding will be used.
count The number of times the resource record was observed via passive DNS replication.
time_first, time_last UNIX epoch timestamps with second granularity indicating the first and last times the resource record was observed via passive DNS replication.
zone_time_first, zone_time_last UNIX epoch timestamps with second granularity indicating the first and last times the resource record was observed via zone file import.

Service limits

The number of concurrent connections to a DNSDB API server may be limited. This limit is separate from the quota limit described below. If this limit is exceeded, the HTTP 503 "Service Unavailable" response code will be generated.

API keys may have a rate limit imposed which limits the number of requests that can be made to the /lookup/rrset and /lookup/rdata endpoints. This quota is usually applied on a daily basis and resets daily at 00:00 (midnight) in the UTC time zone. If this limit is exceeded, the HTTP 429 "Too Many Requests" response code will be generated.

The /lookup/rate_limit endpoint returns a JSON map containing a top-level map named rate that contains three numeric keys:

Key Description
reset UNIX epoch timestamp with second granularity indicating the next point in time when the quota limit will be reset. Usually this is at 00:00 (midnight) UTC.
limit The maximum number of API lookups that may be performed during the quota period.
remaining The number of remaining API lookups that may be performed during the quota period.

The following is an example of a /lookup/rate_limit response that indicates that the API key's quota limit will be reset at midnight UTC on June 11, 2015, and that 999 lookups are remaining out of a total quota of 1000 lookups:

{
    "rate": {
        "reset": 1433980800,
        "limit": 1000,
        "remaining": 999
    }
}

For API keys that have no quota limit configured ("an unlimited API key"), the response will set the three rate fields to the value -1, i.e.:

{
    "rate": {
        "reset": -1,
        "limit": -1,
        "remaining": -1
    }
}

Querying the /lookup/rate_limit endpoint does not count against the quota limit.

Additionally, responses from the /lookup/rrset and /lookup/rdata endpoints contain the same information in the HTTP response headers that can be obtained from the /lookup/rate_limit endpoint. These values are embedded as the X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset headers, with the exception that, for unlimited API keys, the fields will be encoded with the strings "unlimited" (for X-RateLimit-Limit) and "n/a" (for X-RateLimit-Remaining and X-RateLimit-Reset). Otherwise, for API keys with a quota limit defined, the values for these headers will be encoded numerically as described above for the /lookup/rate_limit endpoint.

That is, responses to lookups will contain response headers that look like this:

X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 999
X-RateLimit-Reset: 1433980800

Or, in the case of an unlimited API key:

X-RateLimit-Limit: unlimited
X-RateLimit-Remaining: n/a
X-RateLimit-Reset: n/a

Response codes

A table of HTTP response codes that can received.

Response code Description
200 OK If a successful request is returned.
400 Bad Request If the URL is formatted incorrectly.
403 Forbidden If the X-API-Key header is not present, or the provided API key is not valid.
404 Not Found Server If there are no records found for the given lookup.
429 Too Many Requests If the API key daily quota limit is exceeded.
500 Internal Server Error If there is an error processing the request.
503 Service Unavailable If the limit of number of concurrent connections is exceeded.

Example scripts

Example API clients written in Python and shell are available from https://github.com/dnsdb/dnsdb-query.

Example results

All examples are provided in both the ad hoc text result format as well as the JSON result format. For each example, a description of the lookup is provided, then the URL used to retrieve the results and sample curl commands with the current results (at the time of the writing of this document) in both text and JSON formats. For brevity, the results are limited to a total of two records and RRSIGs will be omitted in the examples below.

1. All RRsets whose owner name is www.farsightsecurity.com

URL

/lookup/rrset/name/www.farsightsecurity.com

Sample call: text result

curl -i -H 'Accept: text/plain' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rrset/name/www.farsightsecurity.com"
;;  bailiwick: farsightsecurity.com.
;;      count: 5059
;; first seen: 2013-09-25 20:02:10 -0000
;;  last seen: 2015-04-01 09:51:39 -0000
www.farsightsecurity.com. IN A 66.160.140.81

;;  bailiwick: farsightsecurity.com.
;;      count: 17381
;; first seen: 2015-04-01 13:07:24 -0000
;;  last seen: 2016-07-12 13:14:32 -0000
www.farsightsecurity.com. IN A 104.244.13.104

;;; Returned 2 RRsets in 0.03 seconds.
;;; DNSDB

Sample call: json result

curl -H 'Accept: application/json' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rrset/name/www.farsightsecurity.com"
{"count":5059,"time_first":1380139330,"time_last":1427881899,"rrname":"www.farsightsecurity.com.","rrtype":"A","bailiwick":"farsightsecurity.com.","rdata":["66.160.140.81"]}
{"count":17381,"time_first":1427893644,"time_last":1468329272,"rrname":"www.farsightsecurity.com.","rrtype":"A","bailiwick":"farsightsecurity.com.","rdata":["104.244.13.104"]}

2. All RRsets whose owner name ends in farsightsecurity.com, of type NS, in the farsightsecurity.com zone

URL

/lookup/rrset/name/*.farsightsecurity.com/ns/farsightsecurity.com

Sample call: text result

curl -i -H 'Accept: text/plain' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rrset/name/*.farsightsecurity.com/ns/farsightsecurity.com"
;;  bailiwick: farsightsecurity.com.
;;      count: 51
;; first seen: 2013-07-01 14:14:43 -0000
;;  last seen: 2013-07-17 01:17:44 -0000
farsightsecurity.com. IN NS ns.lah1.vix.com.
farsightsecurity.com. IN NS ns1.isc-sns.net.
farsightsecurity.com. IN NS ns2.isc-sns.com.
farsightsecurity.com. IN NS ns3.isc-sns.info.

;;  bailiwick: farsightsecurity.com.
;;      count: 495241
;; first seen: 2013-07-17 21:26:20 -0000
;;  last seen: 2016-07-12 12:01:16 -0000
farsightsecurity.com. IN NS ns5.dnsmadeeasy.com.
farsightsecurity.com. IN NS ns6.dnsmadeeasy.com.
farsightsecurity.com. IN NS ns7.dnsmadeeasy.com.

;;; Returned 2 RRsets in 0.04 seconds.
;;; DNSDB

Sample call: json result

curl -H 'Accept: application/json' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rrset/name/*.farsightsecurity.com/ns/farsightsecurity.com"
{"count":51,"time_first":1372688083,"time_last":1374023864,"rrname":"farsightsecurity.com.","rrtype":"NS","bailiwick":"farsightsecurity.com.","rdata":["ns.lah1.vix.com.","ns1.isc-sns.net.","ns2.isc-sns.com.","ns3.isc-sns.info."]}
{"count":495241,"time_first":1374096380,"time_last":1468324876,"rrname":"farsightsecurity.com.","rrtype":"NS","bailiwick":"farsightsecurity.com.","rdata":["ns5.dnsmadeeasy.com.","ns6.dnsmadeeasy.com.","ns7.dnsmadeeasy.com."]}

3. All resource records whose Rdata values are the IPv4 address 104.244.13.104

URL

/lookup/rdata/ip/104.244.13.104

Sample call: text result

curl -i -H 'Accept: text/plain' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/ip/104.244.13.104"
www.farsighsecurity.com. IN A 104.244.13.104
farsightsecurity.com. IN A 104.244.13.104
;;; Returned 2 RRs in 0.02 seconds.
;;; DNSDB

Sample call: json result

curl -H 'Accept: application/json' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/ip/104.244.13.104"
{"count":24,"time_first":1433550785,"time_last":1468312116,"rrname":"www.farsighsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}
{"count":9429,"time_first":1427897872,"time_last":1468333042,"rrname":"farsightsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}

4. All resource records whose Rdata values are addresses in the 104.244.13.104/29 network prefix

URL

/lookup/rdata/ip/104.244.13.104,29

Sample call: text result

curl -i -H 'Accept: text/plain' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/ip/104.244.13.104,29"
farsightsecurity.com. IN A 104.244.13.104
www.farsightsecurity.com. IN A 104.244.13.104
;;; Returned 2 RRs in 0.02 seconds.
;;; DNSDB

Sample call: json result

curl -H 'Accept: application/json' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/ip/104.244.13.104,29"
{"count":24,"time_first":1433550785,"time_last":1468312116,"rrname":"www.farsighsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}
{"count":9429,"time_first":1427897872,"time_last":1468333042,"rrname":"farsightsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}

5. All resource records whose Rdata values are the IPv6 address 2620:11c:f004::104

URL

/lookup/rdata/ip/2620:11c:f004::104

Sample call: text result

curl -i -H 'Accept: text/plain' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/ip/2620:11c:f004::104"
farsightsecurity.com. IN AAAA 2620:11c:f004::104
www.farsightsecurity.com. IN AAAA 2620:11c:f004::104
;;; Returned 2 RRs in 0.02 seconds.
;;; DNSDB

Sample call: json result

curl -H 'Accept: application/json' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/ip/2620:11c:f004::104"
{"count":14,"time_first":1433845806,"time_last":1467828872,"rrname":"www.farsighsecurity.com.","rrtype":"AAAA","rdata":"2620:11c:f004::104"}
{"count":5307,"time_first":1427897876,"time_last":1468333042,"rrname":"farsightsecurity.com.","rrtype":"AAAA","rdata":"2620:11c:f004::104"}

6. All resource records whose Rdata values are addresses in the 2620:11c:f000::/36 network prefix

URL

/lookup/rdata/ip/2620:11c:f000::,36

Sample call: text result

curl -i -H 'Accept: text/plain' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/ip/2620:11c:f000::,36"
farsightsecurity.com. IN AAAA 2620:11c:f004::104
www.farsightsecurity.com. IN AAAA 2620:11c:f004::104
;;; Returned 2 RRs in 0.02 seconds.
;;; DNSDB

Sample call: json result

curl -H 'Accept: application/json' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/ip/2620:11c:f000::,36"
{"count":5307,"time_first":1427897876,"time_last":1468333042,"rrname":"farsightsecurity.com.","rrtype":"AAAA","rdata":"2620:11c:f004::104"}
{"count":8046,"time_first":1428586271,"time_last":1468305509,"rrname":"www.farsightsecurity.com.","rrtype":"AAAA","rdata":"2620:11c:f004::104"}

7. All domain names delegated to the nameserver ns5.dnsmadeeasy.com

URL

/lookup/rdata/name/ns5.dnsmadeeasy.com

Sample call: text result

curl -i -H 'Accept: text/plain' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/name/ns5.dnsmadeeasy.com"
farsightsecurity.com. IN NS ns5.dnsmadeeasy.com.
farsightsecurity.com. IN NS ns5.dnsmadeeasy.com.
;;; Returned 2 RRs in 0.02 seconds.
;;; DNSDB

Sample call: json result

curl -H 'Accept: application/json' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/name/ns5.dnsmadeeasy.com"
{"count":1078,"zone_time_first":1374250920,"zone_time_last":1468253883,"rrname":"farsightsecurity.com.","rrtype":"NS","rdata":"ns5.dnsmadeeasy.com."}
{"count":706617,"time_first":1374096380,"time_last":1468334926,"rrname":"farsightsecurity.com.","rrtype":"NS","rdata":"ns5.dnsmadeeasy.com."}

8. All domain names whose mail exchanges are the server hq.fsi.io

URL

/lookup/rdata/name/hq.fsi.io

Sample call: text result

curl -i -H 'Accept: text/plain' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/name/hq.fsi.io"
fsi.io. IN MX 10 hq.fsi.io.
farsightsecurity.com. IN MX 10 hq.fsi.io.
;;; Returned 2 RRs in 0.02 seconds.
;;; DNSDB

Sample call: json result

curl -H 'Accept: application/json' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rdata/name/hq.fsi.io"
{"count":45644,"time_first":1372706073,"time_last":1468330740,"rrname":"fsi.io.","rrtype":"MX","rdata":"10 hq.fsi.io."}
{"count":19304,"time_first":1374098929,"time_last":1468333042,"rrname":"farsightsecurity.com.","rrtype":"MX","rdata":"10 hq.fsi.io."}

9. A wildcard search for RRsets whose owner name is farsightsecurity.com, rrtype is NS, bailiwick is farsightsecurity.com, seen after the date/time 2016-07-12T00:00:00Z with a limit of 100 results.

URL

/lookup/rrset/name/*.farsightsecurity.com/ns/farsightsecurity.com?limit=100&ime_last_after=2016-07-12T00:00:00Z

Sample call: text result

curl -i -H 'Accept: text/plain' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rrset/name/*.farsightsecurity.com/ns/farsightsecurity.com?limit=100&ime_last_after=2016-07-12T00:00:00Z"
;;  bailiwick: farsightsecurity.com.
;;      count: 51
;; first seen: 2013-07-01 14:14:43 -0000
;;  last seen: 2013-07-17 01:17:44 -0000
farsightsecurity.com. IN NS ns.lah1.vix.com.
farsightsecurity.com. IN NS ns1.isc-sns.net.
farsightsecurity.com. IN NS ns2.isc-sns.com.
farsightsecurity.com. IN NS ns3.isc-sns.info.

;;  bailiwick: farsightsecurity.com.
;;      count: 496124
;; first seen: 2013-07-17 21:26:20 -0000
;;  last seen: 2016-07-13 09:38:37 -0000
farsightsecurity.com. IN NS ns5.dnsmadeeasy.com.
farsightsecurity.com. IN NS ns6.dnsmadeeasy.com.
farsightsecurity.com. IN NS ns7.dnsmadeeasy.com.

;;; Returned 2 RRsets in 0.03 seconds.
;;; DNSDB

Sample call: json result

curl -H 'Accept: application/json' -H 'X-API-Key: <elided>' "https://api.dnsdb.info/lookup/rrset/name/*.farsightsecurity.com/ns/farsightsecurity.com?limit=100&ime_last_after=2016-07-12T00:00:00Z"
{"count":51,"time_first":1372688083,"time_last":1374023864,"rrname":"farsightsecurity.com.","rrtype":"NS","bailiwick":"farsightsecurity.com.","rdata":["ns.lah1.vix.com.","ns1.isc-sns.net.","ns2.isc-sns.com.","ns3.isc-sns.info."]}
{"count":496124,"time_first":1374096380,"time_last":1468402717,"rrname":"farsightsecurity.com.","rrtype":"NS","bailiwick":"farsightsecurity.com.","rdata":["ns5.dnsmadeeasy.com.","ns6.dnsmadeeasy.com.","ns7.dnsmadeeasy.com."]}

Changes

2016-07-12:

2015-06-10:

2014-12-02:

2014-07-17:

2013-07-25:

2012-05-02:

2011-05-04: