DNSDB API

Changes:

2015-06-10:

2014-12-02:

2014-07-17:

2013-07-25:

2012-05-02:

2011-05-04:

I. Introduction

DNSDB is a database that stores and indexes both the passive DNS data available via Farsight Security's Security Information Exchange as well as the authoritative DNS data that various zone operators make available. DNSDB makes it easy to search for individual DNS RRsets and provides additional metadata for search results such as first seen and last seen timestamps as well as the DNS bailiwick associated with an RRset. DNSDB also has the ability to perform inverse or rdata searches.

This document describes how to make bulk, automated DNSDB queries via the HTTP API.

II. Authentication

The DNSDB API is provided over an encrypted HTTPS transport over the Internet at the following URL:

https://api.dnsdb.info/

Authentication is performed by providing an API key (a long string of hexadecimal digits) in a special HTTP request header, X-API-Key. For example, if your API key is d41d8cd98f00b204e9800998ecf8427e, then the following HTTP header should be added to the HTTP request:

X-API-Key: d41d8cd98f00b204e9800998ecf8427e

If the X-API-Key header is not present, or the provided API key is not valid, a 403 Forbidden response will be returned to the HTTP client.

To request an API key, please see the Farsight Security service application form.

III. Lookup methods

All DNSDB lookup requests are rooted in URL paths under the /lookup hierarchy. There are two separate lookup methods, "rrset" and "rdata", which are accessed via the URL paths /lookup/rrset and /lookup/rdata respectively.

There is a built-in limit to the number of results that are returned via these lookup methods. This limit can be raised or lowered by setting the "limit" query parameter. E.g., appending "?limit=10000" to the URL path will set the response limit to 10,000 results.

You may filter results by time using the "time_first_before," "time_first_after," "time_last_before," and "time_last_after" query parameters. These parameters expect a UTC timestamp with seconds granularity or a relative time in seconds (preceded by -).

1. rrset lookups

The "rrset" lookup queries DNSDB's RRset index, which supports "forward" lookups based on the owner name of an RRset.

An owner name with a leading asterisk and label separator, (i.e., "*.") will perform a wildcard search for any RRsets whose owner names end with the given domain name. An owner name with a trailing label separator and asterisk (i.e., ".*") will perform a wildcard search for any RRsets whose owner names start with the given label(s). Note that the latter type of query is somewhat more expensive.

rrset lookups can optionally filter based on RRtype and/or bailiwick.

The results of an rrset lookup return zero or more RRsets, along with the following metadata for each result:

count

The number of times the RRset was observed via passive DNS replication.

bailiwick

The "bailiwick" of an RRset in DNSDB observed via passive DNS replication is the closest enclosing zone delegated to a nameserver which served the RRset.

The "bailiwick" of an RRset in DNSDB observed in a zone file is simply the name of the zone containing the RRset.

first seen

A UTC timestamp with seconds granularity indicating the first time an RRset was seen in the given bailiwick via passive DNS replication.

last seen

A UTC timestamp with seconds granularity indicating the last time an RRset was seen in the given bailwick via passive DNS replication.

first seen in zone file

A UTC timestamp with seconds granularity indicating the first time an RRset was seen in the given bailiwick via zone file import.

last seen in zone file

A UTC timestamp with seconds granularity indicating the last time an RRset was seen in the given bailiwick via zone file import.

An rrset search result may be missing either the pair of (first seen, last seen) timestamps from passive DNS replication or from zone file import, but at least one pair of timestamps will always be present. This may change if a fundamentally new data source is introduced in the future, but there will always be at least one timestamp pair associated with an RRset.

rrset lookups follow the URL path scheme:

/lookup/rrset/name/OWNER_NAME/RRTYPE/BAILIWICK

with placeholders OWNER_NAME, RRTYPE, and BAILIWICK. Only OWNER_NAME is required; RRTYPE and BAILIWICK are optional. RRtype ANY may be specified for RRTYPE in order to perform bailiwick filtering without also filtering on a particular RRtype. OWNER_NAME and BAILIWICK are DNS names specified in DNS presentation format. RRTYPE is specified as a DNS RRtype mnemonic.

The RRtype ANY is modified somewhat from its usual meaning. A DNSDB lookup for RRtype ANY will match any RRtype except the DNSSEC-related RRtypes: DS, RRSIG, NSEC, DNSKEY, NSEC3, NSEC3PARAM, and DLV. A new pseudo-mnemonic ANY-DNSSEC has been introduced that will return only those records matching the aforementioned RRtypes.

2. rdata lookups

The "rdata" lookup queries DNSDB's Rdata index, which supports "inverse" lookups based on Rdata record values. In contrast to the rrset lookup method, rdata lookups return only individual resource records and not full resource record sets, and lack bailiwick metadata. An rrset lookup on the owner name reported via an rdata lookup must be performed to retrieve the full RRset and bailiwick.

rdata lookups follow the URL path scheme:

/lookup/rdata/TYPE/VALUE/RRTYPE

with placeholders TYPE, VALUE, and RRTYPE. TYPE and VALUE are required; RRTYPE optionally filters the results by RRtype in the same manner as the rrset lookup.

TYPE specifies how VALUE is interpreted. TYPE may be one of the following:

name

The VALUE is a DNS domain name in presentation format, or a left-hand (".example.com") or right-hand ("www.example.") wildcard domain name. Note that left-hand wildcard queries are somewhat more expensive than right-hand wildcard queries.

ip

The VALUE is one of an IPv4 address, an IPv6 address, an IPv4 network with prefix length, or an IPv6 network with prefix length. If a network lookup is being performed, the delimiter between network address and prefix length is a single comma (",") character rather than the usual slash ("/") character to avoid clashing with the HTTP URI path name separator.

raw

The VALUE is an even number of hexadecimal digits specifying a raw octet string.

IV. Result formats

The DNSDB API supports two result formats: the ad hoc "text" format which is reminiscient of the DNS master file format, and the "json" format which returns one result per line encoded in JSON format. The two formats use the exact same URL scheme. A specific result format can be selected by specifying an Accept header in the HTTP request.

The text format is the default format if no Accept header is specified, or if the Accept header specifies the text/plain type. The JSON format is selected by requesting the application/json type.

The rrset text result format is reminiscient of the DNS master file format. Each result is an RRset separated by two newline characters and is annotated with the bailiwick and timestamp information described above, prefixed with leading ;; characters. At the end of the result document is a footer prefixed with leading ;;; characters. Currently the footer only notes how many results were found and how long it took to query the database. Note that multiple RRsets with the same owner name and RRtype can appear in the result document.

The rdata text result format is similar, except that full RRsets are not returned (only resource records), no metadata is provided, and individual results are separated by a single newline rather than two.

The rrset JSON result format is a document containing one JSON-encoded result object per line. Each result object is an associative array with the following keys.

1. rrset results

rrname

The owner name of the RRset in DNS presentation format.

rrtype

The resource record type of the RRset, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number.

rdata

An array of one or more Rdata values. The Rdata values are converted to the standard presentation format based on the rrtype value. If the encoder lacks a type-specific presentation format for the RRset's rrtype, then the RFC 3597 generic Rdata encoding will be used.

bailiwick

The "bailiwick" metadata value described in the section above.

count

The "count" metadata value described in the section above.

time_first, time_last

UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via passive DNS replication. Will not be present if the RRset was only observed via zone file import.

zone_time_first, zone_time_last

UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via zone file import. Will not be present if the RRset was only observed via passive DNS replication.

2. rdata results

rrname

The owner name of the resource record in DNS presentation format.

rrtype

The resource record type of the resource record, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number.

rdata

The record data value. The Rdata value is converted to the standard presentation format based on the rrtype value. If the encoder lacks a type-specific presentation format for the resource record's type, then the RFC 3597 generic Rdata encoding will be used.

count

The number of times the resource record was observed via passive DNS replication.

time_first, time_last

UNIX epoch timestamps with second granularity indicating the first and last times the resource record was observed via passive DNS replication.

zone_time_first, zone_time_last

UNIX epoch timestamps with second granularity indicating the first and last times the resource record was observed via zone file import.

V. Service limits

The number of concurrent connections to a DNSDB API server may be limited. This limit is separate from the quota limit described below. If this limit is exceeded, the HTTP 503 "Service Unavailable" response code will be generated.

API keys may have a rate limit imposed which limits the number of requests that can be made to the /lookup/rrset and /lookup/rdata endpoints. This quota is usually applied on a daily basis and resets daily at 00:00 (midnight) in the UTC time zone. If this limit is exceeded, the HTTP 429 "Too Many Requests" response code will be generated.

The /lookup/rate_limit endpoint returns a JSON map containing a top-level map named rate that contains three numeric keys:

reset

UNIX epoch timestamp with second granularity indicating the next point in time when the quota limit will be reset. Usually this is at 00:00 (midnight) UTC.

limit

The maximum number of API lookups that may be performed during the quota period.

remaining

The number of remaining API lookups that may be performed during the quota period.

The following is an example of a /lookup/rate_limit response that indicates that the API key's quota limit will be reset at midnight UTC on June 11, 2015, and that 999 lookups are remaining out of a total quota of 1000 lookups:

{
    "rate": {
        "reset": 1433980800,
        "limit": 1000,
        "remaining": 999
    }
}

For API keys that have no quota limit configured ("an unlimited API key"), the response will set the three rate fields to the value -1, i.e.:

{
    "rate": {
        "reset": -1,
        "limit": -1,
        "remaining": -1
    }
}

Querying the /lookup/rate_limit endpoint does not count against the quota limit.

Additionally, responses from the /lookup/rrset and /lookup/rdata endpoints contain the same information in the HTTP response headers that can be obtained from the /lookup/rate_limit endpoint. These values are embedded as the X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset headers, with the exception that, for unlimited API keys, the fields will be encoded with the strings "unlimited" (for X-RateLimit-Limit) and "n/a" (for X-RateLimit-Remaining and X-RateLimit-Reset). Otherwise, for API keys with a quota limit defined, the values for these headers will be encoded numerically as described above for the /lookup/rate_limit endpoint.

That is, responses to lookups will contain response headers that look like this:

X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 999
X-RateLimit-Reset: 1433980800

Or, in the case of an unlimited API key:

X-RateLimit-Limit: unlimited
X-RateLimit-Remaining: n/a
X-RateLimit-Reset: n/a

VI. Example scripts

Example API clients written in Python and shell are available from https://github.com/dnsdb/dnsdb-query.

VII. Example results

All examples are provided in both the ad hoc text result format as well as the JSON result format. For each example, a description of the lookup is provided, then the URL used to retrieve the results, and then the current results (at the time of the writing of this document) in both text and JSON formats. For brevity, RRSIGs will be omitted in the examples below.

The string DNSDB_SERVER should be substituted for the DNSDB API service URL, that is, https://api.dnsdb.info/.

1. All RRsets whose owner name is www.isc.org

URL: DNSDB_SERVER/lookup/rrset/name/www.isc.org

Text result:

;;  bailiwick: isc.org.
;;      count: 195242
;; first seen: 2010-06-24 04:29:04 -0000
;;  last seen: 2011-05-04 13:57:12 -0000
;; first seen in zone file: 2010-07-21 01:58:05 -0000
;;  last seen in zone file: 2011-05-01 00:33:59 -0000
www.isc.org. IN A 149.20.64.42

;;  bailiwick: isc.org.
;;      count: 67987
;; first seen: 2010-06-24 10:34:35 -0000
;;  last seen: 2011-05-04 14:33:24 -0000
;; first seen in zone file: 2010-07-21 01:58:05 -0000
;;  last seen in zone file: 2011-05-01 00:33:59 -0000
www.isc.org. IN AAAA 2001:4f8:0:2::d

;;; found 2 RRsets in 0.02 seconds
;;; SIE DNSDB

JSON result:

{"count": 195242, "time_first": 1277353744, "rrtype": "A", "rrname": "www.isc.org.", "zone_time_first": 1279677485, "zone_time_last": 1304210039, "bailiwick": "isc.org.", "rdata": ["149.20.64.42"], "time_last": 1304517432}
{"count": 67987, "time_first": 1277375675, "rrtype": "AAAA", "rrname": "www.isc.org.", "zone_time_first": 1279677485, "zone_time_last": 1304210039, "bailiwick": "isc.org.", "rdata": ["2001:4f8:0:2::d"], "time_last": 1304519604}

2. All RRsets whose owner name ends in isc.org, of type NS, in the isc.org zone

URL: DNSDB_SERVER/lookup/rrset/name/*.isc.org/ns/isc.org

Text result:

;;  bailiwick: isc.org.
;; first seen: 2010-06-24 03:07:35 -0000
;;  last seen: 2010-09-14 20:37:17 -0000
;; first seen in zone file: 2010-07-21 01:58:05 -0000
;;  last seen in zone file: 2010-08-09 22:11:44 -0000
isc.org. IN NS ams.sns-pb.isc.org.
isc.org. IN NS ord.sns-pb.isc.org.
isc.org. IN NS sfba.sns-pb.isc.org.
isc.org. IN NS ns.isc.afilias-nst.info.

;;  bailiwick: isc.org.
;; first seen: 2010-06-24 22:42:58 -0000
;;  last seen: 2010-09-11 00:39:37 -0000
;; first seen in zone file: 2010-07-21 01:58:05 -0000
;;  last seen in zone file: 2010-08-09 22:11:44 -0000
dv.isc.org. IN NS ns-ext.isc.org.
dv.isc.org. IN NS ns-int.isc.org.

;;  bailiwick: isc.org.
;; first seen: 2010-06-24 03:27:19 -0000
;;  last seen: 2010-09-14 21:08:07 -0000
;; first seen in zone file: 2010-07-21 01:58:05 -0000
;;  last seen in zone file: 2010-08-09 22:11:44 -0000
dlv.isc.org. IN NS ns1.isc.ultradns.net.
dlv.isc.org. IN NS ns2.isc.ultradns.net.
dlv.isc.org. IN NS dlv.ams.sns-pb.isc.org.
dlv.isc.org. IN NS dlv.ord.sns-pb.isc.org.
dlv.isc.org. IN NS ns.isc.afilias-nst.info.
dlv.isc.org. IN NS dlv.sfba.sns-pb.isc.org.

;;; found 3 RRsets in 1.08 seconds
;;; SIE DNSDB

JSON result:

{"time_first": 1277348855, "rrtype": "NS", "rrname": "isc.org.", "zone_time_first": 1279677485, "zone_time_last": 1281391904, "bailiwick": "isc.org.", "rdata": ["ams.sns-pb.isc.org.", "ord.sns-pb.isc.org.", "sfba.sns-pb.isc.org.", "ns.isc.afilias-nst.info."], "time_last": 1284496637}
{"time_first": 1277419378, "rrtype": "NS", "rrname": "dv.isc.org.", "zone_time_first": 1279677485, "zone_time_last": 1281391904, "bailiwick": "isc.org.", "rdata": ["ns-ext.isc.org.", "ns-int.isc.org."], "time_last": 1284165577}
{"time_first": 1277350039, "rrtype": "NS", "rrname": "dlv.isc.org.", "zone_time_first": 1279677485, "zone_time_last": 1281391904, "bailiwick": "isc.org.", "rdata": ["ns1.isc.ultradns.net.", "ns2.isc.ultradns.net.", "dlv.ams.sns-pb.isc.org.", "dlv.ord.sns-pb.isc.org.", "ns.isc.afilias-nst.info.", "dlv.sfba.sns-pb.isc.org."], "time_last": 1284498487}

3. All resource records whose Rdata values are the IP address 204.152.187.5

URL: DNSDB_SERVER/lookup/rdata/ip/204.152.187.5

Text result:

farside.isc.org. IN A 204.152.187.5

;;; found 1 RRs in 0.02 seconds
;;; SIE DNSDB

JSON result:

{"rrtype": "A", "rrname": "farside.isc.org.", "rdata": "204.152.187.5"}

4. All resource records whose Rdata values are addresses in the 204.152.187.0/28 network prefix

URL: DNSDB_SERVER/lookup/rdata/ip/204.152.187.0,28

Text result:

eject.isc.org. IN A 204.152.187.3
k1.isc.org. IN A 204.152.187.4
farside.isc.org. IN A 204.152.187.5
six.vix.com. IN A 204.152.187.6
newtag.isc.org. IN A 204.152.187.7
blackhole.isc.org. IN A 204.152.187.8
newfarside.isc.org. IN A 204.152.187.9
caltest.isc.org. IN A 204.152.187.10
pyrenees-dev.isc.org. IN A 204.152.187.10
nook.isc.org. IN A 204.152.187.12
res1.sql1.isc.org. IN A 204.152.187.13
res2.sql1.isc.org. IN A 204.152.187.14

;;; found 12 RRs in 0.07 seconds
;;; SIE DNSDB

JSON result:

{"rrtype": "A", "rrname": "eject.isc.org.", "rdata": "204.152.187.3"}
{"rrtype": "A", "rrname": "k1.isc.org.", "rdata": "204.152.187.4"}
{"rrtype": "A", "rrname": "farside.isc.org.", "rdata": "204.152.187.5"}
{"rrtype": "A", "rrname": "six.vix.com.", "rdata": "204.152.187.6"}
{"rrtype": "A", "rrname": "newtag.isc.org.", "rdata": "204.152.187.7"}
{"rrtype": "A", "rrname": "blackhole.isc.org.", "rdata": "204.152.187.8"}
{"rrtype": "A", "rrname": "newfarside.isc.org.", "rdata": "204.152.187.9"}
{"rrtype": "A", "rrname": "caltest.isc.org.", "rdata": "204.152.187.10"}
{"rrtype": "A", "rrname": "pyrenees-dev.isc.org.", "rdata": "204.152.187.10"}
{"rrtype": "A", "rrname": "nook.isc.org.", "rdata": "204.152.187.12"}
{"rrtype": "A", "rrname": "res1.sql1.isc.org.", "rdata": "204.152.187.13"}
{"rrtype": "A", "rrname": "res2.sql1.isc.org.", "rdata": "204.152.187.14"}

5. All resource records whose Rdata values are the IP address 2001:500:2f::f

URL: DNSDB_SERVER/lookup/rdata/ip/2001:500:2f::f

Text result:

f.root-servers.net. IN AAAA 2001:500:2f::f

;;; found 1 RRs in 0.15 seconds
;;; SIE DNSDB

JSON result:

{"rrtype": "AAAA", "rrname": "f.root-servers.net.", "rdata": "2001:500:2f::f"}

6. All resource records whose Rdata values are addresses in the 2001:4f8:3:200::/64 network prefix

URL: DNSDB_SERVER/lookup/rdata/ip/2001:4f8:3:200::,64

Text result:

dnsdb.isc.org. IN AAAA 2001:4f8:3:200::15
dnsdb-api.isc.org. IN AAAA 2001:4f8:3:200::15
dnsdb-dev.isc.org. IN AAAA 2001:4f8:3:200::16
dnsdb-api-dev.isc.org. IN AAAA 2001:4f8:3:200::16

JSON result:

{"rrtype": "AAAA", "rrname": "dnsdb.isc.org.", "rdata": "2001:4f8:3:200::15"}
{"rrtype": "AAAA", "rrname": "dnsdb-api.isc.org.", "rdata": "2001:4f8:3:200::15"}
{"rrtype": "AAAA", "rrname": "dnsdb-dev.isc.org.", "rdata": "2001:4f8:3:200::16"}
{"rrtype": "AAAA", "rrname": "dnsdb-api-dev.isc.org.", "rdata": "2001:4f8:3:200::16"}

7. All domain names delegated to the nameserver ns-int.isc.org

NOTE: This query also returns a PTR record for the name in question.

URL: DNSDB_SERVER/lookup/rdata/name/ns-int.isc.org

Text result:

dv.isc.org. IN NS ns-int.isc.org.
0.2.8.0.0.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. IN NS ns-int.isc.org.
65.184.152.204.in-addr.arpa. IN PTR ns-int.isc.org.

;;; found 3 RRs in 0.02 seconds
;;; SIE DNSDB

JSON result:

{"rrtype": "NS", "rrname": "dv.isc.org.", "rdata": "ns-int.isc.org."}
{"rrtype": "NS", "rrname": "0.2.8.0.0.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa.", "rdata": "ns-int.isc.org."}
{"rrtype": "PTR", "rrname": "65.184.152.204.in-addr.arpa.", "rdata": "ns-int.isc.org."}

8. All domain names whose mail exchanges are the server mx.pao1.isc.org

NOTE: This query also returns PTR records for the name in question.

URL: DNSDB_SERVER/lookup/rdata/name/mx.pao1.isc.org

Text result:

b.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.0.0.0.0.0.8.f.4.0.1.0.0.2.ip6.arpa. IN PTR mx.pao1.isc.org.
53.64.20.149.in-addr.arpa. IN PTR mx.pao1.isc.org.
isc.org. IN MX 10 mx.pao1.isc.org.
idn.isc.org. IN MX 10 mx.pao1.isc.org.
lists.isc.org. IN MX 10 mx.pao1.isc.org.
support.isc.org. IN MX 10 mx.pao1.isc.org.
iepg.org. IN MX 10 mx.pao1.isc.org.
sanog.org. IN MX 10 mx.pao1.isc.org.

;;; found 8 RRs in 0.08 seconds
;;; SIE DNSDB

JSON result:

{"rrtype": "PTR", "rrname": "b.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.0.0.0.0.0.8.f.4.0.1.0.0.2.ip6.arpa.", "rdata": "mx.pao1.isc.org."}
{"rrtype": "PTR", "rrname": "53.64.20.149.in-addr.arpa.", "rdata": "mx.pao1.isc.org."}
{"rrtype": "MX", "rrname": "isc.org.", "rdata": "10 mx.pao1.isc.org."}
{"rrtype": "MX", "rrname": "idn.isc.org.", "rdata": "10 mx.pao1.isc.org."}
{"rrtype": "MX", "rrname": "lists.isc.org.", "rdata": "10 mx.pao1.isc.org."}
{"rrtype": "MX", "rrname": "support.isc.org.", "rdata": "10 mx.pao1.isc.org."}
{"rrtype": "MX", "rrname": "iepg.org.", "rdata": "10 mx.pao1.isc.org."}
{"rrtype": "MX", "rrname": "sanog.org.", "rdata": "10 mx.pao1.isc.org."}