Product Blog

The intention of this document is to show you the basics of how to download the WhoisXML API’s NRD2 data feed provided by WhoisXML API to an AWS S3 bucket by leveraging a serverless Lambda function. AWS Lambda functions act as a serverless compute service that allows you to write and execute code without provisioning or managing servers. AWS S3 is an object storage service for storing and retrieving files. This document will guide you through the process of configuring both AWS Lambda and an AWS S3 bucket.  

As part of WhoisXML API’s efforts to continuously improve its products and services, the coverage and quality of the Newly Registered Domains (NRD2) Data Feed significantly improved over the past months. More precisely, the latest Q3 2023 measurements show that the data feed’s coverage increased by 89% since the last quarter of 2022.

Within the next months, we will stop supporting Newly Registered Domain (NRD) 1.0 and steadily continue user migration to NRD 2.0, an upgraded version with more comprehensive data coverage and event monitoring than ever before.

In the first part of this blog, we demonstrated how to download data from WhoisXML API’s daily data feeds right after their publication by using the recently introduced RSS feed as the activator of download. In particular, we showed how to download the list of domains newly registered in the .com top-level domain (TLD). 

Now, to make the task a bit more interesting we demonstrate the use of our domain list with a showcase application: we calculate the list of the most frequent English words in the domain names on that day. This can be interesting in various applications. Domainers, for instance, can get information on the newest trends in domain registrations. Journalists and researchers can get a clue on a topic gaining popularity, etc.

This technical blog aims to demonstrate how to download data from WhoisXML API’s daily data feeds right after their publication. Obtaining lists of newly registered domains and their WHOIS data can be critical in many applications. (We will showcase such an application in the second part of this blog.) Recently an RSS feed has been introduced for the service that informs about data publication immediately, which makes this easy. As a demonstration, we shall go through a particular task: download the list of domains newly registered in the .com top-level domain (TLD). 

We will use a Linux system and its understanding requires intermediate programming and command-line skills. We will use BASH but it is also easy to modify for zsh, which is the default on Mac OS X. We assume Python, with pip and virtualenv.