Introducing Shodan Reports

As some of you have already seen, I’ve been working on a revamped version of Shodan. It already has some cool new enhancements over the current shodanhq.com website, such as support for CSV and JSON exports, prettier search listing, faster results, better integration with Shodan Exploits/ Maps and a lot of small tweaks to make life easier.

The latest feature that I’m adding is¬†Shodan Reports:

Shodan

To get a feel for what I’m talking about, please check out the following example report on Industrial Control Systems:

https://www.shodan.io/report/l7VjfVKc

A report is a snapshot and overview of the search results at the time of report creation. At the moment, it creates a bunch of charts/ graphs for breakdowns on: location, organization, operating system, product, hostname and many more (see the developer documentation for a list of all facets). The purpose for reports is 3-fold:

  1. Pretty Overview That You Can Share
    You should be able to get a basic feel for the devices in the search results just from looking at the report. And if you’re interested in the details, you can perform a live search to get a listing of actual results. Reports are meant to be accessible!
  2. Tracking Results Over Time
    As mentioned earlier, reports are snapshots of the search results as Shodan sees them at the moment. You can create reports every few months to see how things are changing over time (this might actually be built into Shodan Reports as well)
  3. Bookmarks!
    The old shodanhq.com website has bookmarks (there’s a small star icon at the top of the search results) but it was rarely used and didn’t offer much that browser bookmarks didn’t. I’m hoping that these reports will provide a prettier bookmarking mechanism that’s also more useful!

Alright, you’re sold on reports and want to give it a shot yourself – here is how.

  1. Login to Shodan and perform a search. For example: Webcams
  2. Click on “Create Report” and give your report a title such as “Webcams of the World”Server  SQ WEBCAM   Shodan Search
  3. You will be redirected to the page that will list all your reportsShodan-report-start
  4. Now just wait a few minutes for Shodan to generate your report, you will receive an email when it’s done! Once it’s ready, you can follow the link and you should see something like:Shodan-sample
  5. And you can always find a list of all the reports you’ve generated by clicking on the chart icon in the top right cornerShodan-list

I created it because I needed a friendlier way to share search results and I got tired of manually generating my charts for blog posts ūüôā It’s very simple and straight-forward to create reports so give it a try and let me know what you think!

Kicking the Shodan API Up a Notch

Most people think of Shodan as a search engine of banners, and while that was true for a long time it’s slowly grown into collecting more information than just banners. Here are just a few of the things¬†Shodan¬†does nowadays:

  • Checks for heartbleed on all SSL services
  • Gets a list of peers from a Bitcoin server
  • Determines whether a DNS server allows recursive lookups
  • Gets a listing of all the MongoDB databases
  • Grabs the SSL certificate for all SSL services
  • Collects the robots.txt for HTTP services

And that’s just the tip of the iceberg, there is a lot of other protocol-specific data gathered for each service. But making all of that information available through the web interface can be a bit tricky and takes time to get right. Instead of sitting on that information though, I decided to make all of the data available through a new and improved Shodan API. Note that a new web interface is being developed to expose all the cool new data, but if you don’t want to wait you can get your hands on it right now with the API.

Along with access to more detailed data, the new Shodan REST API also provides greater flexibility so you can get a top listing of the stuff you care about. For example, the old API would always return the top 5 countries and cities for your search query. What if you didn’t want those? What if you’d prefer to get a breakdown of the top organizations? Or the uptime distribution? Or you don’t care about the actual results and only want the top 100 cities that match a search query? All of those things are now possible with the new API. And if all you want are the actual results, then searching the new API will be much faster than before.

For most people, the Developer API plan that costs a one-time fee of $9 is enough to run their scripts each month and get what is needed out of Shodan. In case you need more access though, there are 3 new API plans to help you out:

  • Freelancer: $19/ month
  • Small Business: $99/ month
  • Enterprise: $499/ month

Each of them provides increased levels of access, from 1 million results/ month for Freelancer up to completely unlimited access to the REST API at the Enterprise plan.

In addition to the changes to the REST API, I’m also introducing the Shodan Streaming API! It’s basically a real-time firehose of all the data that Shodan is collecting. You can either subscribe to all the data, ask the API to filter the banners based on a port (ex. stream only MySQL and PostgreSQL banners) or¬†only get the geographic information for a banner (for geo visualizations). There are 2 caveats: you have to be on a subscription API plan and it currently only streams ~1% of the data. If you or your company would like 100% of the data please contact me for pricing information. Those notes aside, it still provides quite a bit of data in real-time and I’m hoping it’ll provide a useful way to do cool stuff with Shodan. Here is some basic example code to get you started using the Python library:¬†https://gist.github.com/achillean/f953cb917a7eb2a9f81d

The Shodan API will be a major focus for the next year so¬†watch the documentation for new filters and facets that will be released soon ūüôā

 

Shodan Plugin for Chrome

I’ve just released a new plugin for Chrome browser that lets you see what data Shodan has available for the website you’re currently visiting. This information includes the IP, hostnames (as determined via PTR lookups), location of the server (city, country), operating system, organization that owns the IP space as well as a list of open ports. As soon as you visit a website, it automatically asks the Shodan API (hence the api.shodan.io permission requirement) for all that information and then compiles it into a small pop-up that you can view when clicking on the Shodan icon next to the address bar:

shodan-chrome

The pop-up for the Shodan plugin is as simple as possible, which also means it doesn’t show the full banners for every port. There wasn’t enough room to fit everything in, especially for some of the hosts that have tons of ports open.¬†I’m looking to find a way to show all of that information and more in future releases! For now, I’m happy to finally have a way to quickly see who’s hosting a website (the hostname often gives that away) and if they’re running some unusual services publicly. Oh, and if you see a dark blue square that means it can be opened by your browser and will open up a new tab with the IP:port if you click on it (in case you want to see what it shows when you access the webserver via its IP instead of hostname).

Go download the plugin and let me know what you think!

https://chrome.google.com/webstore/detail/shodan/jjalcfnidlmpjhdfepjhjbhnhkbgleap

Blast from the Past

As I flesh out more of the services that are offered on Shodan, I’ve started to look at some older protocols. To that end, I’ve added the following services:

All of these protocols have been deprecated due to security issues or superseded by better alternatives. Even though they’re probably not in the active minds of the modern sysadmin, these protocols are still alive on the Internet!

Systat

Displays information about the processes that are currently running on the system. Read More

  • Port: 11
  • Results: 2,969

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 47540 3680 ? Ss Mar04 0:01 /sbin/init
root 2 0.0 0.0 0 0 ? S Mar04 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Mar04 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< Mar04 0:00 [kworker/0:0H]
root 7 0.0 0.0 0 0 ? S Mar04 0:00 [migration/0]
root 8 0.0 0.0 0 …

Note that the majority of the results don’t appear to actually be results of systat. Instead, it looks like the port has been re-purposed by a few people to run FTP, SSH and HTTP servers. This will also be true for netstat, where a lot of the results are from popular protocols running on a non-standard port.

Daytime

A simple protocol that returns the current date and time for the server.

  • Port: 13
  • Results: 92,539

Tuesday, March 30, 1993 14:14:55-GMT

Netstat

Shows all the currently active network connections on the device.

  • Port: 15
  • Results: 2,234

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:cvspserver *:* LISTEN
tcp 0 0 *:amandaidx *:* LISTEN
tcp 0 0 *:printer *:* LISTEN
tcp 0 0 *:amidxtape *:* LISTEN
tcp 0 0 *:wbem-https …

Quote of the Day

As the name implies it shows a quote when a user connects to the service. Read More

  • Port: 17
  • Results: 40,574

“We want a few mad people now. See where the sane ones have landed us!”
George Bernard Shaw (1856-1950)

Finger

A service that lets you get information about user accounts on the server. Read More

  • Port: 79
  • Results: 59,699

Line User Host(s) Idle Location
* 66 vty 0 idle 00:00:00

There will be separate blog posts to look at the details of who’s still running these ancient services, but the data’s already been gathered and is searchable on Shodan. So please check to make sure your organization isn’t inadvertently using any of these protocols! I’ll be keeping track of these services over the coming months to determine whether these protocols are becoming more or less active and by how much.

Introducing Shodan Maps

If you’ve followed my blog posts and Twitter stream, you’ve probably seen images like this:

Global Backdoor Exposure

The above is a map of the globally exposed routers that contain a backdoor, and the way I’ve generated those images has until now been an internal tool. After making some changes to increase performance and make it more user-friendly, I’m officially releasing a new way to browse the Shodan search engine in the form of an add-on: Shodan Maps.

Shodan Maps

Shodan Maps provides a new and easy way to search for devices on the Internet and see the results on a map instead of a regular search listing. As you zoom into an area, it will narrow down your search results to only show devices that are within the viewable area. It essentially works and behaves the same way as every other map search service (Google Maps, Bing Maps etc.). It will currently display up to 1,000 results at a time on-screen as well as summary information about all the results (location-independent) such as top 5 services, organizations and countries. Note that it can take up to 30+ seconds sometimes to complete a search, depending on how far zoomed out you are and how many results there are in total.

There are also 3 different map styles that can be used depending on your preference. The default map style is “Satellite”, which is what I’ve been using most often and can be seen in the screenshot above. The next available map style is “Street View (Light)”:

Map Style: Light

And finally there’s “Street View (Dark)”:

Map Style: Dark

The map style can be changed anytime using the settings button next to the big read Search button. And when you click on one of the red dots you will see more information about the device, such as which services it’s running, who owns the IP space and anything else that is useful to know.

Host Information

I’ve found it very insightful to put results on a map and figure out patterns in the data, hopefully Shodan Maps can do the same for you!

Shodan Maps can be unlocked for a one-time payment of $19! No subscriptions, extra fees or anything like that and if you don’t love it let me know for a full refund.

Analyzing NTP Usage on the Internet with Shodan

NTP has been an interesting protocol for a while, I remember first hearing about the reflection attack 4 years ago when HD Moore unveiled some¬†very clever research¬†in how the “monlist” command could be used for various purposes. The gist is that a small packet can cause the NTP service to return a potentially large list of IPs, using the “monlist” command, that it has recently interacted with. And it looks like the protocol has made waves again recently when people started to actually use the “monlist” command in a reflection attack, effectively creating a DDoS. It’s interesting that it’s taken nearly 4 years to take it from HD Moore’s original research to publicly aware DDoS method. Have the attackers not needed a new method until recently, did their tools lag or have people simply not detected that this has been going on for a while? I’m not sure what the answer is and I’d be interested to hear whether others know more about this.

NTP Meet Shodan

To learn more about NTP and how widely deployed it is across the Internet, I’ve added the NTP service to the list of ports that Shodan surveys. So far, Shodan has identified more than 1 million devices that are running NTP on the Internet.¬†And along with adding the service, I’m also making available a variety of new filters and facets. These are currently only accessible via the Shodan API (and any tools using the new API), but the filters will be rolled out to the main website very soon.

Filters

  • ntp.ip
    Search the initial list of IPs that the monlist command returned.
  • ntp.ip_count
    The number of IPs that the initial monlist command returned.
  • ntp.port
    The ports that the listed IPs used to connect to the NTP server.
  • ntp.more
    true or false – Whether or not there was more data to be read (i.e. more than the initial list of values).

Facets

  • ntp.ip
  • ntp.ip_count
  • ntp.port
  • ntp.more

NTP Is Everywhere

Somewhat unsurprisingly, the amount of NTP servers is fairly uniformly distributed across the globe:

NTP Around the World

The exact breakdown is as follows (facet on country:10):

United States 261924
South Korea 153381
Japan 75690
Russia 48508
China 30100
Germany 21659
France 17482
Great Britain 17027
Canada 15892
Italy 10340

It’s interesting that South Korea has a disproportionate amount of NTP servers, but otherwise I’d say the numbers and global distribution look fine.

Watching the Watchers

An interesting side-effect of the “monlist” command is that doing the survey on the entire Internet reveals patterns within the IP lists themselves. For example, faceting on¬†ntp.ip reveals that there are a few standout IPs that get returned by a lot of NTP servers. These top IPs are most likely scanning the Internet for NTP. The majority of the top IPs have websites setup that indicate they’re doing research on the NTP DDoS reflection/ amplification attack, but the #1 IP that is crawling for NTP servers is from China and doesn’t have any research website available or indicative hostname. The IP is “58.215.177.51” and historically it had an HTTP and FTP server running on it, though they aren’t available anymore (perhaps dynamic IP range). If anybody finds out more about that IP and whether or not it’s another research project, please let me know.

Supersize Me!

The default response packet for the “monlist” command returns up to 6 IPs. If the packet has the “more” flag set to true, then you can listen for more results until you have the entire list of IPs the NTP daemon stores. So how many of these NTP servers are very active and have more than just 6 IPs in their list? Lets first take a look at the general distribution of the number of IPs that are returned by NTP (facet on ntp.ip_count):

ntp-initial-iplist

The absolute numbers of the chart don’t reflect all the data that’s stored in Shodan because the facets are still being processed.¬†And among all of the NTP servers found in Shodan that properly responded to the “monlist” command, this is how many had more than 6 IPs available to return (facet on¬†ntp.more):

ntp-more

Note that for these charts NTP servers that returned errors weren’t included. If the servers with errors were included, then there would be ~13.5% with more data. I.e. out of all NTP servers on the Internet, including ones that didn’t respond properly to the “monlist” command for various reasons, ~13.5% of the devices returned more than 6 IP addresses.

Insider Info

The IPs returned by “monlist” aren’t limited to publicly accessible IP addresses, they can also include information about the local, private network! Below is a chart breaking down how many NTP devices at the moment are exposing 1 or more private IP address:

Private IPs Exposed via NTP

At the time of writing, there are nearly 10,000 NTP servers that list a private IP on the 10.0.0.0/8 netblock in their response, and another ~7,000 devices that list an address in the 192.168.0.0/16 IP range. Of note is also that a few devices (4,345)  even include their loopback IP in the response.

All the data is readily searchable via Shodan, so if you want to keep track of these stats and more just grab a free API key to get started. There’s also an updated Python library¬†that exposes all of the facets, including the new NTP-related ones (stay tuned for more info on the API).¬†There’s a lot more cool stuff that can be done with NTP and it’ll be exciting to see what further research into this area discovers!

Quick Statistics on the Router Backdoor on Port 32764

As soon as the backdoor information was posted on github¬†I got curious about how common the issue would be, so I added port 32764 to the list of ports that I survey. I had set it up and moved on to work on other things, but the topic came up again recently so I thought I’d take a quick look at the data and see what can be learned.

A lot of devices respond on port 32764, but the majority of them are going to be firewalls, random web servers or other shenanigans that have nothing to do with the backdoor. To only get results that mattered, I searched not just for port 32764 but also for the strings “scmm” and “mmcs” (the same value just w/ different endianness). At the time of writing there are 6,401 devices in Shodan that match the criteria for routers with the potential backdoor vulnerability.

Global Backdoor Exposure

Global Backdoor Exposure

It’s interesting that the US doesn’t lead the number of public devices, since in general it’s at the top for online devices. Instead, we see that Great Britain leads the charge with 2,228 devices followed by Italy with 1315. Both of those are a bit surprising and probably indicate a bias in the routers that are purchased/ sold in those regions (more on that later).

Organizational Breakdown

Top Organizations

The largest ISPs in the respective countries (based on market capitalization) are leading the charge in backdoored devices. The organization with the most devices is Telecom Italia, so I took a deeper look into the data in Italy. To do so I wrote a quick Python script to go through the first 1,000 results in Italy, see if they have a HTTP service running as well and if they do I extract the product information from the “realm” section of the header:


from collections import defaultdict
from shodan import Shodan

stats = defaultdict(int)

api = Shodan('YOUR API KEY')

results = api.search('port:32764 country:it mmcs OR scmm', limit=1000)
for result in results['matches']:
 host = api.host(result['ip_str'])

if 80 in host['ports']:
 # Get the port 80 banner
 banner = None
 for tmp in host['data']:
 if tmp['port'] == 80:
 banner = tmp
 break

if not banner:
 continue

# Extract the realm info
 start = banner['data'].find('realm="')
 if start <= 0:
 continue
 start += len('realm="')
 end = banner['data'].find('"', start + 1)
 name = banner['data'][start:end]
 stats[name] += 1

for key,value in stats.items():
 print '%s,%s' % (key, value)

The results of this quick breakdown are below:

Italy Backdoor

I don’t know whether this is a general product bias in Italy or whether Telecom Italia prefers those brands of routers, but Netgear DG834G appears the most popular router in Italy that also has a built-in backdoor. Similar breakdowns can be done for the other countries using a modified version of the above script, which I might do later on (mostly for Great Britain).

Compared to routers that advertise their default credentials (62,883) or old Microsoft IIS servers (5.0 is at 312,334), the total number of routers with a publicly exposed backdoor is very small (6,401). It will be interesting to see whether the service is exposed on other devices in different ways or whether this is the complete picture for the vulnerability’s risk to the Internet. Either way, this is how the data presents the situation at the moment and as things change I will update the information/ charts!