Why Control Systems Are On the Internet

A few weeks ago I asked people on Twitter what sort of new ports/ services they’d like me to add to Shodan. I received a lot of awesome feedback which resulted in Shodan now crawling for more than 170 ports (!!!). One of those requests was for the FINS protocol created by Omron:

omronrequest

By the way, I’m always looking to add more ports to Shodan so if there’s something you’d like to see me add just let me know which port and protocol you’re interested in! Anyways, I did some quick Google-ing to learn more about this FINS protocol and I stumbled across the following advice in the official Omron documentation (PDF):

Just to reiterate: they’re saying that because their device (Omron PLC) isn’t a Windows-based operating system that makes it impenetrable to the standard hacking methods. And I’m not sure what they mean with the following sentence about not responding to “standard ethernet protocol commands”, since the FINS protocol in this case operates over UDP and/ or TCP. Either way, this is a good example of why many control systems can be found on the Internet. This document is a few years old now (released in 2009) so Omron as a company might’ve improved their stance on Internet-security, but control systems are a slow-moving world and this sort of mentality has lingered around for a long time.

So what about the initial request to add Omron FINS to Shodan? After reviewing the pcaps for Wireshark and trying to find a simulator, I hit a road block and stopped making progress. Fortunately, Stephen Hilt picked it up as a challenge and within a few days was able to create fully-working Nmap scripts for both TCP and UDP versions of the Omron FINS protocol. If you’re interested in doing ICS analysis with Nmap, that should be your goto location for getting started. Thanks to Stephen’s work, I was able to convert the NSEs into Python scripts for my crawler and it’s now possible to find Omron FINS devices on the Internet via Shodan:

https://www.shodan.io/search?query=port%3A9600+response+code

omron

The data is still flowing in so the results are on the lower-bound at the moment, but it’s been added to the list of services that Shodan permanently crawls for to keep track of how the exposure of these devices changes over time.

PS: If there is a port/ protocol that you’d like to see in Shodan please email me the information to jmath@shodan.io 🙂

Telnet is dead. Long live Telnet!

Telnet just doesn’t seem to die, and with millions of devices on the Internet still running it I don’t expect it to go away anytime soon. In fact, new products often use Telnet as an easy way to enable remote access to their device (see https://github.com/swisspol/GCDTelnetServer). To better keep track of which features the Telnet servers of the Internet support, the banners for Telnet now also include information about which options were negotiated during the initial connection:

telnet

In addition to the banner, you can now access the dodontwillwont options that the Telnet server supports. You’ll also be able to facet on these properties once the next API update comes out, which will make it easy to get a feel for how Telnet usage changes over time. Hopefully, more secure options will become easier to work with for the average developer and we’ll see fewer Telnet in the coming years.

Mapping Uncharted Territory

Shodan collects a ton of data (1+ billion banners/ month) and it can be difficult to find the needle in the haystack. Sometimes the banner itself doesn’t tell you much about the device, for example a search for wind farms by Nordex is “jetty 2000”. Not exactly what you’d think when trying to locate wind farms. So most people end up searching Shodan using queries shared by other users and don’t really get to experience the fun in discovering new types of devices.

shodan-explorer

To help browse through the data that Shodan gathers I wrote a simple webapp called Shodan Explorer:

http://explorer.shodanhq.com/#/explore

It randomly pulls an IP out of the stream of data that Shodan gathers, checks that the IP runs a webserver (HTTP) and then embeds the IP in an iframe so you can see it in your browser. Explorer is a simple, quick and hopefully fun way to check out all the weird stuff that’s on the Internet. If you end up finding something new please share the search result (Twitter, Blog, Shodan Search Directory, etc.) so we can start classifying all the unknown types of devices that are being crawled.

Note: the website has to be hosted on HTTP and not HTTPS since browsers don’t let you load mixed content on a HTTPS connection.

Keeping Track of How You Manage Your Server

A lot of people prefer to avoid the terminal and instead want a user-friendly web interface to manage their server. So to determine which solution is the most popular Shodan has started to crawl the Internet for cPanel (2082, 2083), WHM (2086, 2087) and Webmin (10000)! The banners are all fairly standard HTTP(S) responses so I won’t show those, but lets take a look at who runs what and how they compare.

cPanel

Reporthttps://www.shodan.io/report/9wDbB4lo

The majority of devices are located in the United States (117,000) followed by Canada (9,000) and the United Kingdom (7,000). So cPanel is definitely way more popular in the US than anywhere else. And it’s especially popular at the hosting provider Unified Layer, which is responsible for 21,000 of those US installations.

cPanel Report   Shodan

Webmin

Report: https://www.shodan.io/report/asclj6Dy

There are fewer Webmin instances on the Internet than cPanel, but the distribution of them is wider across the globe; i.e. it’s not as US-centric as cPanel. The US still leads with 38,000 instances, but not that much more compared to the following countries France (11,400) and Germany (11,000). In terms of hosting providers though there is once again a standout: OVH. Their organization accounts for roughly 10,000 of the installed Webmin instances.

Webmin Report   Shodan

Web Host Manager (WHM)

Report: https://www.shodan.io/report/bYB41Q3f

WHM is a bit different in that it is the software to manage cPanel accounts. This effectively gives us a measurement of which devices are being used by re-sellers. Unsurprisingly, the distribution of devices in terms of countries and organizations is very similar to cPanel. US with 97,000 devices, followed by Canada (9,000) and the UK (7,000) once again.

WHM Report   Shodan

Call for Ports

Am I missing some ports or services? Is there something you’d like to see Shodan crawl? Then let me know and send me an email (jmath@shodan.io) or tweet at me (@achillean) with the ports that you’d like me to add!

SSH Revamp

SSH was one of the first protocols that I started crawling for 5 years ago because just connecting to the daemon already tells you what it’s running. I.e. you don’t have to send any data to SSH in order to get something interesting back. There have been some incremental improvements to add product and version detection but beyond that it’s stayed mostly the same.

Introducing the new, sexier SSH banner:

banner

 

The crawlers now collect the key, key type, fingerprint, MAC and cipher used for each successful SSH connection! And alongside these changes the API has also been enhanced with 4 new facets for SSH:

  • ssh.cipherssh_cipher
  • ssh.fingerprint
  • ssh.macssh_mac
  • ssh.typessh_type

I hope you enjoy the new banner and information that’s being gathered for SSH now! Let me know if there are other banners you’d like to see improved as well.

Shodan Add-on for Firefox

It’s now possible to see what information Shodan has available on a server from within Firefox thanks to the new Shodan add-on created by @PaulWebSec and @romainletendart! It’s a minimalistic yet powerful add-on to see what the website you’re visiting is exposing to the Internet. And the add-on will also tell you other information about the IP, for example who owns the IP space (organization), where it’s located and if possible the operating system it’s running. You can download the add-on from here:

https://addons.mozilla.org/en-US/firefox/addon/shodan-firefox-addon/

And once you have it installed, you just need to click on the small Shodan icon to get an overview of what services that server is running on the Internet:

146342

And what’s really cool is that the authors of the add-on made the code available at the following GitHub repository:

https://github.com/PaulSec/Shodan-Firefox-Addon/

So if you have ideas, feedback or other suggestions on making it better just submit an issue or submit a pull request! And once again a big thank you to @PaulWebSec and @romainletendart for creating and sharing the add-on!

Using Shodan from the Command-Line

Have you ever needed to write a quick script to download data from Shodan? Or setup a cronjob to check what Shodan found on your network recently? How about getting a list of IPs out of the Shodan API? For the times where you’d like to have easy script-friendly access to Shodan there’s now a new command-line tool appropriately called shodan.

Shodan CLI Usage

Installation

The shodan command-line interface (CLI) is packaged with the official Python library for Shodan, which means if you’re running the latest version of the library you already have access to the CLI. To install the new tool in Linux simply execute:

easy_install shodan

Or if you’re running an older version of the Shodan Python library and want to upgrade:

easy_install -U shodan

Once the tool is installed, you have to initialize the environment with your API key using shodan init:

shodan init YOUR_API_KEY

Usage

At the moment, the shodan CLI supports 6 commands. Note that for each command you can learn more about the options it supports by supplying the –help flag.

1. count

Returns the number of results for a search query.

$ shodan count microsoft iis 6.0
5310594

2. download

Search Shodan and download the results into a file where each line is a banner serialized in JSON as specified in https://developer.shodan.io/api/banner-specification

By default it will only download 1,000 results, if you want to download more look at the –limit flag.

For example, to download the latest 1,000 Microsoft-IIS 6.0 servers indexed by Shodan into a file called microsoft-data.json.gz use the following command:

download

This is the command that you should be using the most, since it lets you save your results and process them afterwards using the parse command. Because paging through results uses query credits, it makes sense to always store searches that you’re doing so you won’t need to use query credits for a search you already did in the past.

3. init

Initialize the shodan CLI. This is the first command you have to run for the tool to work, if you’re unsure about how to install the CLI please read the section above on installation.

shodan init YOUR_API_KEY

4. myip

Returns your Internet-facing IP address.

$ shodan myip
199.30.49.210

5. parse

Use parse to analyze a file that was generated using the download command. It lets you filter out the fields that you’re interested in, convert the JSON to a CSV and is friendly for pipe-ing to other scripts. For example, here’s the command to output the IP address, port and organization in CSV:

$ shodan parse --fields ip_str,port,org --separator , microsoft-data.json.gz

And this is what the output in your terminal would look like:
parse

6. search

This command lets you search Shodan and view the results in a terminal-friendly way. By default it will display the IP, port, hostnames and data. You can use the –fields parameter to print whichever banner fields you’re interested in. For example, to search for Microsoft IIS 6.0 devices and print out their IP, port, organization and hostnames use the following command:

$ shodan search --fields ip_str,port,org,hostnames microsoft iis 6.0

search

Final Thoughts

I hope you’ve gotten a good idea of what the shodan CLI can do and how it might make your life easier. The tool is still in its early stages but it’s served me well so far. At this point, I’d love to get some feedback on things you’d like to see improved by submitting issues on the GitHub repository:

https://github.com/achillean/shodan-python

Coincidentally, that’s also the place where you can see the code for the tool in case you’re curious about the inner-workings. Please submit ideas for improvements and let me know via email (jmath@shodan.io) or Twitter whether this is useful to you!

Introducing Shodan Reports

As some of you have already seen, I’ve been working on a revamped version of Shodan. It already has some cool new enhancements over the current shodanhq.com website, such as support for CSV and JSON exports, prettier search listing, faster results, better integration with Shodan Exploits/ Maps and a lot of small tweaks to make life easier.

The latest feature that I’m adding is Shodan Reports:

Shodan

To get a feel for what I’m talking about, please check out the following example report on Industrial Control Systems:

https://www.shodan.io/report/l7VjfVKc

A report is a snapshot and overview of the search results at the time of report creation. At the moment, it creates a bunch of charts/ graphs for breakdowns on: location, organization, operating system, product, hostname and many more (see the developer documentation for a list of all facets). The purpose for reports is 3-fold:

  1. Pretty Overview That You Can Share
    You should be able to get a basic feel for the devices in the search results just from looking at the report. And if you’re interested in the details, you can perform a live search to get a listing of actual results. Reports are meant to be accessible!
  2. Tracking Results Over Time
    As mentioned earlier, reports are snapshots of the search results as Shodan sees them at the moment. You can create reports every few months to see how things are changing over time (this might actually be built into Shodan Reports as well)
  3. Bookmarks!
    The old shodanhq.com website has bookmarks (there’s a small star icon at the top of the search results) but it was rarely used and didn’t offer much that browser bookmarks didn’t. I’m hoping that these reports will provide a prettier bookmarking mechanism that’s also more useful!

Alright, you’re sold on reports and want to give it a shot yourself – here is how.

  1. Login to Shodan and perform a search. For example: Webcams
  2. Click on “Create Report” and give your report a title such as “Webcams of the World”Server  SQ WEBCAM   Shodan Search
  3. You will be redirected to the page that will list all your reportsShodan-report-start
  4. Now just wait a few minutes for Shodan to generate your report, you will receive an email when it’s done! Once it’s ready, you can follow the link and you should see something like:Shodan-sample
  5. And you can always find a list of all the reports you’ve generated by clicking on the chart icon in the top right cornerShodan-list

I created it because I needed a friendlier way to share search results and I got tired of manually generating my charts for blog posts 🙂 It’s very simple and straight-forward to create reports so give it a try and let me know what you think!

Kicking the Shodan API Up a Notch

Most people think of Shodan as a search engine of banners, and while that was true for a long time it’s slowly grown into collecting more information than just banners. Here are just a few of the things Shodan does nowadays:

  • Checks for heartbleed on all SSL services
  • Gets a list of peers from a Bitcoin server
  • Determines whether a DNS server allows recursive lookups
  • Gets a listing of all the MongoDB databases
  • Grabs the SSL certificate for all SSL services
  • Collects the robots.txt for HTTP services

And that’s just the tip of the iceberg, there is a lot of other protocol-specific data gathered for each service. But making all of that information available through the web interface can be a bit tricky and takes time to get right. Instead of sitting on that information though, I decided to make all of the data available through a new and improved Shodan API. Note that a new web interface is being developed to expose all the cool new data, but if you don’t want to wait you can get your hands on it right now with the API.

Along with access to more detailed data, the new Shodan REST API also provides greater flexibility so you can get a top listing of the stuff you care about. For example, the old API would always return the top 5 countries and cities for your search query. What if you didn’t want those? What if you’d prefer to get a breakdown of the top organizations? Or the uptime distribution? Or you don’t care about the actual results and only want the top 100 cities that match a search query? All of those things are now possible with the new API. And if all you want are the actual results, then searching the new API will be much faster than before.

For most people, the Developer API plan that costs a one-time fee of $9 is enough to run their scripts each month and get what is needed out of Shodan. In case you need more access though, there are 3 new API plans to help you out:

  • Freelancer: $19/ month
  • Small Business: $99/ month
  • Enterprise: $499/ month

Each of them provides increased levels of access, from 1 million results/ month for Freelancer up to completely unlimited access to the REST API at the Enterprise plan.

In addition to the changes to the REST API, I’m also introducing the Shodan Streaming API! It’s basically a real-time firehose of all the data that Shodan is collecting. You can either subscribe to all the data, ask the API to filter the banners based on a port (ex. stream only MySQL and PostgreSQL banners) or only get the geographic information for a banner (for geo visualizations). There are 2 caveats: you have to be on a subscription API plan and it currently only streams ~1% of the data. If you or your company would like 100% of the data please contact me for pricing information. Those notes aside, it still provides quite a bit of data in real-time and I’m hoping it’ll provide a useful way to do cool stuff with Shodan. Here is some basic example code to get you started using the Python library: https://gist.github.com/achillean/f953cb917a7eb2a9f81d

The Shodan API will be a major focus for the next year so watch the documentation for new filters and facets that will be released soon 🙂

 

Shodan Plugin for Chrome

I’ve just released a new plugin for Chrome browser that lets you see what data Shodan has available for the website you’re currently visiting. This information includes the IP, hostnames (as determined via PTR lookups), location of the server (city, country), operating system, organization that owns the IP space as well as a list of open ports. As soon as you visit a website, it automatically asks the Shodan API (hence the api.shodan.io permission requirement) for all that information and then compiles it into a small pop-up that you can view when clicking on the Shodan icon next to the address bar:

shodan-chrome

The pop-up for the Shodan plugin is as simple as possible, which also means it doesn’t show the full banners for every port. There wasn’t enough room to fit everything in, especially for some of the hosts that have tons of ports open. I’m looking to find a way to show all of that information and more in future releases! For now, I’m happy to finally have a way to quickly see who’s hosting a website (the hostname often gives that away) and if they’re running some unusual services publicly. Oh, and if you see a dark blue square that means it can be opened by your browser and will open up a new tab with the IP:port if you click on it (in case you want to see what it shows when you access the webserver via its IP instead of hostname).

Go download the plugin and let me know what you think!

https://chrome.google.com/webstore/detail/shodan/jjalcfnidlmpjhdfepjhjbhnhkbgleap