Incident Analysis of Compromised OpenBSD 3.0 Honeypot
Michael Anuzis, July 2002
To make up for this lack of hardware, it seemed like a good idea to use an OS I am very familiar with. It just so happened a few recent vulnerabilities were just made public for the OpenBSD operating system, which just happens to be the UNIX-based operating system I am most familiar with. It seemed like setting up an OpenBSD honeypot was just meant to be. Many honeypots today tend to be Redhat Linux as it has a horrible track record for security. It therefore makes for a good honeypot since the goal is to get hacked. OpenBSD on the other hand is arguably the most secure networking OS available out of the box. It was created from the ground up with security in mind and went over five years without a remote hole in the default install... until now. Two major vulnerabilities have just been released that target OpenBSD. One of them targets the Apache Web Server (not enabled by default) and the other effects OpenSSH, which is enabled by default!
Recently scripts that make it simple to exploit both of these vulnerabilities were released on bugtraq making it very easy for even the most neophyte script kiddie to break into an unpatched OpenBSD installation. Unfortunately, that's just what happened here...
This honeynet is a great example of how few resources need to be used to run a successful honeypot. The layout was very
simple. One system running OpenBSD3.1 in the front acting as a firewall/IDS, and a private network in the back behind NAT.
With web, e-mail, and a whole array of other services running on the gateway, I couldn't afford to bimap my single IP address
to the honeynet to let all traffic pass through, making a more functional model of a honeynet (for those of you unfamiliar
with OpenBSD's firewall, bimapping an IP provides the same functionality as setting a DMZ on a router). Instead, the honeynet was set up via a few simple port
redirections in NAT.
The Pros of this layout:
The rules from the OpenBSD gateway firewall that are relevant to the honeynet are available below with their respective
Note: The first three rules must be placed before your default deny rule (if you have one).
Note 2: dc0 is the external interface
#clean up fragmented and abnormal packets
scrub in all
#honeypot pass shell on 128
pass in log quick on dc0 inet proto tcp from any to any port = 128 flags S keep state
#pass all from outside to honeypot
pass out log quick on dc1 from any to 192.168.3.100/32
pass in log quick on dc1 inet proto udp from 192.168.3.100/32 to any port = 514
rdr on dc0 proto tcp from any to any port 22 -> 192.168.3.100
rdr on dc0 proto tcp from any to any port 128 -> 192.168.3.100
Simple as that!
Four primary methods of logging were used:
Syslogd: Syslogd was set up to use bash keystroke logging, as can be done with this patch from the honeynet project. Unfortunately, the OpenSSH vulnerability runs /bin/sh, not /usr/local/bin/bash, and I was not able to successfully statically bind bash to /bin/sh without greatly sacrificing the functionality of OpenBSD's compiling features. Since I favored the thought of a hacker compiling a rootkit over getting an easy to read log of what commands were run, I left /bin/sh be and thus did not successfully get the recorded keystrokes in that fashion.
Tcpdump: Tcpdump was set running in the background on the gateway on the internal interface. This was done by simply running the following command as root:
# tcpdump -n -i dc1 -s 2500 -w /home/mike/honeypot/day1.dump host 192.168.3.100 &
Where syslog failed, tcpdump prevailed. Every command that was entered by the hackers was caught by tcpdump. It's interesting to note that even though it's the SSH daemon that's hacked, the connection that takes place is not via ssh, and so everything is in clear text. The tcpdump logs are available later in the report.
Snort: Snort was the backbone of the logging. While it didn't catch anything tcpdump didn't catch too, it organized everything in way that made it easy to tell exactly when the honeypot was hacked!
Snort was run with the following command:
# /usr/local/bin/snort -D -d -c /home/mike/hm/snort/snort2.conf -i dc1 -l /home/mike/honeypot
Note: it's possible to run snort on two different interfaces at the same time. Snort was still running
on the external interface (dc0) throughout the entire honeypot lifetime.
If you've got a good eye, you probably noticed the snort above isn't running the default 'snort.conf'.
To catch the traffic on the internal network, I created a whole new snort.conf file called snort2.conf by using the complicated method below:
1. cp snort.conf snort2.conf
2. Edit snort2.conf and add the following line near the beginning: var HONEYPOT 192.168.3.100/32
3. Edit snort2.conf and add the following line at the end: include $RULE_PATH/honeypot.rules
The honeypot.rules file I created and used is available here. Simply put, it logs all TCP traffic as "Honeypot TCP Traffic", it logs all UDP traffic as "Honeypot UDP Traffic", it logs all ICMP traffic as "Honeypot ICMP Traffic", and it catches the GOBBLES SSH vulnerability that was used to compromise my honeypot.
Using Snort and ACID
Snort is the IDS, or Intruder Detection System, which acted as the primary
tool used in knowing when the honeypot was hacked. It was used in conjunction with ACID to make it easy for other members of the DHP to help me keep an eye on the honeypot so we could catch it
as soon as possible after the initial compromise took place. ACID turned out to be extremely useful as it allowed me to jump
in right in the middle of the hack and start analyzing the hackers actions through an easy-to-use web interface.
Here's an interesting shot of hacker #1 trying to restart inetd.
But wait a minute, what do we have here? The same hacker is trying to run a root shell as "httpd" from inetd! It's too bad this does not work, as we can see from this syslog message:
Jul 3 15:21:00 pufferfish inetd: httpd/tcp: unknown service
Silent on the set, and... Action!
It was a nice day. I had just finished a very enjoyable lunch when I decided to check in on the ACID console...
to see that two hackers were attacking my honeypot at the same time!
Imagine, you've already checked your ACID console about 300 times today and seen nothing interesting every single time, it's the first day your first honeypot is up and you're beginning to wonder if it's going to get hacked anytime soon... when all the sudden you see something like this... Look out, it's on now! Who knows what kind of exciting things can be learned from this dynamic duo attacking your honeypot at the same time?!
Well the truth is... not much. The following is a list of the 11 different connections made by the two hackers. The lists were created with Ethereal's TCP Stream Follow function.
Hacker 1 logs in and looks around to see what's running.
Hacker 1 checks open connections and takes a seemingly pointless look around at files on the hard drive and their permissions.
Hacker 2 logs in and looks for tcsh and checks the open connections.
Hacker 2 looks at the fake website and begins to look for the real one.
Hacker 1 looks at inetd.conf, tries to edit it but can't due to terminal type, looks at some files in /var/log, checks the network interface and starts to ping the gateway. After the connection dies the ping continues and it was still hours later after I closed out the hackers and returned home.
Hacker 2 tries to backdoor inetd and checks netstat to see it's not working. He also creates a directory called "nn" in /usr/libexec and tries to use the fetch command.
Hacker 2 does just about nothing.
Hacker 2 does just about nothing.
Hacker 1 tries to retrieve a file called newnick.tar.gz from his site.
Hacker 1 tries wget and sees it's not installed, then tries lynx but sees he can't without a valid terminal type. Why can't he just use scp or FTP and give us something exciting to analyze?
Hacker 1 stops apache, tries to backdoor inetd, and looks for a *get program to get his newnick.tar.gz.
Some of the highlights include:
10 of these: EXPLOIT ssh CRC32 overflow /bin/sh
13 of these: SUCCESSFUL Gobbles OpenSSH challenge-response exploit
14 of these: GOBBLES OpenSSH challenge-response exploit attempt
And 29 of these: ATTACK RESPONSES id check returned root
The reason there were 29 ID checks was because the script they used automatically checked their ID every time they logged in again. The number was about double the number of exploit attempts because it was trigger twice for each time it happened. Once by the snort on the external interface, and again by the snort on the internal interface. There was also one instance where the first hacker manually checked their ID himself.
One of the hackers took particular interest in what the Apache server was used for. It's interesting to note that if he checked the IP he was attacking from his web browser he would see the actual web server running on the gateway, and not the one on the honeypot. Apache was also running on the honeypot for two main reasons:
1. To make the honeypot look like it was really being used for something.
2. To help take the attention off of the fact that syslog was running. This was more influencial when syslog was renamed and running as "screend" and /etc/syslog.conf was renamed to /etc/screen.conf to make it appear as though syslog wasn't running on the system. With 8 apache children and a honeypot user (idle for hours so as to not scare the hackers) running the real screen software making it look as though it's legitimate, hopefully the hacker would let something like 'screend -f /etc/screen.conf' slip their attention in the "ps aux" queries even if he/she did know that screen doesn't run like that!
The hacker's search for information on the website can be found here. It's interesting what length this hacker went to in search of information as to the honeypot's purpose.
Five attempts of binding a root shell in inetd were made:
Unfortunately this didn't work as fetch wasn't installed on the system.
0030: 060b 8e29 2f75 7372 2f73 6269 6e2f 6170 ...)/usr/sbin/ap
0040: 6163 6865 6374 6c20 7374 6f70 3a20 6874 achectl stop: ht
0050: 7470 6420 7374 6f70 7065 640a tpd stopped.
To safely end the events and leave the honeypot in a condition safe for me to analyze it at a later time the following rules were quickly added to /etc/pf.conf:
#people to block
block in log quick on dc0 from HACKER.1.IP/32 to any
block in log quick on dc0 from HACKER.2.IP/32 to any
#pass in log quick on dc0 inet proto tcp from any to any port = 22 flags S keep state
#honeypot pass shell on 128
#pass in log quick on dc0 inet proto tcp from any to any port = 128 flags S keep state
The first two rules were added to block the hackers from visiting the real web server at my IP and following the link currently on the front page pointing to the class I taught on 'OpenBSD and Network Security', giving them a very big hint they were on a honeypot, which I did not want to happen.
The 2nd two rules were previously there and commented out to prevent anyone else from hacking the honeypot so I could get back home later to analyze exactly what happened.
Important Note: Even after adding these rules it's very important to remember that your hackers connections will not be killed instantly as they are established and in the firewall's state table. You must run pfctl -F state to flush out all the state entries and kick their butts out for good (at least from those IPs)!
A few important things we can learn from this honeypot:
It's 1:59:46am EST. A honeynet like this is so simple in nature to set up that it was designed, configured from scratch, hacked, analyzed, and this paper was written, all within a 28 hour period. This is Michael Anuzis, signing off.