GUIDE TO (mostly) HARMLESS HACKING
Vol. 8: Through the looking glass: Finding evidence of your
by Chris Kuethe
University of Alberta
Department of Mathematical Sciences
You've subscribed to Bugtraq and The Happy Hacker list, bought
yourself a copy of _The_Happy_Hacker_, and read _The_Cuckoo's_Egg_
a few times. It's been a very merry Christmas, with the arrival
of a cable modem and a load of cash for you, so you run out and
go shopping to start your own hacker lab. A week or so later,
you notice that one of your machines is being an especially slow
slug (Hi Sherwood!) and you've got no disk space. Guess what
- you got cracked, and now it's time to clean up the mess. The
only way to be sure you get it right is to restore from a clean
backup - usually install media and canonical source - but let's
see what the h4x0r left for us to study.
In late October of this year, we experienced a rash of attacks
on some workstations here at the University of Alberta's Department
of Mathematical Sciences. Most of our faculty machines run RedHat
5.1 (there's a good platform to learn how to try to secure...)
since it's cheap and easy to install. Workstations are
often dual-boot with Windows 95, but we'll be phasing that out
as we get Citrix WinFrame installed. This paper is an analysis
of the compromise of one professor's machine.
One fine day I was informed that we'd just had another break-in,
and it was time for me to show my bosses some magic. But like
a skilled card shark who's forced to use an unmarked deck, my
advantage of being at the console had been tainted. Our cracker
had used a decent rootkit and almost covered her tracks.
In general, a rootkit is a collection of utilities a cracker
will install in order to keep her root access. Things like versions
of ps, ls, passwd, sh, and other fairly essential utilities will
be replaced with versions containing back doors. In this way,
the cracker can control how much evidence she leaves behind.
Ls gets replaced so that the cracker's files don't show up, and
ps is done so that her processes are not displayed either. In
general, after a crack, you can bet something very important
on a sniffer having been installed. Packet sniffers - programs
that record network traffic - are not part of a rootkit per se,
but they are nearly as loved by hackers as a buggered copy of
ls. Who wouldn't want to try snarf other user's passwords?
In nearly all cases, you can trust the copy of ls on the cracked
box to lie like a rug. Don't bet on finding any suspicious files
with it, and don't trust the file sizes or dates it reports;
there's a reason why a rootkit binary is generally bigger than
the real one, but we'll get there in a moment. In order to find
anything interesting, you'll have to use find. Find is a clever
version of 'ls -RalF <w> | grep <x> | grep <y>
| ... | grep <z> '. It has a powerful matching syntax
to allow precise specification of where to look and what to look
for. I wasn't being picky - anything whose name began with a
dot was worth looking at. The command: find / -name ".*"
Sandwiched in the middle of a ton of useless temporary files
and the usual
'.thingrc' files (settings like MS-DOS's .ini) we found '/etc/rc.d/init.d/...'.
Yes, with 3 dots. One dot by itself isn't suspicious, nor are
two. Play around with DOS for about two seconds and you'll see
why: '.' means "this directory" and '..' means "one
directory up." They exist in every directory and are necessary
for the proper operation of the file system. But '...' ? That
has no special reason to exist.
Well, it was getting late, and I was fried after a day of
class and my contacts were drying up, so I listed /etc/rc.d/init.d/
to check for this object. Nada. Just the usual System V / RedHat
5.1 init files. To see who was lying, changed my directory into
/tmp/foo, the echoed the current date into a file called '...'
and tried ls on it. '...' was not found. I'd found the first
rootkit binary: a copy of ls written to not show the name '...'
Now that we knew that '...' was not part of a canonical distribution,
I moved into it and had a look. There were only two files; linsniffer
and tcp.log. I viewed tcp.log with more and made a list of the
staff who would
get some unhappy news. Ps didn't show the sniffer running, but
ps should not be trusted in this case, so I had to check another
We were running in tcsh, an enhanced C-syntax shell which
asynchronous (background) job execution. I typed './linsniffer
&' which told tcsh to run the program called linsniffer in
this directory, and background it. Tcsh said that was job #1,
with process ID 2640. Time for another ps - and no linsniffer.
Well, that wasn't too shocking. Either ps was hacked or linsniffer
changed its name to something else. The kicker: 'ps 2640' reported
that there were no processes available. Good enough. Ps got cracked.
This was the second rootkit binary. Kill the sniffer.
Now we check the obvious: /etc/passwd. There were no strange
entries and all the logins worked. That is, the passwords were
unchanged. In fact the only weird thing was that the file had
been modified earlier in the day. An invocation of last showed
us that 'bomb' had logged in for a short time around 2:35 am.
That time would prove to be significant. Ain't nobody here
but us chickens, and none of us is called bomb...
I went and got my crack-detection disk - a locked floppy with
binaries I trust - and mounted the RedHat CD. I used my clean
ls and found that the real ls was about 28K, while the rootkit
one was over 130K! Would anyone like to explain to me what all
those extra bytes are supposed to be? File has the answer: ELF
32-bit LSB executable, Intel 80386, version 1,
dynamically linked, not stripped. Aha! So when she compiled it,
kiddie forgot to strip the file. That means that gcc left all
its debugging info in the file. Indeed, stripping the program
brings it down to 36K, which is about reasonable for the extra
functionality (hiding certain files) that was added.
Remember how I mentioned that the increased file size is important?
This is where we find out why. First, new "features"
have been added. Second, the
binaries have verbose symbol tables, to aid debugging without
having to include full debug code. And third, many script kiddies
like to compile things with debugging enabled, thinking that
it'll give them more debug-mode backdoors. Certainly our 'kiddie
was naive enough to think so. Her copy of ls had a full symbol
table, and source and was compiled from /home/users/c/chlorine/fileutils-3.13/ls.c
- which is useful info. We can fetch canonical distributions
and compare those against what's installed to get another clue
into what she may have damaged.
I naively headed for the log files, which were, of course,
pure as the driven snow, but I still did find out something useful:
this box seemed to have TCP wrappers installed. OK, those must
have failed somehow since she got in to our system. On RH51,
the TCP wrappers live in /usr/sbin/in.* so what's this in.sockd
doing in /sbin? Being Naughty, that's what. I munged in.sockd
through strings, and found some very interesting strings indeed.
I quote: You are being logged , FUCK OFF , /bin/sh , Password:
, backon . I doubt that this is part of an official RedHat release.
I quickly checked the other TCP wrappers, and found that RedHat's
in.rshd is 11K, and the one on the HD was 200K. OK, 2 bogus wrappers.
It seems that, looking at the file dates, this cracked wrapper
came out the day after RH51 was released. Spooky, huh?
I noticed that these binaries, though dynamically linked,
used libc5, not libc6 which we have. Sure, libc5 exists, but
nothing, and I mean nothing at all uses it. Pure background compatibility
code. After checking the other suspect binaries, they too used
libc5. That's where strings and grep (or a pager) gets used.
Now I'm getting bored of looking by hand, so lets narrow our
search a little using find. Try everything in October of this
year... I doubt our cracker was the patient sort - look at her
mistakes so far - so she probably didn't get on before the beginning
of the month. I don't claim to be a master of the find syntax,
so I did this:
find / -xdev -ls | grep "Oct" | grep -v "19[0-7]"
In English: start from the root, and don't check on other
drives, print out all the file names. Pass this through a grep
which filters everything except for "Oct" and then
another grep to filter out years that I don't care about. Sure,
the 80's produced some good music (Depeche Mode) and good code
UN*X / BSD) but this is not the time to study history.
One of the files reported by the find was /sbin/in.sockd.
Interestingly enough, ps said that there was one unnamed process
with a low (76) process id owned by uid=0, gid=26904. That group
is unknown on campus here - whose is it? And how did this file
get run so early so as to get that low a PID? In.sockd has that
uid/gid pair... funky. It has to get called from the init scripts
since this process appears on startup, with a consistently low
PID. Grepping the rc.sysinit file for in.sockd, the last 2 lines
of the file are this:
#Start Socket Deamon
Yeah, sure... That's not part of the normal install. And Deamon
is spelled wrong. Should a spell checker be included as an crack-detector?
Well, RedHat isn't famous for poor docs and tons of typos, but
it is possible to add words to a dictionary. So our cracker tried
to install a backdoor and tried to disguise it by stuffing it
in with some related programs. This adds credibility to my theory
that our cracker has so far confined her skills to net searches
for premade exploits.
The second daemon that was contaminated was rshd. About 10
times as big as the standard copy, it can't be up to anything
but trouble. What does rsh mean here? RemoteSHell or RootShell?
Your guess is as good as mine.
So far what we've found are compromised versions of ls, ps,
rshd, in.sockd, and the party's just beginning. I suggest that
once you're finished reading this, you do a web search for rootkit
and see how many you can scrounge up and defeat. You have to
know what to look for in order to be able to remove it.
While the log files had been all but wiped clean, the console
still had some
errors printed on it, quite a few after 0235h. One of these was
a refusal to
serve root access to / via nfs at 0246h. That coincided perfectly
with the last access time to the NFS manpage. So our script kiddie
neat, and she tried to mount this computer via NFS, but she didn't
set it up properly. All crackers, I'd say, make mistakes. If
they did everything perfectly we'd never notice them and there
would be no problems. But it's the problems that arise from their
flaws that cause us any amount of grief. So read your manuals.
The more thoroughly you know your system, the more likely you
are to notice abnormalities.
One of the useful things (for stopping a cracker) about NFS
is that if the server goes down, all the NFS clients with directories
still mounted will hang. You'll have to 120-cycle the machine
to get it back. Hmmm. This presents an interesting tool opportunity:
write a script to detect an NFS hack, and if a remote machine
gets in, ifconfig that interface off. Granted, that presents
a possible denial-of-service if authorized users get cut off.
But it's useful if you don't want your workstation getting compromised.
At this point I gave up. I learned what I'd set out to do
- how to find the crap left behind by a cracker. Since the owner
of this system had all her files on (removed) removable media
there was no danger of them being in any way compromised. The
~janedoe directory was mounted off a jaz disk which she took
home at night, so I just dropped the CD into her drive and reinstalled.
This is why you always keep user files on a separate partition,
why you always keep backups and why it's a good plan to write
down where to get the sources for things you downloaded, if you
can't keep the original archives.
Now that we've accumulated enough evidence and we're merely
spirited sluggers pulverizing and equine cadaver, it's time to
consider the appropriate response. Similar to Carolyn's you-can-get-punched
and you-can-go-to-jail warnings, I would suggest that a vicious
hack back is not appropriate. In Canada, the RCMP does actually
have their collective head
out of the sand. I am not a lawyer, so don't do anything based
on these words except find a lawyer of your own. With that out
of the way, suffice it to say that we're big on property protection
here. Aside from finding a lawyer of your own, my advice here
is for you to call the national police, whoever they are. People
like the RCMP, FBI, BKA and KGB probably don't mind a friendly
phone call, especially if you're calling to see how you can become
a better law-abiding citizen. Chances are, you'll get some really
good tips, or at least some handy references. And of course you'll
know someone who'll help you prosecute.
My communication with RCMP's Commercial Crimes unit (that
includes theft of computing and/or network services) can be summarized
as follows: E-mail has no expectation of privacy. You wish email
was a secret, but wake up and realize that it's riskier than
a postcard. As a sysadmin, you can do _ANYTHING_ you want with
your computer - since it's your responsibility either because
you own it or because you are its appointed custodian - so long
as you warn the users first. So I can monitor each and every
byte all of my users send or receive, since they've been warned
verbally, electronically and in writing, of my intent to do so.
My browse of the FBI's website shows similar things. But that
was only browsing. Don't run afoul of provincial or state laws
regulating the interception of electronic communication either.
NOTE: While I have attempted to make this reconstruction of
events as accurate as possible, there's always a chance
I might have misread a log entry, or have misinterpreted
something. Further, this article is solely my opinion,
and should not be read as the official position of my employer.
Appendix A: Programs you want to have in a crack-detection
find, ps, ls, cp, rm, mv
gdb, nm, strings, file, strip
(gnu)tar, gzip, grep
less / more
vi / pico
tcsh / bash / csh / sh
These should all be statistically linked. Items separated
by commas, I mean "all of these," items separated
by forward slashes mean "pick your favorite."
Appendix B: References
About the filesystem
McKusik, M.K. , Joy, W.N. , Leffler, S.J. , Fabry, R.S.
"A Fast File System for UNIX" Unix System Manager's
Computer Systems Reseach Group, Berkeley. SMM-14. April
LEA and Computer Crime
Chris Kuethe: System Administrator - U of A Math Dept
office: CAB553, x1704 cell: 903.9475
This is a list devoted to *legal* hacking! If you plan to
information in this Digest or at our Web site to commit crime,
Foo on you! Happy Hacker is a 501 (c) (3) tax deductible organization
in the United States operating under Shepherd's Fold Ministries.
This is all a plot to save your immortal souls!
Happy Hacker Grand Pooh-bah: Carolyn Meinel