How do you audit your Linux environment? How do you track after changes to your files? What kind of processes are running on your system at any given moment? What uses the most resources? Valid questions, all. Special contributor Dedoimedo gives us the straight scoop on “audit.”. Editor’s note: Call for submissions: are you an OS expert? Can you provide some special insight, some tips and tricks, or just plain illuminate an obscure feature in your OS of choice? We’d like to publish it.
We’ve monitored system activity with sar. We used iostat, vmstat and dstat to collect statistics
on resource usage. We learned about pacct for process accounting. We also worked with top in
batch mode and atop. Finally, we worked with some high-end debugging tools like OProfile and
strace. Links below. But we did not use any utility for file auditing.
Today, we do that. The name of the game: Linux audit.
What Linux audit be?
Linux audit is a built-in mechanism in the kernel, which allows you to intercept changes to
monitored files and write them to a log on the disk. The mechanism consists of three major
components: auditd, the audit daemon (background service),
auditctl, the program used to control the auditing system, and
ausearch, a tool to query audit daemon logs.
To get audit configured, you will need to install the audit (or auditd) package, if it’s not
already installed. Once installed, you will probably want to setup audit to run on startup,
using a system command-line tool like chkconfig, for instance.
You can also manually control the service via /etc/init.d.
Why not use version control?
That’s a good question. The big difference between audit and version control is that the
first is mandatory. Version control works only if people want to cooperate. Let’s see an
example.
Take subversion (svn), for instance. You can have all kinds of files and directories under
version control. Indeed, every time you update your files from the central repository, you
will override any local changes done to your files. While svn blame will tell you about file
edits against the subversion server, it won’t tell you anything about local changes.
Furthermore, this requires another layer of setup.
An alternative is configuration management, like cfengine. This could work, too. You will
have a static baseline to revert to, deleting any unwanted changes to your files. However,
you will not know, in between period runs, who made changes to your files – or why.
Both version control and configuration management are excellent resources, but they rely on
good will. They work extremely well in an environment where you have cooperative users
working toward a common goal. If you merely want to police your files against all and any
desired and undesired changes, then you want auditing.
So auditing it shall be.
Audit in action (auditctl)
Let’s see how this thing works. First, let’s make sure the service is running:
Now, we can start working on collecting audits. You can configure auditing rules on the fly
via auditctl, plus you can set some defaults using the configuration files.
/etc/audit/auditd.conf is used to configure log file location,
size and flushing (rotation), and various actions like mail and disk space events.
/etc/audit/audit.rules lists auditing rules that are loaded on
the service start, restart or reload.
But we will start with using auditctl from the command line. You will need root or sudo to be
able to conduct system-wide auditing. Let’s try an example, which is pretty much what you
will find in the man page. We will configure a watch using auditctl.
Or in text, if you can’t see the image:
auditctl -w /etc/hosts -p war -k hosts-file
What do we have here?
-w flag specifies the file to be watched.
-p sets the permission filter; in other words, what activityyou want to
audit. You have four options, r = read, w = write, x = execute, a = attribute change, which
could be owner, permissions or one of the special file attributes, depending on the
filesystem. -k is a filter, which you can later use to search
through your records in a more meaningful manner. Call it a tag if you will.
Other useful flags
There are many more flags for auditctl. For instance:
-W will remove the watch on a specified file.
-S allows you to trace specific system calls. You can specify
them individually or use all for all system calls. This is somewhat similar to -e flag for
strace, as we’ve seen in the separate tutorial.
Flags -a and -A allow you to add
rules to the end or the beginning or the rules list, respectively. Now, -d will delete a specific rule. -D will
delete all rules.
Adding rules
Adding rules uses a list,action pair, where you first specify
thelist entry, which could be one of the following: task, which is generated when a process
is created, entry, when you enter a system call, exit when you exit a system call, and so
forth. Action can be never and always.
We will see more examples later on. Now, we have a watch in place, let’s edit our /etc/hosts
file and then see what ausearch gives us.
ausearch
ausearch can be run any time you want, while auditd is running. The output produced is
extremely detailed and should not be read at face value. You may probably want to parse the
information, to grab only whatever interests you.
The basic usage is:
ausearch <filename>
But this will produce an ugly output. You can use the -i flag to make it more reasonable to
read; even so, you will have a lot of data to process.
What do we have here?
type tells us what kind of record we have. For file watch
rules, it will normally be PATH. For system calls, it will be SYSCALL.
msg is the record timestamp. name tells us
the filename for files. You also have file type, permissions, owner, group, device major and
minor numbers, as well as the inode number.
For system calls, it’s a little more complicated. syscall tells
us what system call was, uh, well, called. success tells us if
it worked. You also get the process id (pid) and the parent process id (ppid). There’s a long
set of real and effective user and group ids. You also have the console number.
exe specifies the command that was run. And so forth. Detailed and only
useful for experienced administrators knowing what they need.
This is very similar to going under /proc and burrowing into data for specific processes, but
the thing is, it’s done automatically and faster than a human could possibly log information.
Other useful flags
-k will search by specific keys (tags) only. Otherwise, why did
we use them in the first place? -m will filter only specific
types of records. -sc will match only specific system calls.
-p will only match specific processes.
You can also redirect your output into text editors or files.
More examples
There are quite a few examples in the man page. Let me be a monkey and repeat some of them
here, for the sake of education.
To see files opened by a specific user:
auditctl -a exit,always -S open -F auid=510
We will always monitor the exit from the open system call for the audit uid (auid) 510, which
might correspond to some user, like roger. auid is the original ID the user logged in, so if
he/she changes his effective UID while working, it won’t be masked in the log. We want to
monitor the exit from the system call, because if it fails, we will know the exact error.
Another example is:
auditctl -a exit,never -S mount
It says, don’t log any mount system calls.
Here, rather than collecting as much as we can, we suppress output. This could be useful if
you know you might have too many logs or the information is just not interesting. For
example, in the NFS environment, you may not want to do any auditing on the client side and
will instead focus on doing it on the server side. Besides, in this case, there are other
tools you may want to use. For example, pacct, access control via access.conf, traffic
logging, export options for your NFS areas, and more.
One more for ausearch:
ausearch -ts today -k hosts-file -x vi
-ts tells us the timestamp we want to search through our logs; today, only. -k narrows down
our search to the hosts-file key only. -x specifies the exact name of the executable run
against the watched file. In this regard, our example looks for any attempts to open or
modify the file with vi text editor today.
That’s enough, I think.
More stuff
There are more helper tools that can help you work with auditing. One such tool is
aureport, which you can use to generate summary reports of
collected logs. This is somewhat similar to -c flag in strace.
More reading
A handful of my system administration oriented articles.
10 super-cool Linux hacks
you did not know about
Linux super-duper admin tools:
lsof
Linux super-duper admin tools:
screen
Linux super-duper admin tools:
OProfile
Linux super-duper admin tools:
Strace
Some external links:
Linux
audit files to see who made changes to a file
Finally, man pages:
And that would be all folks!
Conclusion
You have just gained another powerful tool that you can use to make your Linux environment
smarter and better controlled. Alongside an ever-growing repertoire of powerful utilities,
audit will help you build your Linux street credit and make your work easier. audit is very
simple to use, although processing the collected logs can take time and effort. You need to
master all the little flags and understand how the system works before you can delve deeper.
If you have any useful tools you considered using but didn’t dare try, feel free to ping me
and I will testdrive them for you, trying to provide you with the best and most modular
advice while striving for strategic imperatives rather than tactical keyboard hammering.
Cheers.
About the author:
Igor Ljubuncic aka Dedoimedo is the guy behind dedoimedo.com. He makes a living out of his
very hobby – Linux, and holds a bunch of certifications that make a nice pile in the bottom
drawer.
great article!
I’m sure this will come in handy.
Definitely bookmarking this one.
Finally, an “OS” article on OSnews worthy of bookmarking it. We need more of these kind of articles.
We sure do. I’ve got another great one lined up about NTFS from someone, great stuff. Very long, so it’ll be a bit of work to get ready, but it’s great.
Maybe it’s just me being bloody minded, but why should I care if someone made a local change to a file managed by my configuration management system and the change gets over written? That’s the entire point of configuration management such as cfengine or Puppet. The configuration management system is canonical. If someone attempts to make a local change outside of configuration management:
a) They’re Doing It Wrong and therefore shouldn’t be making such a change anyway.
b) I want their changes to be overwritten due to the above.
Audit tools such as tripwire and audit are useful for finding potentially malicious changes to key system files, but I don’t see why you’d try to use something like this as a replacement for something like Puppet. It’s Apples and Oranges.
Yep. cfengine/puppet and audit/tripwire complements eachother, they’re not mutually exclusive.
configuration management does many things audit doesn’t, like keeping a history of changes so that you can always rollback to something that worked. cfg management also tells you exactly what change was done, not just that something was done.
Edited 2010-11-15 23:35 UTC
This is indeed a fine article… first time I ever voted an article up here. I don’t use Linux at all but I read the thing. Clear, focused, to the point. Well done!
Now I’m eagerly waiting for the NTFS article.
The problem with Linux auditd, if there can be said to be one, is that it is different and (IMO) needlessly different from the Solaris/FreeBSD auditd. Could have been the same but isn’t. Much like pam differences, but with less need, this just makes switching between systems a little more annoying and frustrating.