Tag: papertrail

Find Shellshock Exploit Attempts & Probes From the Command Line and Papertrail

Never hurts to make sure

I’ve written about Papertrail a few times before, I love the service and it’s just too valuable to not use.

Papertrail makes it super easy to find Shellshock exploit attempts and probes. Probes are just checking a machine to see if it’s vulnerable to Shellshock. If you’re using CloudFlare, you’ll never see any Shellshock attempts show up in your logs, CloudFlare doesn’t even let them through.

See If Shellshock Affects You

Checking to see if your system is vulnerable to Shellshock is quite easy. It takes a relatively simple bash command:

env x='() { :;}; echo vulnerable to shellshock' bash -c "echo All good"

Run that code in a terminal. If you see All good, you’re not vulnerable. However, if you see vulnerable to shellshock, you are potentially vulnerable.

Yahoo-WinZip-Servers-Shellshock-Bug

Shellshocker.net provides a script that will download, compile, and install the newest version of bash for you. You should only use it though if your Linux distribution hasn’t already provided updated security release packages. If you’re interested, the code that runs Shellshocker.net is available on GitHub.

Find Shellshock Attemps and Probes Via The Command Line

This is very easy as long as you know the location of your Apache access log file. It’s typically something like /var/log/apache2/access.log. Assuming that’s the location of your Apache access log file, this command will pull out all the Shellshock probes and attempts:

grep '() {' /var/log/apache2/access.log

If nothing was returned, that means nobody has been trying to exploit Shellshock on your system, or even checking to see if your system is susceptible to Shellshock. If results are returned, look them over carefully to examine where the attempts are coming from, an IP address will be associated with every attempt.

Shellshocker.net Checker

Shellshocker.net also provides a bash script to check your machines for the Shellshock vulnerability. You can download the script and run it manually from your terminal, or, if you have cURL installed, run the following command:

curl https://shellshocker.net/shellshock_test.sh | bash

Running that command will produce results similar to the screenshot seen below. It checks for a number of Shellshock related vulnerabilities.
shellshocker

Find Shellshock Attemps and Probes With Papertrail

Go to your Papertrail events tab and search for the following:
"() {"

If anything is returned, those are Shellshock probes. Some example probes are listed in the gist that’s embedded below. None of the offending IP addresses have been redacted.

These actually made it through to Papertrail, which shouldn’t happen since longren.io sits behind Cloudflare. I’ll open a support ticket with them about it and update this post later.

0

Send Apache Logs to Papertrail With Rsyslog

Over the last few days, I’ve been looking at Apache web server logs, a lot, mostly quick checks for Shellshock probes and exploit attempts. All on client servers, thankfully. All of the servers I operate through DigitalOcean are patched up. It just so happens that all the sites I host have their DNS hosted by Cloudflare, which has been blocking all Shellshock attempts.

A majority of my sites send their Apache logs to Papertrail. Having all my apache logs easily accessible and searchable is extremely nice. It’d make sniffing out Shellshock attempts quite simple. You can check for Shellshock attempts relatively easily from the command line, as well, something like the command below would work:

grep '() {' /var/log/apache2/access.log

1. Setup Rsyslog to Send to Papertrail

Anyway, sending Apache logs to Papertrail is pretty easy. I’m going to assume you’ve already setup rsyslog to send logs to Papertrail. If not, this post should help.

2. Add CustomLog Directive To Your VirtualHost

You just need to modify your virtualhost configuration and add a CustomLog directive. Here’s what I do to send longren.io logs to Papertrail:

CustomLog "| /usr/bin/logger -t httpd -p local1.info" "%{%b %d %X}t longren.io apache %h %l %u %t "%r"%>s %b "%{Referer}i" "%{User-agent}i""

The -t httpd piece sets the service name for Papertrail. The -p local1.info flag sets the priority. You’ll want to change the longren.io piece in the above code to whatever site you’re capturing logs for. You can also change or remove apache that immediately follows longren.io.

3. Reload Apache

After you’ve added the CustomLog directive to your virtualhost, you’ll want to reload Apache:

sudo service apache reload

That’s all there is to it. You should start seeing your Apache logs in Papertrail shortly after reloading Apache.

0

Monitor SSH Activity on Your Server with Papertrail & Saved Searches

Search for common phrases related to SSH logins

Papertrail is good for all types of things, I especially enjoy it because it makes finding problems quite easy. Another fun thing to do with Papertrail is see how many people are trying to either login to your server via ssh, or attack sshd itself.

Find Potential Break-In Attempts

You can save searches in Papertrail, which makes finding certain events extremely easy. When I want to see potential break-in attempts, I have a saved search for not listed in AllowUsers. My server names have been blacked out in the image below.
papertrail-not-listed

Show All sshd Associated Event Logs

To see all events associated with sshd, you can save a search for program:sshd. This will show all log entries generated by sshd. Again, server names have been censored.
papertrail-program-sshd

Papertrail has groups that you can put your servers into. Saved searches can be applied to a specific group of servers, or all servers. It’s really quite nice, especially if you have a server that almost nobody ever logs in to. Papertrail also lets you know the last time a log message was received from a server. For most servers, the last log event will have been moments ago.

Receive Alerts for Saved Searches

One of the best features are the alerts on saved searches. If you have a server that nobody but you logs into but you, you can setup an alert for when someone logs in via SSH. Most of the time the alert will just be you logging in, but this is excellent information to have if someone else does manage to login. Someone could have obtained your private key or your password.

You can specify the minimum number of events needed before the alert is activated. I’ve got an alert setup that sends me an email after my public key has been accepted 20 times. The alert is sent once an hour. I have the minimum number of events so high because I login frequently with FileZilla and make small edits to files, which causes a lot of SSH logins.
papertrail-saved-search-alert

If you have to manage any servers at all, you might as well give Papertrail a try. They have a free plan that lasts forever. If you find that you need more than 100MB of log storage space, you can get 1GB for $5 per month, which is plenty for most folks. $5 only includes 1 day of log retention, $7 per month gets you a week of log retention.

0

Receive Alerts On SSH or SFTP Logins with Papertrail

Frustration-free log management, plus a lot more

I’ve been a huge fan of Papertrail ever since I discovered it, probably about a year ago or so. I use it mostly to monitor server logs. I currently have two servers setup to send syslog messages to Papertrail.

The Papertrail Events dashboard can be a bit overwhelming at first, but the provided search is powerful and allows you to finely control which log messages you see and which you don’t.

You can even setup saved searches to fire when a specific event occurs. For example, I have a saved search that searches for the following:
Accepted publickey for tyler

When that message shows up in Papertrail, it means that I logged in, or that someone else has logged in using my SSH key. This can be quite handy, especially if you’re a one man shop like me and are usually the only person that has SSH or SFTP access to a server.

Getting a DigitalOcean VPS added to Papertrail, especially if it’s running Debian or Ubuntu, is super easy. It just requires that you modify /etc/rsyslog.conf and add a line to the end of the file that will send a copy of the system logs to Papertrail.

Papertrail can monitor application logs, too, such as Apache httpd logs and MySQL server logs, although that takes a bit more configuration to get working properly.

If nothing else, it’s just nice having system logs aggregated in one central place, where everything is easy to search through, making it easy to find exactly what you’re looking for. If you’re an admin for one server or hundreds of servers, Papertrail could turn out to be one of your favorite tools. It’s definitely one of my favorites.

I suggest you give Papertrail a try, can’t hurt, they even have a plan that’s free forever. It’s definitely a great service for monitoring server logs.

0

Aggregate System and Application Logs with Papertrail

Frustration-free log management

I’ve been using Papertrail for a few months now, and absolutely love it. Being able to search logs across all my servers at once is crazy nice.

I can even get alerts when someone logs in via SSH, which, by itself, has made Papertrail well worth it.

A non-production server was compromised, due to a since-rectified configuration issue. Papertrail notified me almost immediately, allowing for immediate action to be taken.

There’s a variety of pricing plans, and there’s even a free for life plan, which includes plenty of features for most folks. I’m currently on the free plan, but plan on upgrading soon. Adding more servers and will need the extra space at Papertrail.

Do you use a log management service?

View Results

Loading ... Loading ...

In addition to collecting logs from your servers, you can also send logs from your applications. Got a PHP application that’s erroring out for some reason? You can send that error to Papertrail for later investigation.

Same deal with Apache logs, MySQL logs, and pretty much every other piece of software that generates logs.

Not many limits on what you can configure Papertrail to do for you. It’s very powerful.

I suggest you give it a try. Installation is super easy, especially if you’re using rsyslog. Below is a screenshot of their installation instructions. Doesn’t get much easier than that.
papertrail-install

0