A response (of sorts) to the hacked Asterisk server post-mortem
<DISCLAIMER>
I am in no way trying to poopoo the work done by the author of this post. I really like when people come out and show what they found and how they fixed things. I also didn't attempt to contact the author as I'm sure he's been inundated with emails/comments on what he woulda/shoulda/coulda done. My excuse is I was up one evening and couldn't sleep. As such I was catching up on some reading (no downtime, ever) and this was one of the articles I read. I thought I'd go through the various comments and offer my rather uneducated opinions...
</DISCLAIMER>
Jan 7 15:05:31 asterisk userhelper[305]: running '/sbin/reboot -f' with root privileges on behalf of 'root'
At this point, I'd assume that the entire server has been compromised. This is the point I would have (if possible), shut the server down and move to a DR site or backup server to provide the services required. The fact that someone other than yourself or one of your trusted sys-admins has rebooted the server from the root account means you no longer control the server. From here I would have started by looking at the various shell history files to see what else had been run.
I'm a little curious how the attackers got root or even system level access to the system by simply "hacking" the Zap/Skype trunks available on the system. In my fairly limited experience in VOIP compromises, the attackers have mostly found loopholes in the dial plan and used these loopholes to make international calls.
Next, I decided to check the /etc/password file to see if the hacker created any backdoor user accounts. I didn't see any, however I did notice that nobody account had this:
nobody:x:99:99:Nobody:/:/bin/bash
Here's a good example of where something like OSSEC would have come in handy. OSSEC would have notified the sys-admins of the change to the system accounts as it happened (IF the mail system notification was configured and not disabled by the attacker).
So if the 'nobody' account is able to get bash-level access, they can view and modify any Asterisk file! That means access to the SIP passwords and the ability to launch the Asterisk CLI! From the Asterisk CLI you can do just about anything, including make outbound calls or even delete critical Asterisk files.
I find this hard to believe. Then again I haven't played with an Asterisk installation in ages. I'm a big fan of running services under their own account. This would have mitigated this issue somewhat. The attacker would have had to have changed the shell of the user running the Asterisk service.
Looking at the 'history' command output for user 'root' I saw these two suspicious commands executed:
333 su aster
334 su nobody
This is another area where OSSEC would have saved the day with its watcher service and email notification. The su action would have triggered an alert to the sys-admins notifying them of the action. Hopefully the sys-admin would have realized something was up and done a little digging.
Surprisingly, the hacker didn't clean up their history. Or if they did, they forgot to clean up these two commands. I don't have an 'aster' account, so that command fails, but obviously the 'su nobody' works, causing the root user to login as 'nobody'. The beauty of using the 'nobody' account is that all the commands executed by the hacker doesn't show up in the 'root' admin account that most IT admins monitor. At that point I could no longer see the commands the hacker was executing under the 'nobody' bash shell. Even when I logged in as 'nobody' (after changing password) I couldn't see any history.
There are a number of ways to ship bash command history off to a remote syslog server. These logs should be reviewed regularly. That said, this is probably a fairly difficult thing to do if you're a small IT shop. A properly configured SEIM installation with alerts for commands being run by strange or system users would have caught this as well.
Interestingly, the 1.call file the hacker chose to do was very simplistic. It didn't do anything other than dial the number and then hang around for 36000s doing nothing. No prompts played or anything. Obviously, if someone answered the call, they'd likely hang up in 5-10s. It seems as though the hacker was making prank calls, since their script didn't do anything. What sort of motive is that?
My only guess here is it's probably some kind of premium rate number for incoming calls. I've seen this happen locally, along with the "free international calls" hacks in the past. It's unfortunate that people/hackers are no longer interested in simply showing their skills, it's all about the money now and if this was anything other than a premium rate scam of some kind, I'd be very surprised.
Lessons Learned:
- Never allow SSH from the outside, even though it is convenient to remotely administer that way. If you MUST have SSH open to the outside, you should consider using Fail2Ban, which blocks IP addresses . It even supportsblocking hackers brute-force attacking your SIP credentials should you have port 5060 open to the Internet for remote phones.
- Make sure you regularly check your /etc/passwd and /etc/shadow to ensure no new accounts and that accounts don't have /sbin/bash access (except those you know about, such as 'root')
- Use LogWatch to email you daily reports of what's happening on your Asterisk box, such as logins, failed SSH attempts, etc.
1. If only life and system administration were that simple. In the real world, this isn't really an easy one to sell. Personally I change what port SSH listens on along with a firewall burst rate limiter and fail2ban. That along with SSH key authentication instead of simple password authentication should be enough.
2. OSSEC file watcher with email notification should resolve this issue quite nicely. It'll also alert the sys-admin should new packages be installed etc..
3. While daily reports are great, I've found that if you receive them daily and they look very similar, you will tend to skim read them and eventually ignore them altogether. A better option would be to be emailed only when things look wrong or change unexpectedly.
I'm not 100% sure yet if there is still a rootkit on the Linux box. Although there are anti-virus utilities for Linux, I am loathe to put anti-virus on a production PBX if it stays resident in memory and uses CPU cycles.
He's absolutely right here. I'd go with either rkhunter or chkrootkit here. Both of these tools can be run and updated from Cron which would mean little interaction from busy sys-admins. Also the lack of a daily report from either tool could be a red flag that something is up...
Update:
Found this interesting error in the logs:
zdump[27161]: error: Bind to port 10001 on 0.0.0.0 failed: Address already in use.
This should have been stopped by a local firewall running on the system. I had a system once that was compromised via SSH (a story for another time) and the attackers had dropped a SSH scanner which didn't work on account of the egress filtering in place on the system. Similarly this attack should have been stopped by a firewall filtering incoming connections to "unnecessary" or unknown services.
Looking through to comments I see many people mention OSSEC and Rkhunter which makes me feel a little warm and fuzzy inside. I know this is all probably a little redundant but it was on my mind. I also know that securing systems may be on the back burner for many system administrators who are not only pressed for time, but also money. I've been in that boat a few times. These days it's difficult to keep on top of systems and their state of (in)security which makes a proper monitoring system all the more important, even if it's a simple homegrown solution to begin with.
w3af scripting
So the very awesome scanner w3af can be scripted. This is very handy if you want to automate scans of a particular site. There's even plugins to mail the output to you which is even cooler. It's been a while since I've played with w3af so this isn't going to be a post about the tool. I just wanted to share this little script which I'm currently tinkering with. As part of an appsec assessment you'll want to run a couple of automated scans against your target before digging into the manual stuff. There are a few options out there and w3af should definitely be one of them.
Anywho..here's the script file:
# super scan with w3af pluginsdiscovery webSpider,hmap,fingerprint_os,fingerprint_WAF,findvhost,allowedMethods,robotsReaderaudit dav,formatString,globalRedirect,osCommanding,sqli,xpath,xss,xst output console output htmlFile output config htmlFile set fileName /tmp/w3afreport.html back back target set targetOS unix set targetFramework php set target http://wwww.zonbi.org back start
And that's pretty much it. You will want to adjust things like targetOS, targetFramework and obviously target. This script will dump an HTML report into /tmp when it's done.
To fire off the scan you'll just pass w3af_console the -s option like so:
w3af_console -s superduperpooperscan.w3af
Pretty handy I think.
peas...
Bypassing outbound egress restrictions with cntlm and Putty
*** disclaimer: this is "lame" but handy in a pinch ***
A few of us work in very corporate environments. This means we will likely have no access to SSH services on the outside. This can be a rather large problem. Along with these restrictions, there may also be a rather nasty proxy that blocks access to fun web sites like Dark Reading and pretty much any decent blog running common blogging software. Yay corporate blanket blocks...
In order to get around these restrictions we're going to need a couple of things.
1. An SSH Server on the outside running on port 80 or 443. This can be done by either running the ssh server on those ports or by using a little iptables magic (if you have root access)
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 22
2. The CNTLM package for Windows / Linux / OS X
The CNTLM package is a neat little authentication proxy that will help authenticate yourself to the outbound proxy and allow an SSH client (such as Putty) to use it as an intermediate proxy to tunnel itself out. Before I get myself lost in a word puzzle it looks a little something like this:
workstation -> cntml (authenticates to proxy) -> proxy server (authentication optional) -> outside world
CNTLM Configuration
So first we need to configure cntlm.ini (or conf on Linux/OSX) with our authentication settings and what port to listen on. My config looks a little something like this:
I have left out the Password option as I don't like to store passwords in config files if I have to. Yes, there are options to store a hash (MD5) instead of the actual password, but still. All this means is that when you do run CNTLM you'll get prompted for your password. With the config file complete, you then fire up cntlm with a couple of command line options to get it to listen on the local port (8080 in my case).
cntlm -c <config file> -I -v
-c for the configuration file we want to use (cntlm.ini on Windows)
-I for prompting for a password
-v for debugging messages so if there are problems, we will know
Ok, we now have our authentication proxy working. This will forward our credentials to the outbound proxy from any application that connects to the cntlm proxy, which in our case will be Putty.
Putty Configuration
The Putty configuration is very simple. All we need to do is tell Putty to use our local cntlm proxy server as an outbound HTTP proxy. The settings are here:
And that's about it. CNTLM will forward your credentials through to your corporate proxy and Putty will use the combination of CNTLM and the evil Corporate Proxy to tunnel its way out. I've had issues with ISA killing my connection every now and again as it's not obviously an HTTPS connection traveling through it. It's very annoying, but not unworkable.
You can then use Putty to create port forward through to external services you want to access which is even handier, but a story for another day.
I know this is extremely lame, but it's been a life saver since being stuck in corporate land.
Fighting plagiarism in information security
I totally stole the title of this blog post from @securityninja who wrote this. Yes, I'm making light of a very crap situation but that's just me.
If you've been living under a rock you should read this from Attrition and this from @corelanc0d3r
The InfoSec Institute have been stealing from Peter. They have been selling a course which included printed materials. Someone figured out that these printed materials were a blatant copy of the work that Peter has been publishing on his site FOR FREE !!!
I'm not going to go into detail over this as there is plenty of information out there for you to read. All I'm asking is that you support Peter. Either through spreading the word that InfoSec Institute are a bunch of stealing butt monkeys or through donating a little money to the guy. He's already spent a fairly large portion of money defending his case and he needs all the help he can get. The guys and girls behind Brucon and Backtrack DE have already made sizable contributions, along with a number of awesome people within the community.
So help a guy out and let's put a stop to the constant flow of bullshit that seems to be flooding the industry of late.
./matt
@0xznb
#zacon wrap up
Overview
Zacon 2011 has come and go once again. This year was the third year the conference was run with some small but no less awesome changes. New stuff this included:
- Friday night social
- Challenge by @tomwells80
The format and venue were once again the same. I was a little concerned with the shorter time slots used. The shortest being around 25 minutes, which included setup time and questions at the end. This meant speakers seemed a little rushed on a couple of occasions but I guess it's up to the speaker to ensure that he keeps within the assigned time limit (Yusuf). I would have liked to have seen lightning talks as well, having seen how awesome they are at conferences like Defcon and Brucon.
That said, I believe the conference achieved all it set out to accomplish. The idea being to give local guys a platform to share their research ideas or various "leetness" as for most there simply is no other option available.
Friday night lights
Friday was a new addition to Zacon with the main idea being to socialize and share stories over a few beers. The venue was awesome with a little bar for those who liked a beer or two. I really enjoyed the evening. It brought a fairly mixed group of people together, giving everyone a chance to network and get to know one another.
Two talks were given on Friday. Roelof of Paterva/Maltego fame gave an awesome talk on some of the very very cool stuff that's happening in Maltego. It was evident that Roelof has been doing this for years, he was very relaxed and had a great mixture of humour and technical stuff that left the audience going "faaaarrrrkkk!!!!"
Following that Marco of Sensepost gave his talk on Pickles. While a lot of the more technical details went straight over my head, I did get the general theme of the talk and really enjoyed it. It's always great to see the local guys sharing stuff that makes the international conferences.
I would really have liked to have seen Haroons talk from 44con, it sounds like it was a very interesting talk. Alas MH couldn't make Zacon due to awesomeness happening in the Thinkst space.
What followed the two talks can be described as "hackers drinking beer and shooting the cool Jozi evening breeze". A fantastic evening.
The Main Event
Once again held at one of the UJ lecture caves, the main event was not without slip ups. None that you'd want to hurt someone's feelings over though. The talks were great and the vibe was certainly what I believe the Zacon people were after. That said the innertubes were still an issue and there did seem to be a lack of 3G coverage which made following tweets etc a little trying. Like I said, nothing to really get your knickers in a twist over.
The talks
Great talks for the most part. A couple I missed due to a raging headache from late nights / bad food intake.
Robert Gabriel - Real world soc
An interesting idea but it seemed like he was reading from a vendor sheet more than giving a talk he'd done research on. That said it was his first talk so you can forgive him.
Jeremy du Bruyn - The who's who of hash cracking
I'd been looking forward to this talk since seeing the program. Jeremy gave a great overview of hash cracking techniques and pitfalls. Very informative and very useful if you're a pentester.
Reino Mostert
Apart from looking like he'd spent the evening mainlining caffeine, Reino gave a very interesting talk on NNTP caches. I'm still not sure how much NNTP gets used in environments where this type of stuff would be effective but it was still interesting.
Yusuf Motara - Functional programming
A very good talk on function programming and how security can be built in from the start. I'm still not fond of code being used in demos as it can end up leaving a lot of the audience in the dark if they're not familiar with the language. If someone can keep me interested in a talk about a coding language I'm not likely to use in real life is saying something.
Jameel Haffejee - OS X Sandboxing
Fantastic talk by Jameel (whose surname I am apparently unable to ever spell correctly) on how OS X Sandboxing works. And no talk would be complete without a leet tool. Again Jameel came to the fore with a very cool tool for playing with OS X sandboxes. Very cool stuff.
Ian de Villiers - SAP Proxy Pwnage
Ian rocks. Plain and simple. Simply awesome talk. No need to say any more.
Manuel Corregedor - Rootkits
Is going to jail :) Very interesting talk on Rootkits and how he's been playing around with them. Unfortunately the real world application of Manuel's research is only likely to land him in jail.
Erik Gibson - Unlocking the secrets within
Very good talk on lock picking and lock sport. This is something I'm very interested in so it was great to catch it. I'm hoping this takes off in Jozi and we do get some stuff going on the locksport side.
Alas I took a break for the last 3 talks as my brain was screaming at me :(
Keynote: Richard Thieme
Apart from the crappy sound and choppy video (it's Africa) Richard gave a very interesting talk. It was almost like Dan's talk from Brucon, only a little more structured. Yes, there were a few gems within his stream of consciousness and I really enjoyed his talk as a whole.
To sum up
Zacon III was a success from my point of view. It supplied a platform for security people in South Africa to get on stage and share their stuff. It proved to many that you don't need to spend buckets of money to get real value from a security conference. I'd like to have seen lightning talks at the conference as I think it's a nice way to break up the long sitting listening to talks. The coffee on site was a very welcome touch and getting food is easy.
All in all, Zacon rocked and I'm already looking forward to next year.
The video is being posted here
The audio is being posted here
Great job volunteers & people@zacon...