Asterisk Hack Post-mortem

Having your production Asterisk-based phone system hacked is no fun, as I have learned from first-hand experience over the past few days. Even the best of IT administrators taking ever security precaution in the book dreads the day their critical server gets hacked. You hope you've done everything possible to stop your servers from being hacked, but you are never 100% sure. There is always some hacker smarter than you, but more importantly, smarter than the best security practices you put in place. Hackers always seem to find a new hole to exploit.

Since I spent the last couple days poring through the Linux system logs and the Asterisk logs, I thought I'd do a detailed post-mortem for the benefit of other Asterisk users. Let us begin...

The first sign of trouble was a few months ago when our international calling was blocked by our service provider for suspicious international calls to Middle East countries. I investigated the Asterisk-based server for any SIP credentials that were easily attacked. There was only a couple of SIP credentials (test softphone accounts) with slightly easily guessed SIP credentials, however it didn't appear these accounts were using in the hacks since the CDR records didn't show these fraudulent calls as coming from these accounts. I changed the SIP passwords anyway just to be safe. To be double sure, I had technical support login to the box and make sure everything was secure. They did see some calls being made from the Asterisk CLI and technical support suggested I change the 'root' password, which I did even though it was a long password. They didn't see anything else out of the ordinary, but they obviously missed something since a month later we were hit again...

I was notified that our phone service provider had put a temporary block on international calling. I checked a system file and saw this scary command run on Saturday:

Jan 7 15:05:31 asterisk userhelper[305]: running '/sbin/reboot -f' with root privileges on behalf of 'root'

Bastard hacker rebooted my Asterisk server! Well, at least he was considerate enough to do it on a weekend when the office is closed. Next, I pored through the CDR records on Monday (1/9/12) and indeed I confirmed fraudulent calls being made on a Saturday (1/7/12) when the office was closed.

Here's a sampling:

"","","9011901140720740717","international","","OSS/dsp","Zap/25-1","Busy","","2011-12-07 04:29:13",,"2011-12-07 04:29:20",7,0,"NO ANSWER","DOCUMENTATION"

"","","s","incoming","","Zap/2-1","","Dial","Zap/g1/01138765063921","2012-01-07 15:00:52",,"2012-01-07 15:00:52",0,0,"FAILED","DOCUMENTATION"

"","","900212641869513","international","","OSS/dsp","SIP/skypetrunk-0945e380","Dial","SIP/skypetrunk/00212641869513","2012-01-07 03:08:05","2012-01-07 03:08:16","2012-01-07 03:08:30",25,14,"ANSWERED","DOCUMENTATION"

"","","900212641869534","international","","OSS/dsp","SIP/skypetrunk-08926d10","Dial","SIP/skypetrunk/00212641869534","2012-01-07 03:11:53","2012-01-07 03:12:02","2012-01-07 03:12:31",38,29,"ANSWERED","DOCUMENTATION"

"","","","incoming","","SIP/skypetrunk-08629a78","","Wait","360000","2012-01-07 03:32:39",,"2012-01-07 03:42:39",600,0,"ANSWERED","DOCUMENTATION"

"","","900212641869534","international","","OSS/dsp","SIP/skypetrunk-0874dc90","Dial","SIP/skypetrunk/00212641869534","2012-01-07 03:51:10","2012-01-07 03:51:19","2012-01-07 03:52:06",56,47,"ANSWERED","DOCUMENTATION"

I bolded a couple of the CDRs above. You'll notice the hacker hit both our PRI trunk (Zap/g1) and our Skype SIP trunk. Well, at least he's an equal opportunity hacker attacking all our trunks! Hack our traditional PRI, ok, I can accept that, but attacking my beloved Skype? Unacceptable! shame-on-you

It was pretty simple to discover which calls were fraudulent. I simply ran this command below which searches for "OSS/dsp" in the CDR folder. This will display any Asterisk CLI (command line) commands being executed. Other than voicemail access you shouldn't see anything. If you do, you've likely been hacked:

Continue reading Asterisk Hack Post-mortem...

Tags: asterisk, bash, cdr, cron, hacked, hacker, linux, nobody, post-mortem, rootkit, sip, skype Related tags: answered documentation, documentation international, skypetrunk answered, international calling, asterisk based, asterisk