Getting attacked

I run my webserver and mailserver on a Linode. I run a bunch of different websites, lots of domains, lots of subdomains. Had problems couple of days ago and it turned into a saga.

I woke up on the 9th and found that the webserver was unresponsive. Email was not collected. I could not ssh to the Linode. Wow. I went to the Linode site and the dashboard said it was still up and running. I tried to ssh in again, and it worked but very slowly. I found that I was being attacked and it had brought the Linode to its knees. So first things first, I rebooted it. When it came up I turned off Apache. Now it was responsive again and I could work out what was happening.

Apache access logs showed that I was getting slammed with these requests:

www.hgriggs.com 88.35.66.19 - - [09/Oct/2013:07:38:06 -0400] "POST / HTTP/1.1" 200 35219 "-"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
www.hgriggs.com 82.90.76.116 - - [09/Oct/2013:07:38:06 -0400] "POST / HTTP/1.1" 200 35219 "-"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
www.hgriggs.com 93.66.187.75 - - [09/Oct/2013:07:38:07 -0400] "POST / HTTP/1.1" 200 35219 "-"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

Same thing, over and over and over again. 20 times a second. This was hitting a WordPress blog, and the process was overwhelming WordPress, MySQL and Apache. I turned Apache back on, and CPU usage soared, and everything started to become unresponsive again. My first attempt at getting things fixed was to remove the blog. I moved it aside, and put a small 105 byte static HTML page there. That helped a lot. Apache was able to respond really well to that, and the load was not too great. I shortened that static HTML page to 65 bytes. With the load being handled, I could have a look at what was happening and see if I could do something better about it.

I looked in the Apache access logs to see where it was coming from. It was coming from about 20,000 different IP addresses, so I couldn’t just add rules and block the source with iptables. My initial thought was that this was a botnet. I did a lot of reading about how to control this. I guess it was not a botnet, but someone sending a series of attacks with spoofed IP addresses.

I checked system logs to see if anything else was happening, and noticed that port 25 was being hammered. Sendmail was handling it okay. I increased the sendmail greet_pause, and that helped. All the attack traffic was dropped. They were connecting and blasting away with garbage before the greeting came, so sendmail just dropped the connection.

That left me with a lot of inbound traffic, and not a lot of download traffic. Apache and sendmail were handling the load, and everything was stable. It was just depressing. I found that my logs were rapidly expanding, so I dropped the log level on sendmail. My reading indicated that I just had to wait it out. So I did. The attacks continued for about 40 hours. I checked the rate occasionally. There were times when there only 5 attacks a second, mostly 20, sometimes it went up to about 40 attacks a second. Apache handled it all okay. Sendmail occasionally hit the maximum connection limit and throttled things back. The Linode was handling things okay, but it was slowly grinding away at my monthly network allocation.

I decided to look at the traffic so I used tcpdump to see what was coming in. I was curious if there was any POST data in the port 80 attacks, and what was being blasted in the port 25 attacks. Turns out it was roughly the same stuff. It was garbage data, mostly control characters, NULLs, and badly formatted XML. I think it was attempting to do buffer overflows, or maybe just overload the processors. Or something.

POST / HTTP/1.1
Accept: */*
Accept-Language: en-us
Content-Type: application/octet-stream
Content-Length: 217
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: hgriggs.com
Connection: Keep-Alive
Cache-Control: no-cache

.qzr..q..^P.$.7@.......w.^M...uV.b.;L.....:..m.7!yX..@.l[k...j.W.a....=I..S.rw..^M].I...    Gg\.....}g@...
[.0Y..Q.....KK...(.   ...s%..>`.wi.^.0.<....L.....v.....
{.Y....W[u..+$.....+.....

But the tcpdump gave me an idea. They were targeting hgriggs.com. It was out of action anyway, with the blog moved aside, so I had nothing to lose. I changed the DNS for hgriggs.com to 127.0.0.1. After a couple of hours, and the DNS change had time to propagate, all the attacks stopped. I was hoping that the port 80 attacks would stop, but wasn't expecting the port 25 attacks to stop too. I guess they weren't doing MX lookups, just hammering away at port 25 and port 80 on hgriggs.com. So I hope they have fun hammering on their own machines via the loopback device.

I checked the IP addresses at the end, and it was up to 80,000 different IP addresses. I doubt if this is a botnet. They were probably just spoofing the IP addresses.

That's where I am now. hgriggs.com is a dead website. No attacks are coming in. I'm going to wait a week and then change DNS back and see if they are still hammering at it.

I am going to move the blog to a subdomain and keep the main domain as a throwaway, that I can easily turn off if necessary.

Unsubscribing from digital Wired Magazine for iPad Newsstand

Last year sometime, I subscribed to the digital version of Wired Magazine. A couple of times in the past I had subscribed to Wired on paper, read a few issues, and then let it lapse. True to form, I read a few issues of digital Wired, then stopped. It’s interesting, but takes a long time to work through each issue. and each issue is about half a gig in size. If I didn’t archive and delete the old issues, the Wired app would consume all available space on my iPad. I only have a 16 gig iPad, and memory is getting squeezed, especially once the iPad 3 was released and all my apps got upgraded with retina support and almost doubled in size.

So I decided I didn’t want digital Wired any more. How to unsubscribe?

It’s a good model. Get someone to subscribe, then make it super difficult for them to find how to unsubscribe, and let auto-renewal bring the money in.

I looked around myself a few times and could not find any details about my Newsstand subscriptions. If it was only two or three dollars a year, I might have just let it continue forever and read an issue or two a year. But digital Wired is $20 a year. As my subscription renewal deadline got closer, I started looking harder for how to turn it off.

Finally I found it. On the iPad, run the App Store. On any of the main pages, scroll to the bottom. There are three buttons – Apple ID, Redeem, Support. Choose that left one, the one with Apple ID and your Apple ID. A menu list is displayed. Choose “View Apple ID”. You get a dialog box. Look down for the heading “Subscriptions”, and click the long button that says “Manage”. Then you get to see what subscriptions you have in place, whether Auto-Renewal is in place, and when that is due to happen. I turned off auto-renewal, and it gave me all the details I needed about when the subscription would end.

If I get the urge to resubscribe before it ends, I can go back in here and do it.

WordPress security

I’ve been doing some reading about WordPress security. I already do some iptables blocking on IP addresses that make excessive login attempts, but I was curious about what they were doing and why. I found that there are some good WordPress plugins that add application level security. I installed two plugins.

Simple Login Logs gives me more information about login attempts. I wanted to see what parameters they were using.

Limit Login Attempts will let you do application level blocking of IP addresses, with notifications.

So now I will get more information, and just that little extra bit of security.

Incremental improvements to intruder checking

I’ve been making incremental improvements to my two scripts that check for intruders. Fixed some bugs, improved my understanding, added new features. I’ve been tweaking them since last Friday and now they are solid and working well and reporting well. The system is working really well.

Last night, I had attempts made on POP and WordPress and it accurately identified them and blocked them and reported it to me. I guess that people are getting bored with the Olympics and turning back to attacking.

So now my Linode has a few more levels of safety added. It’s not totally safe, but nothing really is. It’s not important enough to be worth anyone’s attention. The security is good enough for the time being. I’ll keep an eye on things and see if more work needs to be done.

I can think of one more improvement already. Right now, I am polling the log files. Every few minutes I check the log files to see if there is anything new. That’s inefficient. I read that there are new Linux facilities so that you can register an interest in a file, and if the file changes, you get notified. No need to poll – every time it changes you can check for intruders.

Verizon MiFi 2200

Anne is going away to a conference soon. I remembered how easy things were when we were in Australia last year. I had bought a Telstra Mobile HotSpot device and it was a neat little device that connected to the cell phone network and acted as a Wireless Access Point, so the iPhone and iPads could connect out. I started inquiring to see if there was any such device here for not much money.

Turns out my boss has a Verizon MiFi 2200 and he used it extensively for a year and did not renew the contract. I can use it if I want. I went to Verizon to see what it would cost to connect it up for a short time. After working through a lot of options, the easiest way was to just add it to my existing wireless account, activate it, get the activation fee waived, and pay month-by-month $50 a month for 5 gig access. That will see Anne through her conference and beyond. When it’s done, I can cancel service. So basically, $50 for Internet access for Anne’s iPad for the month. I would have preferred $10 a month for that, but it is what it is.

I activated it last night at Verizon, then took the device home and tried to make it work.

It turns on, and I can connect to it with my iPad. I can get to the online controls and change the settings and password. I can see it in action. So connection to the device is working fine, but the device is not connecting back to up the cellular pipe to Verizon.

I downloaded the MiFi 200 manual and actually read it. It appears that there is one more activation step I have to take. I have to connect it to a Windows PC or a Mac, install the software stored on the device, and activate it. Okay, I can do that.

I connected it to my MacBook, and the device showed up. I installed the software and that worked. I ran the VZAccess Manager program and it saw the device and then wanted to set up a Network Profile. That failed. No reason given. It will not go past this step. I suspect the software was written for an earlier version of Mac OS X, and it doesn’t like the version I am running.

I took the device to work, took it to one of our IT staff who is heavily Windows oriented (they are shrinking in numbers), and she connected it to her PC, installed the software and we activated it. It worked. Now I can connect my iPad to the MiFi, and it will connect out as expected. Great. It’s ready for Anne. I’ll get it set up for her iPad tonight, and she can start working with it now so if there are any problems we can get them solved before she goes away, and she gets plenty of experience with it before she has to rely on it. I suspect she might even want to keep it activated when she comes back. That sure would be useful.

I don’t have a charger for the device, but I don’t really need one. It takes a mini-USB cable and I have a spare one that came with an external hard disk. I have spare iPad chargers so the USB cable will plug into one of them.

Having a connection like this is very useful for Anne. She can Skype me and everyone she knows. She can do her email, her browsing, compete in Skywords tournaments, look up maps of her tours, and all the other wonderful conveniences available through Internet access. And especially use the travel tools for the iPad like FlightTrack Pro and FlightBoard. Those two tools are invaluable tools.

So another success. Things have been going well lately, and going right.

Automatic blocking on excessive WordPress logins

After my success with a script to hunt down intruders on POP and IMAP, I wrote a second script to look through my Apache logs for excessive attempts to login to WordPress. I had already had one attack that almost crashed my system in the first week of being on Linode, and I see daily attempts at with some strange parameters included. Now that the basic scripting is done for POP and IMAP, it was easy to adapt that to Apache. Took about ten minutes, put it live, and it promptly starting blocking IP addresses. I’ve blocked about 10 already. The pattern seems to be someone makes 5 attempts from one IP address, waits 5 minutes, then makes another 5 attempts from the next sequential IP address. The five appear the same, but they are attempting different passwords:

91.236.74.131 - - [30/Jul/2012:15:53:57 -0400] "POST /wp-login.php"
91.236.74.131 - - [30/Jul/2012:15:53:58 -0400] "POST /wp-login.php"
91.236.74.131 - - [30/Jul/2012:15:53:59 -0400] "POST /wp-login.php"
91.236.74.131 - - [30/Jul/2012:15:54:00 -0400] "POST /wp-login.php"
91.236.74.131 - - [30/Jul/2012:15:54:00 -0400] "POST /wp-login.php"

These five attempts all meet my thresholds, so the IP address is locked out.

I discovered one problem. Some of these IP addresses try a POP attack first, then an IMAP attack, then the WordPress attack. So the IP address gets added three times to my firewall. I hadn’t considered that this could happen. This is not optimal. Tomorrow I will make it more sophisticated and avoid duplicates.

But overall, I am very happy with the extra security measures.

Youtube is pause free again

For the last year or more, YouTube has been unusable here. Pretty much any video clip you start to play, it freezes after a short while, and the whirling circle appears. I assumed at first it was in the browser, so I searched the Internet and found lots of suggestions. I cleared the cache, cleared my cookies, cleared my plugins. None of these things worked. So I had a long think about it. I use the same desktop at work and at home, with the same hardware. I have no problems with YouTube at work. It is clear on all devices at work, but freezes on all devices at home. My MacBook and iPad will work fine at work, but not at home. So what can it be?

I thought it had to be my router at home. I have a D-Link DGL-4100. It was a top router when I bought, and I think that was 2006. Dang, I didn’t realise it was so old. But it’s still pretty good, except for this YouTube problem. I went over the configuration carefully, and couldn’t find anything. So we’ve pretty much had a YouTube free year.

Tonight, with the Olympics on, I couldn’t stand it. I went over the router configuration again, ever so carefully. And found one item I had overlooked before. I had SPI turned on. That’s Stateful Packet Inspection. The router allegedly looks inside packets for malformation, and tosses away problem packets. Hmm, I wonder if that is the problem?

I turned it off, the router rebooted, and YouTube has not frozen once since then. YouTube is usable again. We can watch the Olympics.

What I would like to know is what happens? Is YouTube taking shortcuts with speed and the packets look bad? Maybe my router firmware is old. Maybe lots of things. But right now, I have Olympic rowing to watch and I will speculate about the why of it after the Olympics.

Automatic blocking of failed POP and IMAP attempts on my mailserver

I have been trying to make my Linode system even more secure. I wanted to implement fail2ban for POP and IMAP, as those two are getting the most attacks made. IMAP is fine, the logs of failed attempts go to /var/log/messages and include the IP address. I could do IMAP easily. But it gets a tiny fraction of attempts compared to POP. POP is the problem. Notice of a failed attempt goes to /var/log/messages, but without an IP address. Notice of an attempt with the IP address goes to /var/log/secure. Two different files, and the only link between them is the date and time.

I looked at the code for popa3d, hoping to include the IP address in the failure log. Not a good thing to do. Updates would erase my work.

I looked at changing the log file that popa3d logs would go to. I really do not like the syslog system. That was a failure for me.

I looked at replacing popa3d and imapd with Dovecot. I installed Dovecot on one of my systems and attempted to configure it. I now have enormous respect for the packagers of Linux distributions. Getting some of these packages working is a difficult feat. Dovecot will handle an enormous range of security measures, and describes them in full. But getting the simplest possible system working was not well described and I was unable to configure it adequately. I could not make it access /etc/passwd and /etc/shadow. Incorrect permissions. I admit the failure was mine. Dovecot appears to be a wonderful system, I am just incapable of understanding how to make it work on my system.

So after spending a lot of time on all this stuff, and learning a lot of things, I resigned myself to accepting the inevitable and writing my own security system like fail2ban. It took me two hours. It works. I read through /var/log/messages and get a list of POP and IMAP failures. IMAP failures have the IP address included, so if those IP addresses have too many failed attempts in too short a time, I ban them immediately with iptables. I store the time of the POP failures, then read through /var/log/secure and find the IP addresses that match the times, count them, and if they exceed threshholds, I ban them immediately with iptables. I tested it using old attempts, and it works fine. I get an alert that an IP address has been banned. They get added to my iptables blacklist as well, so if the system is rebooted, they stay banned. I also note date time and reason, so I can remove them later if I feel generous.

Sometimes you have to accept what you have, and work with it. I accepted the strangeness of the POP logs, and worked with it. The customised approach is one I can live with.

So I put this live, and immediately felt a lot happier and more secure. But what happened then? The Olympics started, and not one single person has attempted an attack. I’ve been getting daily attacks since I started with Linode, and suddenly the Olympics and no attacks. That’s pretty funny.

Can no longer eject media with eject

I have a lot of scripts that rip cds and dvds, and at the end I use the eject command to get the cd or dvd out. It used to work just fine. After the latest Slackware upgrade, this no longer works. I get the error:

eject: unable to eject, last error: Inappropriate ioctl for device

But it works if I am root. I assume a permissions problem.

I tested it with both CDs and DVDs and it behaves the same with both. It won’t eject the media, and it gives that error.

There is one potential reason and that is if something has it open. Because root can eject it and a user cannot, I did not think this was the problem, but it’s still best to check it just in case. It’s a cdrom so you don’t mount it. It is not mounted. But just in case something is using it, as root I did “fuser /dev/cdrom” and “fuser /dev/sr0″ and it showed nothing was accessing it. That eliminated that potential solution.

I could take the easy way out and set up an alias making “eject” = “sudo eject”, but that’s cheating. I would really like to know why. I used “eject -v” for verbose mode and got this result:

eject: using default device `cdrom'
eject: device name is `cdrom'
eject: expanded name is `/dev/cdrom'
eject: `/dev/cdrom' is a link to `/dev/sr0'
eject: `/dev/sr0' is not mounted
eject: `/dev/sr0' is not a mount point
eject: `/dev/sr0' is not a multipartition device
eject: trying to eject `/dev/sr0' using CD-ROM eject command
eject: CD-ROM eject command failed
eject: trying to eject `/dev/sr0' using SCSI commands
eject: SCSI eject failed
eject: trying to eject `/dev/sr0' using floppy eject command
eject: floppy eject command failed
eject: trying to eject `/dev/sr0' using tape offline command
eject: tape offline command failed
eject: unable to eject, last error: Inappropriate ioctl for device

It tries to eject it as a CDROM, and it fails. Then it tries some other methods, fails on each of them, and then reports only the last error result which is for treating it as a tape drive. No wonder it says “Inappropriate ioctl for device”. So that’s pretty useless. I tried it as “eject -rv” for verbose and treat it only as a CDROM, and got this:

eject: using default device `cdrom'
eject: device name is `cdrom'
eject: expanded name is `/dev/cdrom'
eject: `/dev/cdrom' is a link to `/dev/sr0'
eject: `/dev/sr0' is not mounted
eject: `/dev/sr0' is not a mount point
eject: `/dev/sr0' is not a multipartition device
eject: trying to eject `/dev/sr0' using CD-ROM eject command
eject: CD-ROM eject command failed
eject: unable to eject, last error: Input/output error

and now finally we see what the cdrom eject reason is – input/output error. Great. So useful. However, I have seen this sort of error many times before, and it’s often associated with permissions problems. So I really do think it is a permissions problem.

I tried adding my user name to various groups in /etc/group and then trying to eject, but that didn’t work. I think I know the reason why, but I have more tests to make.

Midnight Commander gets a change

When the recent big Slackware upgrade came down the pipe, Midnight Commander got an upgrade. The first thing I noticed was when I next invoked mc, I got this message and I was back at the command line:

Failed to run:
Your old settings were migrated from /home/hgriggs/.mc
to Freedesktop recommended dirs.
To get more info, please visit

http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

When I start it again, it starts up as per usual. Almost as per usual. So far I have noticed one change in behaviour.

Before, if I wanted things to start up again the same, I would have to have focus in the right pane when I quit. Then when I started up again, the current directory would be in the right pane, and whatever directory was in the left pane before I quit would still be there. Focus would be in the right pane again. So I worked like this, and most of my file copy operations would be copying from the right pane to the left pane. In some weird way, I accepted this as normal. After all, as a programmer, when I move the contents of one variable to another, I am copying from right to left. And there are other examples. Over the last 18 years of using Midnight Commander (I started using it in 1994), I got so used to this method of operation that I never thought about it much, just accepted it as the way things were.

After the upgrade, I found that this no longer worked. If I left focus in the right pane when I quit, then when I started Midnight Commander up again, the left and right panes were switched and focus was still in the right pane, and I would have extra keystrokes before I could do what I intended to do. I rapidly realised that it now works in a more sensible way. If I leave my cursor in the left pane, then at startup the current directory is in the left pane, and whatever directory was in the right pane is still in the right pane and focus is in the left pane. Copying actions now go from left to right.

It took only a couple of minutes to overcome the habit of the last 18 years. I much prefer copying from left to right. It just feels right.

←Older