I run my webserver and mailserver on a Linode. I run a bunch of different websites, lots of domains, lots of subdomains. Had problems couple of days ago and it turned into a saga.
I woke up on the 9th and found that the webserver was unresponsive. Email was not collected. I could not ssh to the Linode. Wow. I went to the Linode site and the dashboard said it was still up and running. I tried to ssh in again, and it worked but very slowly. I found that I was being attacked and it had brought the Linode to its knees. So first things first, I rebooted it. When it came up I turned off Apache. Now it was responsive again and I could work out what was happening.
Apache access logs showed that I was getting slammed with these requests:
www.hgriggs.com 126.96.36.199 - - [09/Oct/2013:07:38:06 -0400] "POST / HTTP/1.1" 200 35219 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" www.hgriggs.com 188.8.131.52 - - [09/Oct/2013:07:38:06 -0400] "POST / HTTP/1.1" 200 35219 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" www.hgriggs.com 184.108.40.206 - - [09/Oct/2013:07:38:07 -0400] "POST / HTTP/1.1" 200 35219 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
Same thing, over and over and over again. 20 times a second. This was hitting a Wordpress blog, and the process was overwhelming Wordpress, MySQL and Apache. I turned Apache back on, and CPU usage soared, and everything started to become unresponsive again. My first attempt at getting things fixed was to remove the blog. I moved it aside, and put a small 105 byte static HTML page there. That helped a lot. Apache was able to respond really well to that, and the load was not too great. I shortened that static HTML page to 65 bytes. With the load being handled, I could have a look at what was happening and see if I could do something better about it.
I looked in the Apache access logs to see where it was coming from. It was coming from about 20,000 different IP addresses, so I couldn't just add rules and block the source with iptables. My initial thought was that this was a botnet. I did a lot of reading about how to control this. I guess it was not a botnet, but someone sending a series of attacks with spoofed IP addresses.
I checked system logs to see if anything else was happening, and noticed that port 25 was being hammered. Sendmail was handling it okay. I increased the sendmail greet_pause, and that helped. All the attack traffic was dropped. They were connecting and blasting away with garbage before the greeting came, so sendmail just dropped the connection.
That left me with a lot of inbound traffic, and not a lot of download traffic. Apache and sendmail were handling the load, and everything was stable. It was just depressing. I found that my logs were rapidly expanding, so I dropped the log level on sendmail. My reading indicated that I just had to wait it out. So I did. The attacks continued for about 40 hours. I checked the rate occasionally. There were times when there only 5 attacks a second, mostly 20, sometimes it went up to about 40 attacks a second. Apache handled it all okay. Sendmail occasionally hit the maximum connection limit and throttled things back. The Linode was handling things okay, but it was slowly grinding away at my monthly network allocation.
I decided to look at the traffic so I used tcpdump to see what was coming in. I was curious if there was any POST data in the port 80 attacks, and what was being blasted in the port 25 attacks. Turns out it was roughly the same stuff. It was garbage data, mostly control characters, NULLs, and badly formatted XML. I think it was attempting to do buffer overflows, or maybe just overload the processors. Or something.
POST / HTTP/1.1 Accept: */* Accept-Language: en-us Content-Type: application/octet-stream Content-Length: 217 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Host: hgriggs.com Connection: Keep-Alive Cache-Control: no-cache .qzr..q..^P.$.7@.......w.^M...uV.b.;L.....:..m.7!yX..@.l[k...j.W.a....=I..S.rw..^M].I... Gg.....}g@... [.0Y..Q.....KK...(. ...s%..>`.wi.^.0.