The mail/webserver is dead, long live the mail/webserver
#139 Henry, Wednesday, 06 June 2012 4:30 AM (Category: Hardware)
(Tags: mailserver)

I broke the problem into two parts. Getting mail working, and getting web services working. Mail is the most important, but depends on SpamAssassin. Web is not so important, but should be very easy to get going.

I had done an rsync of the system on the 29th, so at most, I had lost about 6 days data. But I do an automated daily backup at 3am. I should be good with data up till early on the 5th. Not so. On the 28th April, I was fooling with github, and had set up ssh authorised keys and installed a new set at home. My automated backup depends on rsync to my home server, and that depends on ssh authorised keys, and it had failed after 28th April. If I had run just one manual backup, I would have noticed that I had to manually answer yes to an ssh question, and all would have been well. But I didn't. So I didn't have my daily backups. I was stuck with the rysnc data from the 29th May. Good thing I had done that, otherwise I would have been really screwed.

When this is over, I will set up a different backup scheme, and will touch a file with the date of the backup as the filename, then rysnc all my data to where it should go, and then test if that dated file was copied. If not, raise an alarm. That is a simple mechanical test of the backup that will work. But that's for the future. In the meantime, I have mail and web to get running.

I tackled mail first. SpamAssassin has to be installed. I eventually worked out the problem I had. I had installed Slackware on this server back when it was Slackware 12.0. We've had a lot of upgrades since then, and slackpkg has brought me up to date all the way to 13.37, and beyond. Perl has been upgraded many times since then, and some things have changed. Perl modules used to get installed in /usr/lib/perl5/site_perl/version. Old settings still put some there. But SpamAssassin checks the new approved location in /usr/share/perl5. So I could see that modules were installed in the old location, but SpamAssassin could not see them. I used removepkg to delete perl in its entirety, then manually go and clean up all old and new locations so nothing was left behind. Then slackpkg install to put Perl back again nice and clean. Then I wiped out all the cpan data and temporary storage (/root/.cpan) and then I started again. I used "cpan -O" to show all outdated modules, and then "cpan module" to get it up to date. When the installation was clean, I installed all the extra modules I needed. Then I ran "perl Makefile.PL" in the Mail-SpamAssassin directory, and it passed all tests. Installed it. Great. Used sa-update to set up the rules. No problems there either.

Downloaded and installed spamass-milter - no problems. I already had sendmail configured, I already had the startup scripts for it, so I fired it all up and ran tests and it worked perfectly. My sendmail configuration was already setup, so that was it. Mail was running. And mail started flowing in. That was a relief.

Then getting the web services running was not a difficult job. Just mechanical. All the code was in place, I just had to load the databases up, and I had the output from the mysqldump from the 29th. I had a moment of panic when the databases could not be accessed. Database passwords were not being accepted. I did some reading, and realised that I had to execute "flush privileges" for the passwords to take effect. Once that was done, the web server was running. It had taken me four hours, but it was all running again. (Except majordomo, but that can wait till tomorrow night.)

I am not going to let this happen again. Better backups required. Tests of the backups. But more importantly, I don't want to be dependent on hardware. And we've had power outages and Internet outages this year. I don't want this stuff to affect me. And we are going home in a year or two, and I want to have a presence here. I need to look for alternatives.

But in the meantime, it's 4am and I have to go to work in a few hours.