At work, we have a number of small servers set up for specific tasks. One of them is being upgraded. We're building a new one with everything on it, and when it's ready, we will switch it in. The old one is running "Red Hat Linux release 9 (Shrike)" so it's pretty old. The new one we are building has been installed with "CentOS release 5.6 (Final)". So far it's been a disaster setting the new one up.
There are two main problems.
First, the IT guy who installed it, who does all these installations and loves CentOS, does a minimal install. So much stuff I need is missing right from the start. The libxml2 development libraries. Midnight Commander. After the last time I installed our software on one of these CentOS boxes, I have a list of the missing software and I give it to him and ask to have it installed. He does. But this box has to do a lot more work and it needs a lot of stuff that I consider to be normal software on a Linux server, but he considers to be "extras". I can't even set up a cron job without going in as root first and adding my user to /etc/cron.allow.
This box has to accept email on specific domains, pass them to two users, get procmail to process the emails and pass them to two applications which parse them, do database lookups, process them and trigger a range of actions, and send replies back. And the email has to be spam checked. We've used sendmail and spamassasin quite successfully up till now. Last time I set this system up, it took me about 6 hours from scratch, most of it spent installing spamassassin. I have taken about 6 days so far on this task, and it's still not done. I'm waiting for our IT guy to work out how to get the applications to run.
I started configuring it, and kept running into problems. I have to configure sendmail. Oh, you want to configure sendmail? You'll need the sendmail-cf package installed. WTF? This is a core part of a server, in my opinion, but no, here it's an optional extra. I start installing spamassassin, and all the Perl modules it needs. Fail. Some of them have C++ components, and oh dear, the C++ compiler component of gcc is not installed. I need the gcc-c++ component, so I have to ask for it to be installed, and wait. After that, it went okay. But then tying sendmail and spamassassin together got difficult.
libmilter not installed. Oh, you need the sendmail-devel package for that. Eventually I get sendmail configured, spamassassin installed, libmilter installed, spamass-milter downloaded and installed, startup and shutdown scripts created and installed, and it all works. I install the applications that need to run at the end of the chain, and I test them manually and they work fine. I configure procmail for the two users, set the whole chain up and do the final tests. This involves telnetting to the box on port 25 and manually running through SMTP transactions to trigger the whole thing.
Then I discovered the second big problem. CentOS has a security mode where you can specify with very fine granularity what programs can and can't do.
My first test showed this problem. sendmail accepted the email, passed it successfully through spamassassin, handed it to the user via procmail and procmail tried to pass it to the application. I got this in the procmail logs:
/usr/local/epager/bin/epager: line 4: syntax error near unexpected token `(' /usr/local/epager/bin/epager: line 4: `use POSIX qw (strftime);'
The offending line was a simple use POSIX. At first I didn't know what was going on. I removed the POSIX module and wrote a workaround for strftime. This time it treated every line of Perl as something bad. It was as if it was treating the Perl program as a shell script and failing on every line.
/usr/local/epager/bin/epager: line 13: =: command not found /usr/local/epager/bin/epager: line 16: =: command not found /usr/local/epager/bin/epager: line 17: =: command not found /usr/local/epager/bin/epager: line 18: =: command not found /usr/local/epager/bin/epager: line 19: =: command not found /usr/local/epager/bin/epager: line 20: =: command not found /usr/local/epager/bin/epager: line 21: =: command not found /usr/local/epager/bin/epager: line 22: =: command not found
I spent a lot of time trying all sorts of things. Days were lost. Eventually I realised that this was not a problem with my code. It runs perfectly if I am at the command line and run it manually. Procmail was not doing it right. I went and spoke to the IT guy and described the scenario. He told me about the fabulous new security mode that CentOS has. He has to add some clauses to the security configuration somewhere to allow procmail to run my apps. He can't work out what I am trying to do, so he gets me to run as root
and run my tests again while he looks at some security logs and works out what he needs to do. We iterate through this many times and eventually we get to the point where one app will run mostly, but cannot create a new log file (although it can append to existing ones), and the second app will run but do absolutely nothing. Note that these run perfectly if you run them at the command line. Running them from procmail is not successful. And that's where we are right now.
We have to have the logs. We occasionally get subpoenaed and have to get all sorts of details about connections and emails. Without the logs, we can't do what needs to be done.
CentOS is really secure, so secure that the apps that need to run will not run. I've currently spent six days on this task that should have taken 6 hours. I'm not happy, my bosses are not happy. I just have to wait till he can figure it out. I am asking for this new excessive security mode to be removed, but I keep getting told "I'll work it out".
The only good side is that every thing that I have had to do has been documented, so next time I have to set something like this up, I will be able to do it in 6 hours.