More rsync filename weirdness
#203 Henry, 25 January 2015 1:23 PM (Category: Apple)
(Tags: hfs+ rsync space hyphen)

There could be another bit of weirdness here too. I had a file with spaces in the filename and a double hyphen. Like this:

jonah and the whale -- dore illustration.jpg

and this file was repeatedly deleted and copied every single rsync attempt.

Generally, I do not use spaces in my filenames. My primary OS is Linux, at the command line, and spaces are anathema. I have to escape those things all the time. If you are using a GUI only with a GUI file manager, you can use spaces to your heart's content and not notice anything. If you are working on the command line much, you learn to hate spaces in filenames.

These files came from a CD of clipart and this appears to be the only one containing a double hyphen. I don't know why it causes issues, but I renamed it, and all the others to not give me grief. I'll run some tests on it later.

No comments
Mac OS X file system weirdness
#202 Henry, 25 January 2015 12:56 PM (Category: Apple)
(Tags: hfs+ macosx rsync)

I read a post by Linux Torvalds a few days ago where he ranted about the quality of Apple's HFS+. I didn't pay a whole lot of attention to it, but with the rsyncing I am doing today to a HFS+ volume, it came back to me.

During the rsync, I noticed that two directories kept getting deleted and re-copied every single time. Why?

First of all, I had a stupid arrangement on my Linux box. I had two directories set up like this:

-- Ipad
    |-- new
    |   |-- img_0045.jpg
    |   |-- img_0046.jpg
-- iPad
    |-- 20141022_a_richmond
    |   |-- img_0047.jpg
    |   |-- img_0048.jpg
    |   `-- img_0049.jpg

I had one directory called Ipad with a capital I and one directory called iPad with small i and capital P. On the Linux boxes, these are two completely directories. On HSF+ they are the same. There appears to be no case sensitivity. When I created the Drobo volumes, I didn't get a choice about case sensitivity, I only got one choice - HFS+. I think Linus said something about this.

So when rsync is working on the Ipad, it looks at the Ipad/iPad directory and deletes the 20141022_a_richmond directory and copies in the new directory with its files, then moves on to the iPad directory and there's only the new directory now so it deletes it and copies in the 20141022_a_richmond directory and files. Next time, it does the same thing over and over.

Sure, it's my fault for having two stupidly named directories, but Linux handles this just fine rsyncing from one to another. It's only a problem here because HFS+ sees Ipad and iPad as the same directory.

To fix this, on my Linux box, I have merged those two directories.

EDIT: I found Linus' rant.

No comments
Mac Mini Drobo problems
#201 Henry, 25 January 2015 12:05 PM (Category: Apple)
(Tags: macmini drobo rsync)

I'm having problems with the Drobo on the Mac Mini when I am rsyncing data to it.

The Mac Mini appears to be fine. There are four external hard disks attached to it, some through a powered hub. The Drobo is attached directly to the Mac Mini, not through a hub. I use rsync over ssh to duplicate my photos from my Linux box to the Mac Mini onto the Drobo. The script I use to do this is slightly modified from the scripts where I keep my music and my videos and my flacs all synced to the four external drives. I haven't had a problem with them before.

I start the sync process, and it chugs away nicely, transfers photos, puts them in the right place, everything going well, and then the Mac Mini goes grey screen and it's dead. The first time it happened, I restarted the Mac Mini and used Disk Utility on all my hard disks to repair the disks. All was well. A report came up showing that there was a problem which would be sent to Apple. It appeared to show a panic in launchd.

I started the Activity Monitor so I could see what was going on. Spotlight seemed to be working overtime. I guess that when you put a large amount of new data on the system, Spotlight has to index it. I waited for all activity to die down, no load on the CPU or memory, and started the sync process again.

It transferred a bunch of new photos to the new Drobo and then the Mac Mini did the grey screen thing again. I rebooted, ran Disk Utility and repaired all disks, and waited for all activity to cease. A second report about the failure got sent to Apple.

Then I did the sync again, but killed it every now and then so it wasn't doing too much. That worked for the rest of the night. I left it idle overnight.

Next morning, I started again. Not too far into the first sync of the day, it grey screened again. This time the Mac Mini did not come back so well. The top bar did not appear. The external disks did not appear. I think Finder was a bit messed up. Half an hour later, instead of right away, the report for Apple about the failure appeared. I couldn't do anything. By now, I was suspicious of so many USB connections with the external hard disks. I shut everything down, had to pull the plug on the Mac Mini as it would not shut down gracefully. I powered down all hard disks, totally removed all power, went and had breakfast. Then I came back and powered it all up. It all came up again and things worked properly.

I continue to sync the photos in small batches.

I suspect that there are too many hard disks attached for the USB to handle. Or Spotlight is causing me problems. Or the Mac Mini has memory issues. Or any of a bunch of things. I'll start with small things and work up.

First thing - turn off Spotlight. I do not use it. The Mac Mini is used solely for backup (Drobo) and as a media server to my Apple TV. I do not need Spotlight running. Google pointed to this page How To Turn Off Spotlight with the command

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist

and I did that and yes, it disabled Spotlight. I will keep syncing my photos and see if that makes a difference. If not, I will remove the other external hard disks and try again.

I have done long 3 T rysncs of data. But that was with the previous Mac Mini and these same external hard disks. The old Mac Mini was an earlier Intel Mac Mini with 2 gig RAM and it would rsync 3 T of data in one smooth continuous run with no panics or grey screens. The new Mac Mini has been in place about six months, and small amounts of rsync don't bother it, but it's never had to do a large sync like this before.

I'll keep testing and syncing.

3 comments
Linode state of attack
#200 Henry, 22 January 2015 4:44 PM (Category: Network)
(Tags: linode wordpress security)

My Linode still gets plagued with attacks. The biggest attacks are from bots attempting to log in to my Wordpress blogs (wp-login.php), or trying to run the Wordpress API (xmlrpc.php). The login attempts cause a lot of problems and tie up the database and Apache. I am unsure what the xmlrpc attacks actually do, but at times they can max out CPU usage, max out memory, and bring my system to a crawl.

I have resolved these two attacks with a double-pronged approach.

First, I abandoned Wordpress and wrote my own small-footprint blog software. This means there is no wp-login.php and xmlrpc.php on my system. Any requests result in a 404 error and Apache handles that quite well. Secondly, I revamped my cron script that checks for Wordpress access. As any attempt to log in or use xmlrpc is now invalid, I can start blocking IP addresses that make these attempts. I found that the previous version of my blocker had a bug. I fixed the bug, tightened the parameters, tweaked it a little, and now anyone with a persistent attempt at logging in or using xmlrpc will be blocked with iptables. I checked my logs this morning and found several IP addresses had been hammering the login about 5,000 times in a few minutes. These are the attacks that used to bring my system to a crawl, and now they are a brief blip before they are blocked.

I'm feeling happier, but security is a constant process of adjustment and measurement and patching. I might be happier, but I am no less vigilant.

No comments
Hard drive analysis
#199 Henry, 22 January 2015 4:30 PM (Category: Hardware)
(Tags: drobo hdd newegg backblaze)

So now I'm in the market for large internal hard disks to put in the Drobo, Doug supplied this link where Backblaze analyse their hard disk usage. I read their 2013 report, and this is the 2014 report.

They aren't thrilled with the 3T drives. Seagate had a really bad run with 3T drives. Western Digital were okay, but still too many failures.

With the 4T drives, the situation changes and the Seagate 4T drives did really well. I'll keep all this info in mind. But right now, 3T drives are most cost effective, so that's what I'll be looking for.

And today Newegg sent me one of their regular promos and these Western Digital Red NAS 3T drives were on sale at $109 with free shipping. I bought two of them. I'll wait a couple of weeks and get another two. I don't want to get four all from the same batch. There's a chance they will all start failing at the same time.

I'll have that Drobo fully populated soon.

No comments