MP3 tagging
#347 Henry, Sunday, 19 June 2016 10:38 PM (Category: Music)
(Tags: mp3 tag music)

This is a saga about my music library and maintaining it. This is not for everyone, this is my hobby, it gives me pleasure to do this and keep my library maintained, and to write software for it, and write scripts, and fiddle with it. This is about me having fun, my way. Your mileage will undoubtedly vary.

Streaming vs Own Library

I know that most people have abandoned their own music libraries and simply use Spotify or Rhapsody or Apple Music and pay a monthly fee and get access to a reasonable range of music. I hear that Amazon is about to fire up their own streaming library too. That's all good, it's an easy way to get access to good quality music easily, and have zero maintenance.

I don't participate. I have a fine little library of music that spreads across many genres, and is deep in areas that I like. I am particular about some things. For example, I like early Mike Oldfield, especially Ommadawn and Hergest Ridge. I grew up with those two pieces and really love them. I listened to them on CD recently and went WTH? That's not what I remember. And I researched. He remastered them and for my tastes, the new versions sound like completely different pieces of music. I want to listen to what I grew up with, not with what he now considers the definitive version. So I bought the two boxed sets of these two pieces and they contain the new versions and the original versions. I ripped them and can now listen to the original versions, the way my mind hears them. But on the streaming services, I think all you can get is the new versions. That's just one example of why I don't want to rely on the streaming services.

Also bandwidth. My way, my music is on a HDD and there's no wasted bandwidth. This is a downside according to my co-workers who don't want to carry a HDD and manage it and back it up. And they don't care about how much of work's bandwidth they use during the day.

If Spotify lose an artist and have to pull all that artist's music, it doesn't affect me. If Rhapsody has only the last few releases by a band, say Gaye Bykers On Acid, when they had lost their fyre, but don't have their first two releases - Drill Your Own Hole and Stewed To The Gills - that doesn't affect me either. I bought the vinyl and converted them to mp3s myself. Much better music than the later stuff.

So I maintain my own curated library of music.

Library Metadata

But one of the issues of maintaining my own curated library of music, is the metadata - the tags. A library of music has to have accurate metadata so you can find it, sort it, organise it, enjoy it. Music players rely on the tags in the mp3 files to make sense of it all.

So the tags are stored in the mp3 files. The file contains both the music and the metadata. For most people, that is the only location of the metadata, the tags. If they replace their mp3s, say a re-rip for better quality, they have to enter all that data in again. You could use external services to populate the tags for you. I estimate that the public suppliers of track data and tags are pretty good with tags for new rock and pop. Probably 95+% accurate. For older rock and pop, it's really variable. Maybe 70% accurate. For classical music, it's abysmal. So many errors, so much missing data. I care about this stuff, but nearly everyone else doesn't. It's a hobby for me, like collecting stamps. I want the data to be right. Storing the data in the music file and having to re-enter it if the music file changes is not the way I want to go. I want a permanent repository of data that exists outside the music file. And not in a database. I want something simple, text based, and stored with the mp3s so it's portable.

A very long time ago, I collaborated with Richard on this very issue. We went through a number of iterations until we came to a system that we were happy with. In each directory with mp3s, we store a simple text file called track.dat. This contains all the mp3 tag data I could possibly want, in a simple scheme. I have a Perl program called tagger.pl that reads that file and adds the tags to the mp3s. If I replace the mp3s, I just run tagger.pl and it uses the data in track.dat and shoves it into the mp3 so the music players can see it. If I get better data, or fix mistakes in the data, I just run tagger.pl again and again and again and it takes the information in track.dat and stuffs in into the mp3.

The Library

I have my library of music at home, and at work, mostly on Linux machines, and occasionally on a Mac so I can put some music on my iPhone. The prime repository is not a Mac. It is an external hard disk on my Linux desktop. iTunes is not my prime repository, iTunes is just the end of a chain, a conduit for putting the music in my device. I don't listen on my iPhone much any more. I listen mainly on my home and work Linux desktops. So my scheme has to be OS independent; independent of any one music player; simple, text-based for easy editing; and able to work with a lot of data.

I have 123,000 tracks in my library, about 9,000 CDs or LPs or whatever. I have a lot of data. I enjoy working with it, and I enjoy listening to it.

id3v2 and Sort Order

Richard and I developed that scheme in 2005, and we've been using it ever since. We each use it in different ways. It's a great initial design but we've deviated in the application. Shows how good it is that it works for both of us. As the years have gone by, I have improved both the metadata I store in track.dat, and the tools I use to take the metadata out of track.dat and insert it into the mp3s.

I'm on Linux so I use the id3v2 command line utility to put the tags into the mp3s. This utility dates back quite some years, and hasn't been updated in a while. It worked pretty well for me till about 2010, and then the tagging scheme changed a little. I think it handles mp3 tags up to version 2.3, but not 2.4 and beyond. Until last week, I continued to use it.

Over the years, I have only ever had one problem and that was the artist and composer name order. The way I've been doing it, Tim Buckley appears under T for Tim. I would prefer he appear under B for Buckley. Similarly, Claude Debussy appears under C for Claude, rather than D for Debussy. If you want to find Debussy's music, you have to know his first name. I always forget the first names of the classical composers. I want the artists and composers to appear in last name order.

MP3 tags are stored with four-character tags. For example, the artist name (really performer name = P) is stored with the TPE1 tag, and the composer name is stored with the TCOM tag. Modern music players do accept tags for sort order. Sort-artist is stored with TSOP, and sort-composer sort is stored with TCOS. So to achive the desired result for Tim Buckley and Claude Debussy, I would have tags like this:

TPE1=Tim Buckley
TSOP=Buckley, Tim
TCOM=Claude Debussy
TCOS=Debussy, Claude

and the normal name is displayed, but the sort version is used for sorting. Works really nicely. Except, the utility id3v2 does not accept the new tags. I can't put them into the mp3.

There's also one downside to the new scheme. Most modern music players have the concept of an ALBUM-ARTIST. This helps with compilation cds. The album artist is stored with the tag TPE2 and the sortable version in tag TSO2. This is really unfortunate for classical music because traditionally TPE2 was for the orchestra and TPE3 was for the conductor. But now that TPE2 has been co-opted as the album artist, you can't store the orchestra in the mp3. I was bummed out when I found this.

So I would like to use the new tags that allow sort artist and sort composer, but the utility id3v2 does not support them. The source code is available, and I have worked with it. I cannot make the modifications to allow the newer tags, I am not clever enough a programmer.

mid4v2

And that is where matters stood until recently. As part of an effort to find an alternative, I installed a Python module called mutagen which looked really promising. I played with it, and it's neat. I could write little utilities to interrogate mp3s and work with them. And it's all Python. Love it.

Then I found that mutagen comes with a utility called mid3v2 which is a drop-in replacement for id3v2. Same switches, same everything. Presumably the name is something like Mutagen id3v2. It's a Python script, so I have a better chance of understanding it. I've been playing with it, and it works and it's now my base tagging utility. All I had to do to get it into use was change "id3v2" in my tagger.pl to mid3v2. The irony of calling a Python program with a Perl program has not escaped me. Best of all, mid3v2 supports the sort order tags. So now I can do everything I want to do with tagging.

And now I face a new project - a major overhaul of every track.dat file, adding new tags, improving the data to take advantage of the new features, cleaning up some earlier design decisions. And then re-tagging all the mp3s. This is going to be fun.

I have over 10,000 track.dat files. Luckily, I do not have to edit every one by hand. I already have utilities - both shell and Perl - that read all the track.dat and search for one tag and value and replace it with another tag and value. It was short work to enhance that so the search for one tag and value and append another.

So I can say "search for artist = Tim Buckley" and add the line sort-artist = Buckley, Tim" on the next line, and it will read through all 10,000 files and edit them. I backed up my data first. But it works cleanly and nicely and takes about three seconds to read and edit all 10,000 track.dat files. I have been cleaning up composers and artists, and having to make decisions. How do I determine the sort order for Nusrat Fateh Ali Khan? What is the last name of Ralph Vaughan Williams? What about Orlande de Lassus? Does that go under D or L? For me this is fun.

There is one other advantage to mid3v2. It's fast. id3v2 was quite slow on external USB hard drives. mid3v2 is not slow at all.

Summary

So there we are. I have a new project to work on. At the end, I will have a nicer music library that is better organised. I will have learnt a lot in the meantime, and it will give me the opportunity to write new utilities and shell scripts and Perl programs and Python programs. For me, this is fun.

No comments
Linked In
#346 Henry, Sunday, 19 June 2016 6:01 PM (Category: Work)
(Tags: linkedin)

I joined Linked In in 2009 when it was new. It was like Facebook for work. But it didn't really do anything for me. I think it has helped a lot of people get jobs and make contacts, and that's fine for them, but I haven't seen any benefit. I didn't make much effort joining and creating contacts and links and putting up profiles and doing all the regular things that you are expected to. My work situation is fairly stable, and quite busy, and I have not been looking for a job.

Linked In has irritated me, with a lot of nagging emails over the years, although I turned off notifications. That has been better last couple of years, but there were times when it got really annoying.

I've had a lot of my skills confirmed by other people, like my SQL Server skills, and my Microsoft products skills. These days, I wouldn't know how to fire up anything Microsoft, so I don't know how my non-existent skills in these fields got confirmed.

But it doesn't matter. I'm a few years away from retirement now. I think my job is reasonably secure for those last few years. If it wasn't, then I would just retire early. So I don't get any benefit from Linked In, and I don't want to participate any further. And recently, it got bought by Microsfoft. I don't really care about that, but it reminded me that I didn't really want to be in Linked In any more, so I went to delete my account. I couldn't see any reference to it when I logged on, so I did some research and found where it's kept.

Under Privacy and Settings, near the bottom, there's a link to Closing Your Account. They do a short exit interview and ask why you are leaving. I said I was retiring and didn't want it any more. Then for security I have to enter my password again, and there are two buttons - Back To Settings, and Close Account. But Close Account is greyed out and can't be clicked. So right now, it is not possible to close your Linked In account and leave. Perhaps while the Microsoft purchase is taking place, this is a special measure to keep the number of members at normal levels to keep the sale price stable. Or something.

So I can't leave. What can I do? I can poison my data. I am now a 77 year old banker from Philadelphia, with skills in banking and lying, and a work goal of retiring. I altered all my data. I can do that, even if I can't leave. I'll come back after the Microsoft purchase is concluded and see if I can delete my account then.

No comments
Slackware sound mess
#345 Henry, Saturday, 04 June 2016 3:53 PM (Category: Linux)
(Tags: linux sound pulseaudio)

I had got myself into a big mess with Slackware and sound.

In January 2016, Slackware was forced to work with PulseAudio. I had been very grateful for years that Slackware had avoided working with PulseAudio and systemd. Anything involving Lennart Poettering is anathema. When I first experimented with PulseAudio years ago, it was awful and did not work with any of my gear. I was fine with ALSA, it worked on everything I had, I had no problems with it at all. And don't get me started on systemd. Its claim to fame seems to be that it speeds up bootup speeds. That's just fine and dandy, but I boot my systems several times a year, if that, so that's not a big enticement. What is worse for me is creeping featurism. There are a number of things I have read about systemd that make me want to avoid it. So far, Slackware has.

PulseAudio was made a dependency of bluez, the Bluetooth controller, and Slackware was forced to follow. In January, the first changes came down that included PulseAudio, and that started months of serious annoyances.

Since then, I keep thinking about one of my sayings about every app being upgraded until it is useless for its original purpose. I suspect that Slackware is going to go down this path too. Perhaps Linux in general. If it does, I will consider what I will do. I think Apple is going in directions I don't like either. Not sure where I will go. Perhaps I'll try BSD?

But in the meantime, I did my best to get along with PulseAudio. I followed instructions and had sound working on my main desktop with some things and not others. mplayer would work (and therefore aTunes would work), xmms would work, Firefox would work, but Chrome would not. That was good enough, and I left it at that.

Slackware is getting ready for the big jump to release 14.2. Big changes keep coming down the pipeline. I think when it's released, I'll get the DVDs and then do a clean install. It's okay to do incremental upgrades on a regular basis, but eventually you get too much cruft and you need to do a clean install. I think I'm at that stage.

Last week, a new version of Firefox came down, and it really irritated me. It would not play Youtube videos. No sound, no video. I could watch YouTube videos in Chrome but without sound, so I was stuck YouTube-less. I left it to the weekend to investigate.

A few days ago, another big Slackware update came down and things went bad. No sound anywhere.

So I was really stuck. I think much of the problem is cruft. Configuration cruft in Firefox and installation cruft in Slackware.

The weekend came and I made a determined effort to get things working again.

I started with Firefox.

My Firefox configuration is quite old. That was probably what was preventing me from playing YouTube. So first step there was to delete all the .mozilla configuration and start up Firefox as if new. I did that, making one mistake - I forgot to export my bookmarks first.

I set up Firefox again, configuring it to do things my way. I can recreate my bookmarks later. I don't use bookmarks much anyway, so it's no big loss. And once I did this, I could play YouTube videos again, but still without sound. That fixes the Firefox issue, and the sound issue is with Slackware, so I left Firefox and moved to the Slackware configuration.

I spent a few hours researching this online, reading documentation, experimenting, rebooting. I also went back to the original upgrade notes for Slackware in January when PulseAudio was first introduced.

In the end, I got sound working everywhere, with all my applications. I changed three things. I am unsure if all three were necessary, but at the end, I had sound everywhere again.

I tested all my usual sound apps again (mplayer, xmms, aTunes, Firefox, Chrome) and they all worked at extremely high volume.

How to control the volume with PulseAudio? More research and it's with a thing called pavucontrol. I could set the overall volume with that, and then use each application's fine controls to make smaller adjustments.

And after a day of frustration, I have my system working as well as it was back in January before PulseAudio was introduced. Six months of frustration, and a wasted day to cope with shit forced on me. This is why people move to Apple. Apple has a mediocre Unix at the core, but the consumer stuff just works (generally speaking). The more I have to fight this stuff, the less inclined I am to keep fighting with it and the more inclined I am to take the easy way out.

No comments
Let's Encrypt is off beta
#344 Henry, Friday, 15 April 2016 9:52 AM (Category: Web Development)
(Tags: https encryption)

I've been following Let's Encrypt since last year. They provide certificates for encrypting websites so they use https instead of plain http. With automated renewals. And it's free.

I've been planning to do this for some time. I have a few months window before the next set of visitors arrive, so I will start working with it this weekend and attempt to get it all set up.

My Linodes are in a bit of a mess right now. I have two, not one. I need to finalise the new one, move the last bits from the old to the new, and then switch everything to https.

No comments
FruitJuice
#343 Henry, Thursday, 14 April 2016 3:07 PM (Category: Apple)
(Tags: fruitjuice battery macbook)

Chris suggested that I should look at the Mac app FruitJuice as a way of monitoring the battery usage on my MacBook to get a better idea of what is happening when I use the timer on it.

First of all, it's $9.99. That made me squirm. Then I remembered all those early years when software cost $40 to $250, and realised how my perceptions have changed when I find a $10 app feels like too much money. I did my reading, and it seemed interesting enough, so I paid my money and installed in on my MacBook Air. I didn't want to put it on the MacBook yet, I wanted to get a feel for it.

Bought it in the Mac App Store, and got it installed. Ran it, and set it to run at login time. First thing it did was tell me it wanted to do a Maintenance Run. Told me to plug the charger in and get it up to 100%. I did that. Then it told me to take the charger off, and run the Mac till the battery went below 20%. I did that. Took a few hours. Then it was calibrated and it knew what it was doing.

Now, whenever I use my MacBook Air, the FruitJuice app tells me how much time I need to spend on battery power each day. And right now, it tells me to spend one hour and five minutes on battery. This is good information. And it gives me plenty of other info about the state of the battery and my pattern of usage. Neat. I suppose it's worth the $10. If it keeps my battery running for longer before I need to replace it, and optimises the lifespan, then it is worth it.

So with all this info and practice done, it was time to put FruitJuice on my MacBook that is driving the Drobo and BackBlaze, the MacBook that started this whole exercise.

Oh. Rats. That MacBook is locked in at Lion, and FruitJuice will not install. OS is too old. So much for that.

But it's still useful on the two MacBook Airs which are both on the current OS.

No comments