Not long ago, I installed VLC on my home desktop. I forgot how I did it. Time came I needed it on my work desktop and I went to sbopkg and started to install it. Good grief, I sunk into a morass of dependencies. I kept finding more packages that had to be installed and more and more, and then I stopped and though “It wasn’t this difficult when I did it at home.” I looked in /var/log/packages for vlc and saw
/var/log/packages/vlc-1.1.0-i486-1alien
Aha, that’s how I did it. I had downloaded Alien Bob’s pre-built vlc package and it had all the dependencies included. I looked to see if I had kept the original package download, but I must have deleted it in one of my cleaning frenzies.
Before I went looking for the package, I better clean up the mess I had started creating with the sbopkg packages. First of all, I ran
slackpkg clean-system
and it removed all the packages I had just installed in my futile attempt at installing vlc via sbopkg. I hadn’t added all those packages to /etc/slackpkg/blacklist yet, so they all got deleted. Nice clean up system.
I went to find Alien Bob’s package of vlc, but the link I had was http://www.slackware.com/~alien/slackbuilds/vlc/. It didn’t work. Some research indicated that www.slackware.com was out of action. Hardware issues. But they were working on it, and would have a new server up sometime. Never mind, there are always mirrors available. I found one at http://slackware.org.uk/. I downloaded the package, ran installpkg on it, and hey presto, I had vlc installed and running on Slackware 13.37 and all was well.
I like it when these applications (like Sigil and Calibre) have easy installation packages where all the dependencies are taken care of. I am coming to really dislike Linux applications that depend on a wide range of libraries, that might or might not be available, might or not be the right version, might no longer compile or link. It gets too messy. I don’t mind if an executable is really big. It’s not like I am restricted to a 40 megabyte hard disk any more. I have gigabytes free. I would rather have a massive executable with all libraries statically linked into it, than run around and try and get a legion of libraries in sync so a small executable can run.
May 9th, 2012 in
Linux |
No Comments
When I started working with books, I was directed to Calibre for organising books and getting them on the Nook. I downloaded it and tried installing it on Linux and failed miserably. Too many missing dependencies with Slackware. It installed nicely on my MacBook, so I started using it there. It’s really good. It’s a great way of getting your books from non-official sources, making them look good and getting them on your Nook. But for me it was clumsy having it on the MacBook instead of on my regular desktop (Linux). I was introduced to Sigil – an epub editor. It installed easily on Mac and I used it there.
Recently, I had a look at them both again, and decided to try the Linux download again. What a different experience I had this time. This time there were no dependencies. I think they compiled everything they needed statically into the executable. It installed, it ran, and it’s so much better to be able to use it under Linux on my regular desktop.
So I’ve been doing a lot more with ebooks, experimenting and trying to work out what to do. I’m not going to use Calibre to manage my library. The flat model of showing books is like the iTune model and it’s good for 20 or 30 books but after that it sucks. I hate this flat approach. I want a hierarchical organisational structure. Calibre does not have it. However Calibre is wonderful for getting books, setting the meta tags, getting decent covers, checking that they work correctly, seeing how they look. I think Calibre works really well for a lot of people as their final organisational library tool, but I want more.
Getting the books onto my iPad is easy with Dropbox. I put them in my ~/Dropbox directory and pull them off on my iPad. GoodReader will access Dropbox directly, but Stanza will not. However it’s just as easy to open the Dropbox app, download the file and direct it into Stanza. Works quickly and very nicely.
However Stanza might not be around for much longer. I’ve used Stanza on other devices, and it’s been great. At some point it got acquired by Amazon, and allowed to languish. It broke for IOS5, but did get fixed. There’s no guarantee it will stay available and working. It’s a great reader.
I do not have a good workflow yet for processing books. I want to get them in, break them apart, re-apply simpler HTML formatting, add the meta data and a cover, re-build them into epub, and then put them into some organisational structure with easy access. Calibre and Sigil play big roles in this process, but not the final roles. I don’t like the HTML formatting that they apply to books, even when you create them from a scanned and OCRed paper book.
Anyway, I will keep experimenting until I find something that works for me.
I have three forms of media that are important to me – music, video and books. I collect them, I use them. My collections are large. The first two have been tackled and conquered.
I ripped my complete music library many years ago and keep it up to date. I have a simple but effective workflow for processing CDs. I have established sources for CDs, and if I pay more than 50c per CD, then it hurts me. I have established techniques for backing up the music, for tagging it, keeping the tagging data separate and safe and updatable. Music is a medium I have completely tamed.
I have just finished ripping my complete DVD library. I have not completely tamed this medium. There are some DVDs I cannot rip. About 6. I have difficulty ripping my Blu-Rays. I have done it all, but the process is clumsy. I have established techniques for backing up the videos, moving it round, and playing it. I have found ways of tagging the files, but have not get a good method yet of keeping the tagged data separate and safe and updatable. I am working on that.
And that leads to the final medium – books. I have a lot of books. I have paper books all over the house. I have a lot of books in storage back home. This is a bulky medium, much bulkier than cds or dvds. This involves many bookcases of books, boxes of books, piles of book around the house, attracting dust, getting in the way. I want to get rid of most of the paper books and keep only digital copies. But no way am I going to have books with digital restrictions that might one day stop working. And no way do I want to pay money all over again. I’ve bought thousands of books. I am not going to buy them again in digital format. What do I do?
I can’t rip the books like I ripped the cds and dvds. They started in a digital format and were converted to a more converted digital format. I could “rip” books, by scanning every page of my collection, OCRing them, proofreading them, and putting them together in a PDF or epub file. Yes, I could do that, and there are websites out there that have great ideas for automating the scanning of books. Example: www.diybookscanner.org. But scanning a few thousand books would take me more years than I have left.
Remember Fahrenheit 451? Each person was a book, and remembered that book. I see similar patterns developing. Old science fiction that is not being converted to digital form by the publishers, is being converted to digital form by fans, handed around, proof-read, formatted, and made available. I have even seen extremely poor quality official ebook releases of new books that got grabbed and proof-read and cleaned up and released back into the wild. Sure, it’s piracy. But when the pirate version is of far superior quality to the official version, which are you going to choose?
Books are a completely different proposition to cds and dvds. I haven’t really started this project yet. I am still experimenting with tools, with books, with readers. Someday all my ideas will coalesce and I will set up a workflow and start converting my books, and organise my collection. Hopefully this will be completed before we move back to Australia. I do not want to be shipping a container full of books back home. Better by far to bring a large hard disk (or two) home.
For reading, I bought a Nook. Then I bought a Nook for Anne. We used them a lot, and I did a lot of reading and I started to experiment with creating and editing books. Then I bought a third Nook. Don’t know why. Shortly after that, a trade fell in my lap and I swapped a Mac Mini for an iPad. The iPad was a revelation. The touch interface made moving through reference books easy. It was not easy on the Nook. I use GoodReader for PDFs, CHM+ for chm files, and that covers my technical material. I use Stanza for epubs, and I have some technical books in epub. I buy my O’Reilly books in digital format nowadays, and they come in both PDF and epub, so I can pick and choose. The iPad rarely leaves my hands these days. I bought one for Anne too, and hers never leaves her hands. She clutches it while she sleeps. After the complete conversion to iPad, I sold all three Nooks.
So my big new media project is in its infancy. I don’t think I can do what I want legally. I have a lot of thinking to do.
Last week at work, we had to review old code to see if it was useful and needed to be rewritten. We had to read through old C files and try and work out what they were. Luckily, many of them had copious comments at the beginning of each file, describing what they were for, what system they were part of, what they did, who wrote them and who authorised them. This was useful because many of these programs were written before 1993, and everyone who had ever worked on them is long gone. Thanks to sensible comments, we were able to understand what nearly everything was. It gave me a better appreciation for the value of comments.
Much of the documentation and change history had been stored originally in SCCS and then RCS and then CVS, but all of these have long gone too. We now use Subversion, and are thinking about git. Every time we change our revision control system, we lose knowledge. Comments embedded in the code are invaluable. I don’t think everything needs to be in the text, but relevant details about what and who and why and when should be there. At the start of the file. I have done this with my code, mostly, but from now on I will do it assiduously.
I have a whole bunch of really messy bash scripts to do a lot of things. These scripts control my cd ripping and encoding and organisation, and my video ripping. They help me backup and transfer data. They are simplistic and they are out of control.
For example, I have a pair of scripts called music2trap and trap2music. The first one will rsync my /music directory (where I have all my mp3s) with a mount point where my mp3 transfer external hard disk is attached. The second one is for work when I attach the external hard disk and sync it with the
/music directory at work.
They work. But it sucks because I can’t remember what things are, where the mount point is, and if the mount point changes (like the external mounts as /mnt/sde for various reasons) I have to make a temporary change, then forget to change it back, or I create a one-off copy of it, or anything else.
So I got sick of all this. I decided it was time to do the right thing by ALL my shell scripts. I started with this pair. I renamed them more sensibly. I don’t know why they ended up with trap in the name. So now they are music2ext and ext2music.
Both use command line switches for over-rides.
- -h gives Usage, and the usage is useful.
- -d is dry run mode so I can see what will run without running it.
- -s /mnt/sdx lets me over-ride the default mount point.
And now I plan to do the same to all my scripts. Every time I have to use one of my shell scripts, I am going to take it apart and rebuild with these points:
- command line switches
- will have -h and usage
- will have a dry-run mode
- will allow over-rides of directories or mount points with a command line switch
- anything else that might change will have a command line switch
- comments, must have comments all through
- must have sensible names
I read about doing this a long time ago in a book about bash scripts, but it took me this long to understand why you do a decent job of even throwaway scripts. Because throwaway scripts turn into long-standing and often used scripts.
It’s been a long time since I have written here. I’ve been meaning too. I’ve missed putting a lot of stuff up, that would have helped me. Got to start again sometime, and now is as good a time as any.
Last I wrote, I was struggling to rip my DVDs under Linux. No more struggling. I got that whole thing working smoothly. I do it all at the command line, I had a nice smooth and fast workflow with a bunch of shell scripts I wrote, and I ripped my entire video collection. Every DVD, all the movies, all the TV series, all the weird stuff. Every single bit of it. I ended up with about 4,000 video files, taking up about 2 terabytes. The basic tools were vobcopy to rip the DVD to hard disk, lsdvd to work out what was on the DVD, mplayer to check portions of the DVD, and Handbrake to encode to mp4 format. All the files got stored on one big 2T hard disk attached to my Mac Mini. This is broadcast via iTunes, and picked up by my Apple TV device attached to the home theatre. I get to view everything, easily.
No more having to wade through previews that I cannot stop or fast forward through. No more being thwarted in controlling the DVDs. No more having to switch my PAL or SECAM DVDs to my special DVD player. No more having to fuss with regions. All the hateful mess that the media companies force on us is gone. I can watch all the video, that I bought and paid for, the way I want to watch it. Quality is good, sound is good, I’m really happy.
I could be happier. iTunes and Apple TV work fine if you have 20 or video files to choose from. Just like iTunes might work fine if you just have a couple hundred music files. But once you get large collections, iTunes’ flat interface to the data just plain sucks. I have 120,000 mp3s, and about 4,000 mp4s, and moving around that with iTunes sucks. I stored the movie files hierarchically in categories that suit my tastes, but iTunes ignores that. Never mind, it’s the only part I don’t like.
I do have another way of accessing the video. I installed ServeToMe on the Mac Mini, and StreamToMe on my iPad, and it lets me browse my video collection and respects the hierarchy. I can find a video within seconds by traversing the directories, instead of ages when I have to hold the down arrow down and scroll forever using iTunes.
So anyway, the big video ripping project is done. Now I am enjoying the fruits of my labour by watching everything. I remember the sense of relief I felt when I finished ripping all my CDs in one big burst that took about a year, ripping about 5,000 CDs. I had the same sense of relief after finishing ripping all my DVDs. It’s time to settle down and enjoy my music and my videos.
And also time to think about the third form of media that I used to collect – books.
A workmate had a DVD he had bought in the UK. Strummerville – a history of Joe Strummer and the Clash. He couldn’t play it. He wanted to know what it was, and he wanted to see it, one way or another.
I put it into my Blu-Ray player and that said “Cannot play”. No reason given. I put it into my handy-dandy Cyberhome $50 special DVD player, now region-free, and it played just fine. So it wasn’t a bad disc.
I put it in my Linux computer, ripped it, converted it to mp4 and handed it all back to him. That was fine, and he can now play it, but he still wanted to know what was wrong with it, why didn’t it play in his DVD player? It’s not a big deal as he can watch it on his big TV, pushed through his Apple TV. So back to Linux.
lsdvd gave me the answer.
$ lsdvd -x /data/Movies/STRUMMERVILLE
libdvdread: Using libdvdcss version 1.2.11 for DVD access
libdvdread: Couldn't find device name.
Couldn't read enough bytes for title.
Disc Title: unknown
Title: 01, Length: 00:58:00.010 Chapters: 15, Cells: 15, Audio streams: 01,
Subpictures: 00
VTS: 02, TTN: 01, FPS: 25.00, Format: PAL, Aspect ratio: 16/9, Width:
720, Height: 576, DF: Letterbox
Palette: 000000 d38467 1e8080 798273 ae8d8a 94a099 69bfb1 bc8271
b7865a ae8c35 c78d8a 1e8080 728685 bb7d91 1e8080 6c7f88
Number of Angles: 1
Audio: 1, Language: en - English, Format: ac3, Frequency: 48000,
Quantization: drc, Channels: 2, AP: 0, Content: Undefined, Stream id: 0x80
There we go – it’s in PAL format. Most cheap players in the USA only know about NTSC. This one was bought in the UK, so it’s in PAL format.
I thought I better check for the region too, so I used vlc on the actual DVD while watching the stdout/stderr output. This came up:
$ vlc /dev/sr0
VLC media player 2.0.1 Twoflower (revision 2.0.1-0-gf432547)
[0x804cae8] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
libdvdnav: Using dvdnav version 4.2.0
libdvdread: Using libdvdcss version 1.2.11 for DVD access
libdvdnav: DVD Title: Strummerville
libdvdnav: DVD Serial Number: 3cb174b3
libdvdnav: DVD Title (Alternative): Strummerville
libdvdnav: Unable to find map file '/home/hgriggs/.dvdnav/Strummerville.map'
libdvdnav: DVD disk reports itself with Region mask 0x00000000. Regions: 1 2 3 4 5 6 7 8
It is region free. So there we go, it’s PAL and region free.
May 8th, 2012 in
Linux,
Music |
No Comments
Up till now, I have been ripping and encoding DVDs on my Macs. I use Mac The Ripper to rip the DVD and store the entire thing on disk in a VIDEO_TS directory. Then I use HandBrake to encode those VIDEO_TS directories to single files in a mk4 format, and for the foreign films I get it to include subtitles.
But my Mac is worn out and the DVD drive will no longer work. Mac The Ripper has disappeared. The Mac way does not suit my workflow. I wanted to try and move the process to my Linux Slackware desktop. Last time I tried this was with Slackware 13.0 and I failed at installing dvdrip. Now I use Slackware 13.37 and I had hopes that I could get something installed.
I struggled to install dvdrip, and eventually I got it to install. Unfortunately it will not rip my DVDs. There is some mismatch or error and it will not work. I have abandoned dvdrip for the time being.
I discovered a different path and have gone down it.
I now use a number of tools to do my ripping.
I use vobcopy to rip a DVD and create a VIDEO_TS directory with all the components of a DVD in it. I mount the DVD first, not sure why I have to do this, but I mount it, then run vobcopy. Like this:
mount /mnt/cdrom
vobcopy -v -m -i /mnt/cdrom -o /data/Movies
This works exactly like Mac The Ripper does on my Macs, except there are some minor problems. I have only tried it with Region 1 DVDs so far, and don’t know how it will handle other regions yet. I have tried it with foreign language films (Hong Kong Hustle, Curse of the Golden Flower, House of Flying Daggers) and it will get part way through the DVD and then lock up. These are still Region 1 DVDs, but they are foreign language with subtitles. On the other hand, I have Watchmen and it has subtitles but does not lock up. There is something strange with this that I do not understand.
The second part of the operation is to create a mk4 or mp4 file, some small encoded file with the just the essential movie in it that I can play on a laptop. The first step is to determine what title on the DVD is the actual movie. I use dvdrip to view the VIDEO_TS directory. It selects the title it thinks is the movie, probably by the length and the number of frames or chapters. I can play it from dvdrip to make sure I have the right title. Once I am sure, I quite DVD and move to the next tool.
I installed HandBrake. This gives me a command line version called HandBrakeCLI. There are instructions for the command line version of Handbrake. The title number I discovered in dvdrip goes in switch -t. For my first cut, I am using this command line to create a mp4 file:
HandBrakeCLI -v --preset=Normal -t1 -i /data/Movies/BUBBA_HO_TEP/VIDEO_TS -o ~/Bubba_Ho_Tep.mp4
Just like on the Mac, HandBrake takes time to do this encoding. I haven’t mastered subtitles yet, but it’s supposed to handle them. The end result is very nice when played back with mplayer, but xine has a really hard time with it. Lots of playback errors, pixellation, stuttering. I will have to experiment with the settings to get a format that is rock solid.
So, despite major ups and downs with installing software, I now have a method of ripping and encoding most of my DVDs on my Linux Slackware desktop. It’s a good start. I will keep working at it, and try and solve the subtitles, and try and solve the foreign language ripping problem.
I resumed my effort to install dvdrip on Slackware 13.37 using sbopkg.
The problem was that during the creation of the package, it was failing because it could not find any man pages.
Appending installation info to /tmp/SBo/package-dvdrip/usr/lib/perl5/perllocal.pod
mv: cannot stat `/tmp/SBo/package-dvdrip/usr/share/man': No such file or directory
So it’s time to leave sbopkg and do it manually. I went to slackbuilds.org and searched for dvdrip, downloaded the source tarball – dvdrip-0.98.11.tar.gz – and the slackbuild tarball – dvdrip.tar.gz – and put them together. Same result, but I could look inside dvdrip.SlackBuild and see what was happening. It wants to copy man pages from the source tarball, but there aren’t any in there. I commented out that section, and tried it again. Success. It finished and it created a package to install. I installed it. Then I ran dvdrip.
Well, would you look at that. I get a brief flash of a splash screen and then a dvdrip window ready for action. And there’s a very nice feature – Debug -> Check Dependencies. I run this and get this screen:
![[dvdrip dependency screen]](/wp-content/uploads/2011/07/dvdrip_depends.png)
There are three issues:
- rar is a worry, because I’ve got version 4.01 installed and they want a maximum of 2.99. I read the dvdrip notes, and they say version 3 and up will not work, and they provide a version of rar that does. I’ll uninstall the current version of rar, and install the one that they recommend.
- I don’t care about mjpegtools because I have no need of VCD or SVCD encoding.
- fping bothers me because I did install it, and dvdrip is not recognising it.
So fping needs work. I did install it, but it’s installed in /usr/sbin and a regular user does not have access to it. I set up a symbolic link:
ln -s /usr/sbin/fping /usr/bin/fping
and tried it:
This program can only be run by root, or it must be setuid root.
Hmm. Okay, I’ll accept this challenge and the security risk.
chmod u+s /usr/sbin/fping
and there we go. dvdrip can see it and it’s okay. It’s not really important, it’s there for clustering and I have no need to rip dvds and encode them over a cluster of computers. I don’t have a cluster of computers.
And now, we have no real problems with dependencies:
![[dvdrip good dependencies]](/wp-content/uploads/2011/07/dvdrip_depend2.png)
Now to see if I can rip a DVD. I put in a DVD, I create a new project, and wow, this looks really nice. I get things set up, read the table of contents, it automatically selects what it thinks is the real movie amongst all the junk, and it’s right. Then I hit Rip It. This is what I get.
![[dvdrip transcode problems]](/wp-content/uploads/2011/07/transcode_error.png)
I can play that part of the DVD fine. I can look at it, but I can’t rip it. More research necessary.
Okay, that’s pretty sad. This appears to be a known problem, and no-one has figured out what it is. There are heaps of links to forums about it.
But in my reading of these forums, I came across mention of vobcopy. I installed it, and played with it, and it does part of what I want. If I use a command like this:
mount /mnt/cdrom
vobcopy -i /mnt/cdrom -m -o ~/Movies
Then under ~/Movies it will create a directory named after the DVD (eg WATCHMEN), and under that it will create the VIDEO_TS directory, and then in there it will rip the entire DVD and create the VOB and BUP and IFO files. This is great. This is the step that I did on my Mac with Mac The Ripper. From this directory, I can recreate the DVD. I can also play the DVD using xine or mplayer, and I might even be able to feed it to dvdrip and just use dvdrip for the encoding to other formats. More experimentation needs to be done.
I have failed to install dvdrip on Slackware 13.37. It’s a little failure and I might be able to get past it, but I was told that Handbrake works on Linux too. I’ve been using Handbrake on the Mac for a while and it’s quite nice, so I thought I would try to install it before going back to figuring out the problems with dvdrip.
So I fired up sbopkg and found handbrake and started the process. Interestingly, Handbrake requires a lot of libraries but these are all included in the Handbrake sbopkg package, and they get downloaded and compiled as well. That’s a neat feature. All these dependencies compiled just fine, which did surprise me. I see a lot of warnings in the compilations, and that’s not something I like to see. In my own coding, I aim for no warnings. I want the code squeaky clean. So seeing hundreds of warnings for silly things is disconcerting.
Unfortunately, there are some dependencies that handbrake expects that are not supplied, and the sbopkg compilation stops. It needed webkit, and that failed because icu4c wasn’t available. I installed icu4c successfully. Then webkit needed libsoup. I installed that okay. Then Webkit failed on a Perl library – Switch.pm.
Can't locate Switch.pm in @INC
I try and use cpan to install Switch.pm, but that leads me down to a deeper mess. Switch.pm needs Filter::Util::Call and Text::Balanced. They both install successfully. Switch.pm fails two tests and cpan won’t let me install it, unless I force it. I force it. Then I try webkit again. Good grief, webkit takes a long time to compile. I went to a long lunch, I came back, and an hour after that, it finished. Fantastic. Time to try and install Handbrake again. I start it running and go do things, and eventually it finishes successfully. Wonderful.
But when I went to run it, all I see is HandBrakeCLI. What? No GUI version? Did I miss something? More research needed.