I just uploaded a release candidate for Universal Extractor v1.6. Details and download links can be found in the MSFN forum thread. If all goes well, we should actually have a 1.6 final release relatively soon.
All of Legroom.net was offline from Monday through Wednesday morning. This is because my truly dumbass server colocation provider:
- sold me a new colo plan hosted in a data center that was being shut down
- neglected to warn me at the time of sale that it was being sut down
- neglected to warn me at any point before it was shut down, taking out my server in the process
- disconnected technical support / customer service numbers for the data center
- failed to mention the shutdown on their website and network status pages (which, btw, was also broken at the time)
- failed to respond to the urgent ticket I had submitted about the mysterious server outage in any reasonable amount of time
- when they did finally respond, about ten hours later, sent the reply to my e-mail address hosted on the very server that was shut down
Once I finally got someone on the phone, and was informed of the extremely unexpected data center shutdown, I was told that all servers were being moved to their Dallas data center facility. All Chicago customers were supposed to have been notified 60 days ago of the move, which didn't do me a whole lot of good as I wasn't yet a customer 60 days ago. Not that it stopped them from selling service in that data center anyway. Fucktards.
So the server was supposed to be available again Tuesday afternoon. Needless to day, Tuesday afternoon, evening, and night all passed, and still no server. Finally I called again Wednesday morning to see wtf was going on, and was told that the server had already been powered up (since about 9:30pm Tuesday night, according to my logs), but nobody bothered to connect a damn ethernet cable.
Everyone I spoke with during this ordeal (once I finally figured out how to get in contact with them) was polite and seemed genuinely surprised and regretful that I had not been contacted about the relocation, and moving an entire data center in two days is no easy task, so it's hard for me to be upset with them on that level. But the complete and utter lack of notification, the fact that I was sold a plan in a data center that was already scheduled to be shutdown, and the apparently lack of competence necessary to connect an ethernet cable to bring the server back online is just flat out unacceptable.
Alright, I'm finished my rant. For now. Just wanted to let everyone know what happened. With any luck (well, a lot of luck with these people), Legroom.net stay online without any more unexpected outages.
Somebody just pointed out to me that the download links for AutoFLAC were not working. This occurred during one of the server migrations I've been working on over the past few weeks, so I'd say it's probably been unavailable for about 2 weeks. I do apologize for this oversight - the autoflac binaries are (currently) served from a separate server than the rest of the files on this site, and I simply overlooked this when migrating servers.
For now, this has been corrected by copying the files back to the appropriate location. In the near future, I'm going to setup a new download system so that all files are tracked and served from the same location, which will prevent this issue from reoccurring, as well as giving me some more flexibility and control over the downloads. Stay tuned.
Oh, and speaking of AutoFLAC, I should mention that development is still ongoing (same with Universal Extractor), just at a much slower pace than in the past. There are numerous reasons for this, which I won't get into here, but I did want to let all of you know that updated/enhanced versions are still coming at some point.
Magnatune is an independent record label and online music distribution channel. Why am I writing about it here? Because as the title suggests, it's frickin' awesome.
I've been listening to my music collection in digital form for many years now. I first ripped my entire CD collection in 2000, and have since ripped it twice more in progressively better quality. My final rip two years ago was done as lossless FLACs (see my AutoFLAC utility for more information), so I'll never have to rip them again. I've also long had a computer connected to my home theater system, primarily so I could listen to my music collection through may main stereo system. A completely digital music collection, combined with an intelligent media player like Amarok, offers countless advantages and flexibility over manually playing individual CDs.
Despite my love of digital music, until recently I've never purchased a single song or album digitally. Instead, I've continued to purchase, and subsequently rip, compact discs. Why? Because most online music services suck balls. Why?
- Digital Restrictions Management (aka, DRM) - this anti-consumer technology implements technical controls to limit what you can and can't do with music that you legally purchased. Want to listen to it on something other than an "approved" player? Tough. Want to copy it to and play it from another computer? Nope. Want to edit/remix it? Not gonna happen. Want to send it to a friend so he can check it out? Ha, now you're just being silly.
Think you own it and should be able to play it forever? Better hope your music service/provider doesn't shut down.
- Quality - Most digital music has been sold as 128 Kbps tracks. This is atrocious quality. Why would I want to buy something of significantly worse quality (digital music) than the same exact product I've been buying for years in stores (compact discs). Recently many services have begun selling 256 Kbps tracks, which is a huge step in the right direction, but even this is far inferior to lossless CD quality music.
- Proprietary/restrictive application requirements - Most services require some kind of proprietary application to download and/or play the music. Why should I be forced to install a really crappy media player, that doesn't even run on my operating system, just to buy and download music? Why do I need to install some proprietary browser plugin just to download files that my browser is perfectly capable of downloading by itself?
- DRM - Yes, this point is worth repeating. DRM is defective by design, and it's important to treat it as such.
- Physical media - I like physical media. I like album art. I like liner notes and lyrics. I like having a physical backup in case something happens to my digital copy. I like the whole CD experience. Simply buying a digital copy of a song and downloading it to your computer without all of the value-added features is really rather lame by comparison.
- Have I mentioned DRM is evil?
I could go on, but that purpose of this post is not to rant about online other music services. The reason I'm posting this is because I want to call attention to a really great online music service: Magnatune. Their motto is "We are not evil," and unlike Google, which shares a similar motto but is so busy compiling every last detail of your life into one massive database to remember that, Magnatune actually means it. From there front page:
We work directly with independent musicians world-wide to give you downloads of MP3s and perfect-quality WAV files. We never work with major labels, and our musicians always get 50%. You can listen to every album in its entirety before buying or becoming a member.
They expand much more on this. The reasons behind Magnatune's creation are also quite interesting, and that 50/50 split directly with artists is a big deal. Be sure to check out their full info page for all the details.
Ok, so Magnatune says they're not evil, has a lot of propaganda on their site discussing that fact, and seems to be quite fair to the artists involved, but how does that affect me as a customer? Well, it basically translates into a wonderful customers-first policy that is extremely rare today. To be completely honest, and this is something you'll very, very rarely hear me say, I was 100% satisfied by the entire browsing and ordering process. Let me walk you through it:
- Decide on something to buy. Magnatune makes this quite easy:
- Music can be searched/browsed by genre, artist, or album, and includes best-selling and newest lists
- ALL music on the site can be sampled in its entirety in reasonable quality streaming audio. These "preview" tracks are actually just MP3s that are streamed directly from Magnatune's servers.
- You can listen to the samples through an embedded media player in your web browser or by opening playlist links in your own preferred media player. Magnatune even offers a special programming interface that allows the service to be directly integrated with music players so I can, for example, browse, play, and purchase any album directly within Amarok.
- Once you've found something you like, click Buy Download
- Note that you also have the choice to buy a physical CD instead and have it shipped to you
- Choose how much you want to pay, from $5 - $18. $8 is the default, though I've been choosing $10 (remember, 50% goes directly to the artists, not the label or any other middle men)
- Enter payment info
- It's not necessary to register or create a permanent account
- It's not necessary to give them your e-mail addresses, although a receipt will be e-mailed to you if one is provided
- Payment information (eg., credit card number) is never saved on the server, unlike most other online retailers who don't even give you this option
- Click Pay
- Note the provided username and password, then click on the download link
- Begin downloading in choice of audio format
- FLAC, Ogg Vorbis, MP3 (both constant and variable bitrate), AAC, and even WAV are allowed
- Album cover art is also available for download
- If desired, forward download link and password to up 3 friends
- Yes, this is permitted and even encouraged: http://magnatune.com/info/give
- Sit back and enjoy your new music in any way that you want
Now, with all that said, I'd be remiss if I didn't also mention the one negative aspect of the site. I mentioned at the very beginning that Magnatune is an independent record label. While this is a good thing, it does have one significant drawback: Magnatune as a much more limited selection than any of the major record labels. Basically, if you're used to buying music from a particular artist at Target or Best Buy, you will not find that artist's music on Magnatune. If you're only looking for big-name bands, you probably won't find much here. That's an unfortunate reality of the current industry environment. If, however, you're looking for some good music from lesser-known artists (I strongly recommend Rob Costlow if you like solo piano, and the rock group Atomic Opera is also worth checking out), you're not likely to find a better experience anywhere else.
I'm definitely not one to gush about, well, pretty much anything, but in this case I was so impressed, and so pleased, with my experience that I wanted to share it here. By all accounts, Magnatune is a good company doing good business and providing a great service to music fans and musicians alike. Check them out.
Note: All LegRoom.net e-mail will be unavailable beginning 12:00am CST 08/17/08 through 12:00am CST 08/18/08. Details are below.
I'm about to begin migrating the LegRoom.net e-mail server to a new system. As with the recent web server migration, this will be a completely new environment - new hardware, new host, new mail server applications, radically different configuration, etc. It's going to take some time to completely migrate all e-mail, update the server-side configuration for each user, test the new server-side configuration, test the client-side migration, and document the client-side migration. I'll let everyone know once the migration is complete.
I just completed migrating the LegRoom.net web server to a new hardware platform. It's now running on a dedicated host at a commercial hosting provider. The site should be much faster and more responsive now, as it's no longer running on a poky 33.6 KBps upstream DSL connection. It should also up more reliably as well now that it's no longer running under a virtual Xen instance on my desktop. :-)
In addition to the new hardware, I also made quite a few under-the-hood changes. I upgraded Drupal, changed up the apache configuration used for the website, restructured some of the website directories on the fileserver, and even changed the name of the database. LegRoom.net's been running for close to 6 consecutive years now, through various OS upgrades and hardware migrations, and it's built up a lot of cruft over the years. Since this was already a major change from what I had going before, I also took the time to reorganize things to make maintenance easier going forward.
Of course, I also tried my hardest to make sure everything looked exactly the same to end-users visiting my site, so unless you're reading this point you hopefully won't notice that anything has changed at all (aside from, of course, the speed). If you do find something that doesn't work - missing page, broken download link, lost credentials, etc. - please let me know ASAP.
Enjoy the upgraded site.
I came across a new (to me) Linux-related website a couple months ago that rather impressed me (which is something that doesn't happen all that often). The name of the site is Linux Kernel Newbies, and it's located at http://kernelnewbies.org/.
I stumbled across the site while looking for a good kernel changelog. Most changelogs that I've been able to find discuss the changes in one of three formats
- List changes/commits made in each release candidate
- List all individual commits made during the release cycle
- Briefly summarize major changes or new features
None of these really provided the information that I was looking for. Documenting changes for each release candidate is fine if you're actually using/testing -rc kernels, but it's a pain when looking for changes from version to version because it requires looking through multiple posts or documents. The commit list approach is also fine for the gritty details, but unfortunately the summaries of each change are rather cryptic and often don't mean a lot to people not actively involved in the development. The new feature and major change approach is nice in that it's easily digestible and hits the highlights, but unfortunately it usually doesn't cover enough detail for me.
While searching for a decent changelog that was something in between the detailed commit list and a high-level summary, I found the LinuxChanges page on the Linux Kernel Newbies wiki. This is almost exactly what I've been looking for. They do a great job of describing all of the new/important features of the given kernel release, including providing links to the actual commit records if you really want the full details. They also provide a list of all individual commits,
logically grouped and sorted, which makes it much easier to understand what was changes. Finally, they even cover the highlights of new/upcoming patches are are actively being development for succeeding kernel releases.
While the changelog is what keeps me coming back every couple of months, Linux Kernel Newbies also offers a few other useful resources that may be of interest to Linux users, such as the KernelGlossary, FAQ, and Forum. The homepage also provides links to other content on the site.
I don't have any affiliation with the site, and to be honest I haven't spent much time on the site outside of the changelog pages, but even so I found it so useful that I wanted to mention it here. Hopefully some others can benefit from this site as well.
This website gets a lot of account registrations. Most of these are obvious spam accounts using fake e-mail addresses. Since e-mail confirmation is required on all new accounts, these accounts are never confirmed and therefore are never fully activated. I routinely run through the account list and delete these accounts to help keep things manageable. I also run through all of the new confirmed accounts every couple of weeks looking for obvious spam accounts that actually use a real e-mail address and remove these as well.
Since this website went live, that's pretty been the only account maintenance that I've done. At this point, 1 year and 4 months later, there are nearly 400 active accounts. This is after the obvious spam accounts were removed as described above. Unfortunately, the vast majority of these accounts have not been used at all since their initial creation. A lot of these accounts are spam accounts that use legitimate sounding names and e-mail addresses. A lot of them are also one-time use accounts: the user signs up, does whatever he wanted to do once, and then never returns (or at least never uses the account again).
Due to the large volume of inactive accounts, I decided to start cleaning up these old, unused accounts as well. For the first batch, I'm deleting all accounts that meet these two criteria:
- the account was created more than six months ago
- the account was last accessed more than six months ago
So, basically, any user that signed up for a LegRoom.net account more than six months ago and has not used the account since then has had his account deleted. After the first batch of deletions, the total account count dropped from nearly 400 to about 130. Much better. :-)
If you've been affected by this, the easiest solution would be to simply recreate your account, and login periodically to keep it active. I'll probably continue to do this on a semi-regular basis going forward. If you have any questions or complaints, feel free to either leave a comment or send me an e-mail.
I had an issue with free disk space (or, more appropriately, a lack there of) on my server a while back. After some investigation, I discovered that my MySQL databases had ballooned in size to nearly 10 GB. Actually, figuring out that the /var/lib/mysql directory was taking up so much space wasn't that hard, but understanding why and what to do about it took a while (yes, I'm sometimes slow about such things).
It turns out I had two issues. The first is that MySQL configuration, by default, maintains binary logs. These logs "contain all statements that update data or potentially could have updated it (for example, a DELETE which matched no rows). Statements are stored in the form of 'events' that describe the modifications. The binary log also contains information about how long each statement took that updated data." This is fine and all, but (again by default) these log files are never deleted. There is a (configurable) max file size for each log, but MySQL simply rolls over to a new log when it's reached. Additionally, MySQL rolls over to a new log file on every (re)start. After a few months of operation, it's easy to see how this can take up a lot of space, and my server had been running for nearly four years.
Complicating matters somewhat was the fact that the default name of the binary logs changed at some point (and, according to the current docs, now appears to have changed back. As a result, I have several gigabytes worth of logs using the old naming convention, as well as several gigabytes worth of logs using the newer convention. Yay.
Like I said, recognizing that MySQL was taking up a lot of space is not hard, but I'm paranoid about my data and didn't want to risk losing anything. So, I kept putting it off until I was literally running out of space on a near daily basis. At that point I began doing research and figured out all of the above information. I also found a quick an easy way to fix the problem.
Note: This is meant for a standalone MySQL server. I'm not sure how it may affect replication, so please do not follow these instructions on a replicated server without additional research.
First of all, the binary logs typically reside in /var/lib/mysql/. You can check to see how much space they're currently taking up with this one-liner:
du -hcs /var/lib/mysql/*bin.* | tail -n 1. If it's more than a few hundred megabytes, you may want to continue on.
Next, check to see if you were affected by the name switch like I was. This is unlikely unless you've been running the server for at least a year or so, but it definitely doesn't hurt to check. Look at all
*bin.* files. If they're all named the same, such as mysqld-bin.000001, then you're fine. If you see some with a different name, such as both mysqld-bin.000001 and hostname-bin.000001, then you have an outdated set of logs doing nothing but taking up space. Look at the timestamps of the .index file for each set. One should be very recent (such as today), the other not. Once you've identified the older set, go ahead and delete all of them; they're no longer being used.
Finally, for the current set, login to MySQL as an admin user (eg.,
mysql -u root -p). You'll want to run the following two commands:
mysql> FLUSH LOGS;
mysql> RESET MASTER;
That's it. Depending on the size and number of your logs, those two commands may take a while to run, but the end result is that any unsaved transactions will be flushed to the database, all older logs will be dropped, and the log index will be reset to 1. In my case, these two steps dropped my from 9.6 GB down to about 5 MB. Good stuff.
Of course, this is simply a workaround to the problem, not a proper solution. What I'd really like to do is either automate this process so that I don't have to worry about the logs getting out of control, or even better configure MySQL to automatically flush its own logs after some period of time or it reaches a certain total file size. I haven't found any way to do this just yet, though I admittedly haven't looked too hard. I'd appreciate any recommendations, though.
In my last post I mentioned that I recently had a hardware failure that took down my server. I needed to get it back up and running again ASAP, but due to a large number of complications I was unable to get the original hardware up and running again, nor could I get any of the three other systems I had at my disposal to work properly. Seriously, it was like Murphy himself had taken up residence here. In the end, rather desperate and out of options, I turned to Xen (for those unfamiliar with it, it's similar to VMware or Virtual Box, but highly geared towards server0. I'd recently had quite a bit of experience getting Xen running on another system, so I felt it'd be a workable, albeit temporary, solution to my problem.
Unfortunately, the only working system I had suitable for this was my desktop, and while the process of installing and migrating the server to a Xen guest host was successful (this site is currently on that Xen instance) it was not without it's drawbacks. For one thing, there's an obvious performance hit on my desktop while running under Xen concurrently with my server guest, though fortunately my desktop is powerful enough that this mostly isn't an issue (except when the guest accesses my external USB drive to backup files; for some reason that consumes all CPU available for about 2 minutes and kills performance on the host). There were a few other minor issues, but by far the biggest problem was that the binary nVidia drivers would not install under Xen. Yes, the open source 'nv' driver would work, but that had a number of problems/limitations:
- dramatically reduced video performance, both in video playback and normal 2d desktop usage
- no 3d acceleration whatsoever (remember, this is my desktop system, so I sometimes use it for gaming)
- no (working) support for multiple monitors
- significantly different xorg.conf configuration
In fairness, issues 1 and 2 are a direct result of nVidia not providing adequate specifications for proper driver development. Nonetheless, I want my hardware to actually work, so the performance was not acceptable. Issue 3 was a major problem as well, as I have two monitors and use both heavily while working. I can only assume that this is due to a bug in the nv driver for the video card I'm using (a GeForce 8800 GTS), as dual monitors should be supported by this driver. It simply wouldn't work, though. Issue 4 wasn't that significant, but it did require quite a bit of time to rework it, which was ultimately pointless anyway due to issue 3.
So, with all that said, I began my quest to get the binary nVidia drivers working under Xen. Some basic searches showed that this was possible, but in every case the referenced material was written for much older versions of Xen, the Linux kernel, and/or the nVidia driver. I tried several different suggestions and patches, but none would work. I actually gave up, but then a few days later I got so fed up with performance that I started looking into it again and trying various different combinations of suggestions. It took a while, but I finally managed hit on the special sequence of commands necessary to get the driver to compile AND load AND run under X. Sadly, the end result is actually quite easy to do once you know what needs to be done, but figuring it out sure was a bitch. So, I wanted to post the details here to hopefully save some other people a lot of time and pain should they be in a similar situation.
This guide was written with the following system specs in mind:
- Xen 3.2.1
- Gentoo dom0 host using xen-sources-2.6.21 kernel package
- a non-Xen kernel must also be installed, such as gentoo-sources-2.6.24-r8
- GeForce 5xxx series or newer video card using nvidia-drivers-173.14.09 driver package
Version differences shouldn't be too much of an issue; however, a lot of this is Gentoo-specific. If you're running a different distribution, you may be able to modify this technique to suit your needs, but I haven't tested it myself (if you do try and have any success, please leave a comment to let others know what you did). The non-Xen kernel should be typically left over from before you installed Xen on your host; if you don't have anything else installed, however, you can do a simple
emerge gentoo-source to install it. You don't need to run it, just build against it.
Once everything is in place, and you're running the Xen-enabled (xen-sources) kernel, I suggest uninstalling any existing binary nVidia drivers with
emerge -C nvidia-drivers. I had a version conflict when trying to start X at one point as the result of some old libraries not being properly updated, so this is just to make sure that the system's in a clean state. Also, while you can do most of this while in X while using the nv driver, I suggest logging out of X entirely before the
Here's the step-by-step guide:
uname -rto verify the version of your currently running Xen-enabled kernel; eg., mine's 2.6.21-xen
- verify that you have both Xen and non-Xen kernels installed:
cd /usr/src/ && ls -l
- eg., I have both linux-2.6.21-xen and linux-2.6.24-gentoo-r8
- create a symlink to the non-Xen kernel:
ln -sfn linux-2.6.24-gentoo-r8 linux
- install the nVidia-drivers package, which includes the necessary X libraries:
emerge -av nvidia-drivers
- this will also install the actual driver, but it'll be built and installed for the non-Xen kernel, not your current Xen-enabled kernel
- determine the specific name and version of the nVidia driver package that was just installed; this can be found by examining the output of
emerge -f nvidia-drivers(look for the NVIDIA-Linux-* line)
- extract the contents of the nVidia driver package:
bash /usr/portage/distfiles/NVIDIA-Linux-x86_64-173.14.09-pkg2.run -a -x
- change to the driver source code directory:
- build the driver for the currently-running Xen-enabled kernel:
IGNORE_XEN_PRESENCE=y make SYSSRC=/lib/modules/`uname -r`/build module
- assuming there are no build errors (nvidia.ko should exist), install the driver:
mkdir /lib/modules/`uname -r`/video
cp -i nvidia.ko /lib/modules/`uname -r`/video/
- if necessary, log out of X, then load the driver:
- if necessary, reconfigure xorg.conf to use the nvidia binary driver rather than the nv driver
- test that X will now load properly with
- if appropriate, start (or restart) the display manager with
Assuming all went well, you should now have a fully functional and accelerated desktop environment, even under a Xen dom0 host. W00t. If not, feel free to post a comment and I'll try to help if I can. You should also hit up the Gentoo Forums, where you can get help from people far smarter than I.
I really hope this helps this helps some people out. It was a royal pain in the rear to get this working, but believe me, it makes a world of difference when using the system.