LegRoom.net Kick back and relax

I had an issue with free disk space (or, more appropriately, a lack there of) on my server a while back. After some investigation, I discovered that my MySQL had ballooned in size to nearly 10 GB. Actually, figuring out that the /var/lib/mysql directory was taking up so much space wasn't that hard, but understanding why and what to do about it took a while (yes, I'm sometimes slow about such things).

It turns out I had two issues. The first is that MySQL configuration, by default, maintains binary logs. These logs "contain all statements that update data or potentially could have updated it (for example, a DELETE which matched no rows). Statements are stored in the form of 'events' that describe the modifications. The binary log also contains information about how long each statement took that updated data."^[1] This is fine and all, but (again by default) these log files are never deleted. There is a (configurable) max file size for each log, but MySQL simply rolls over to a new log when it's reached. Additionally, MySQL rolls over to a new log file on every (re)start. After a few months of operation, it's easy to see how this can take up a lot of space, and my server had been running for nearly four years.

Complicating matters somewhat was the fact that the default name of the binary logs changed at some point (and, according to the current docs, now appears to have changed back. As a result, I have several gigabytes worth of logs using the old naming convention, as well as several gigabytes worth of logs using the newer convention. Yay.

Like I said, recognizing that MySQL was taking up a lot of space is not hard, but I'm paranoid about my data and didn't want to risk losing anything. So, I kept putting it off until I was literally running out of space on a near daily basis. At that point I began doing research and figured out all of the above information. I also found a quick an easy way to fix the problem.

Note: This is meant for a standalone MySQL server. I'm not sure how it may affect replication, so please do not follow these instructions on a replicated server without additional research.

First of all, the binary logs typically reside in /var/lib/mysql/. You can check to see how much space they're currently taking up with this one-liner: du -hcs /var/lib/mysql/*bin.* | tail -n 1. If it's more than a few hundred megabytes, you may want to continue on.

Next, check to see if you were affected by the name switch like I was. This is unlikely unless you've been running the server for at least a year or so, but it definitely doesn't hurt to check. Look at all *bin.* files. If they're all named the same, such as mysqld-bin.000001, then you're fine. If you see some with a different name, such as both mysqld-bin.000001 and hostname-bin.000001, then you have an outdated set of logs doing nothing but taking up space. Look at the timestamps of the .index file for each set. One should be very recent (such as today), the other not. Once you've identified the older set, go ahead and delete all of them; they're no longer being used.

Finally, for the current set, login to MySQL as an admin user (eg., mysql -u root -p). You'll want to run the following two commands:
mysql> FLUSH LOGS;
mysql> RESET MASTER;

That's it. Depending on the size and number of your logs, those two commands may take a while to run, but the end result is that any unsaved transactions will be flushed to the database, all older logs will be dropped, and the log index will be reset to 1. In my case, these two steps dropped my from 9.6 GB down to about 5 MB. Good stuff.

Of course, this is simply a workaround to the problem, not a proper solution. What I'd really like to do is either automate this process so that I don't have to worry about the logs getting out of control, or even better configure MySQL to automatically flush its own logs after some period of time or it reaches a certain total file size. I haven't found any way to do this just yet, though I admittedly haven't looked too hard. I'd appreciate any recommendations, though.

• ^[1] http://dev.mysql.com/doc/refman/5.0/en/binary-log.html

In my last post I mentioned that I recently had a hardware failure that took down my server. I needed to get it back up and running again ASAP, but due to a large number of complications I was unable to get the original hardware up and running again, nor could I get any of the three other systems I had at my disposal to work properly. Seriously, it was like Murphy himself had taken up residence here. In the end, rather desperate and out of options, I turned to Xen (for those unfamiliar with it, it's similar to VMware or Virtual Box, but highly geared towards server0. I'd recently had quite a bit of experience getting Xen running on another system, so I felt it'd be a workable, albeit temporary, solution to my problem.

Unfortunately, the only working system I had suitable for this was my desktop, and while the process of installing and migrating the server to a Xen guest host was successful (this site is currently on that Xen instance) it was not without it's drawbacks. For one thing, there's an obvious performance hit on my desktop while running under Xen concurrently with my server guest, though fortunately my desktop is powerful enough that this mostly isn't an issue (except when the guest accesses my external USB drive to backup files; for some reason that consumes all CPU available for about 2 minutes and kills performance on the host). There were a few other minor issues, but by far the biggest problem was that the binary nVidia drivers would not install under Xen. Yes, the open source 'nv' driver would work, but that had a number of problems/limitations:

1. dramatically reduced video performance, both in video playback and normal 2d desktop usage
2. no 3d acceleration whatsoever (remember, this is my desktop system, so I sometimes use it for gaming)
3. no (working) support for multiple monitors
4. significantly different xorg.conf configuration

In fairness, issues 1 and 2 are a direct result of nVidia not providing adequate specifications for proper driver development. Nonetheless, I want my hardware to actually work, so the performance was not acceptable. Issue 3 was a major problem as well, as I have two monitors and use both heavily while working. I can only assume that this is due to a bug in the nv driver for the video card I'm using (a GeForce 8800 GTS), as dual monitors should be supported by this driver. It simply wouldn't work, though. Issue 4 wasn't that significant, but it did require quite a bit of time to rework it, which was ultimately pointless anyway due to issue 3.

So, with all that said, I began my quest to get the binary nVidia drivers working under Xen. Some basic searches showed that this was possible, but in every case the referenced material was written for much older versions of Xen, the Linux kernel, and/or the nVidia driver. I tried several different suggestions and patches, but none would work. I actually gave up, but then a few days later I got so fed up with performance that I started looking into it again and trying various different combinations of suggestions. It took a while, but I finally managed hit on the special sequence of commands necessary to get the driver to compile AND load AND run under X. Sadly, the end result is actually quite easy to do once you know what needs to be done, but figuring it out sure was a bitch. So, I wanted to post the details here to hopefully save some other people a lot of time and pain should they be in a similar situation.

This guide was written with the following system specs in mind:

• Xen 3.2.1
• Gentoo dom0 host using xen-sources-2.6.21 kernel package
  • a non-Xen kernel must also be installed, such as gentoo-sources-2.6.24-r8
• GeForce 5xxx series or newer video card using nvidia-drivers-173.14.09 driver package

Version differences shouldn't be too much of an issue; however, a lot of this is Gentoo-specific. If you're running a different distribution, you may be able to modify this technique to suit your needs, but I haven't tested it myself (if you do try and have any success, please leave a comment to let others know what you did). The non-Xen kernel should be typically left over from before you installed Xen on your host; if you don't have anything else installed, however, you can do a simple emerge gentoo-source to install it. You don't need to run it, just build against it.

Once everything is in place, and you're running the Xen-enabled (xen-sources) kernel, I suggest uninstalling any existing binary nVidia drivers with emerge -C nvidia-drivers. I had a version conflict when trying to start X at one point as the result of some old libraries not being properly updated, so this is just to make sure that the system's in a clean state. Also, while you can do most of this while in X while using the nv driver, I suggest logging out of X entirely before the modprobe line.

Here's the step-by-step guide:

1. Run uname -r to verify the version of your currently running Xen-enabled kernel; eg., mine's 2.6.21-xen
2. verify that you have both Xen and non-Xen kernels installed: cd /usr/src/ && ls -l
   • eg., I have both linux-2.6.21-xen and linux-2.6.24-gentoo-r8
3. create a symlink to the non-Xen kernel: ln -sfn linux-2.6.24-gentoo-r8 linux
4. install the nVidia-drivers package, which includes the necessary X libraries: emerge -av nvidia-drivers
   • this will also install the actual driver, but it'll be built and installed for the non-Xen kernel, not your current Xen-enabled kernel
5. determine the specific name and version of the nVidia driver package that was just installed; this can be found by examining the output of emerge -f nvidia-drivers (look for the NVIDIA-Linux-* line)
6. extract the contents of the nVidia driver package: bash /usr/portage/distfiles/NVIDIA-Linux-x86_64-173.14.09-pkg2.run -a -x
7. change to the driver source code directory: cd NVIDIA-Linux-x86_64-173.14.09-pkg2/usr/src/nv/
8. build the driver for the currently-running Xen-enabled kernel: IGNORE_XEN_PRESENCE=y make SYSSRC=/lib/modules/`uname -r`/build module
9. assuming there are no build errors (nvidia.ko should exist), install the driver:
   • mkdir /lib/modules/`uname -r`/video
   • cp -i nvidia.ko /lib/modules/`uname -r`/video/
   • depmod -a
10. if necessary, log out of X, then load the driver: modprobe nvidia
11. if necessary, reconfigure xorg.conf to use the nvidia binary driver rather than the nv driver
12. test that X will now load properly with startx
13. if appropriate, start (or restart) the display manager with /etc/init.d/xdm start

Assuming all went well, you should now have a fully functional and accelerated desktop environment, even under a Xen dom0 host. W00t. If not, feel free to post a comment and I'll try to help if I can. You should also hit up the Gentoo Forums, where you can get help from people far smarter than I.

I really hope this helps this helps some people out. It was a royal pain in the rear to get this working, but believe me, it makes a world of difference when using the system.

In case anyone was wondering why legroom.net was offline for the past few days, it's because of a hardware failure that occurred Thursday morning. The server hosting legroom.net failed due to a yet unknown hardware issue. As a result, all web/mail/database/file/etc. services have been unavailable since then. As of early Sunday morning, I just finished migrating all of the data from the old server (well, I hope I got all of it) to a new OS install on a different, though temporary, system. As far as I can tell, all internet functionality should be restored at this point, including e-mail.

Needless to say, this was a major inconvenience, but I'm reasonably certain I didn't lose any data. However, if you happen find any errors on the site, or any missing content, please let me know ASAP. Also, if you sent me an e-mail at any point from Thursday morning to Sunday night, please resend it as I may not have received it.

This site will probably go down a couple more times, albeit very briefly, over the next couple of days as I continue working on the migration. I still have a few remaining issues to take care of. I'm hoping that by Tuesday the site will be up again for good (at least until the next migration to a permanent home...).