Software

Download/Saving Streaming Videos

Submitted by jbreland on Thu, 03/29/2007 - 10:16

Streaming (aka, embedded) video has been around since the beginning of the internet, and while it can certainly be a very useful technology, I hate companies and websites that require you to stream the video and don't even offer an option to download it. What if you want to view it multiple times? What if you want to show a friend or co-worker? Tough! You have to stream it (re-download it) every single time you watch it. The only possibly reason for doing this that I can think of is that it supposedly allows the hosting site more control over the video, but it creates a severe inconvenience for users, prevents a number of possible customers from watching it in the first place (eg, if they're not running a "blessed" operating system, browser, or plugin), and exponentially increases bandwidth costs for the hosting provider. Additionally, the very fact that the movie has to be sent to the client to allow it to be played and displayed means that the client still gets a copy and can still save it anyway, just with much greater hassle.

If you want to save a streaming video from a site that doesn't offer a proper download link, you have a few options. Like most things in technology these range from easy but limited to difficult but extremely flexible. For now, I'm going to discuss the easy approach. :-) I'm also going to assume you're using Mozilla Firefox (if you're not, you really should be).

There are a number of Firefox extensions available that can greatly assist with saving embedded/streaming video clips. A simple search for "download video" on the Firefox Add-ons returns 19 extensions. I've personally used VideoDownloader in the past, and it worked well.

If you don't want to install a new extension just for downloading videos, and option is to use Greasemonkey scripts. Greasemonkey is a really powerful extensions that allows users to create or install custom scripts that can change the behavior of any web page. In this case, a Greasemonkey script can analyze the page for embedded video clips, then automatically add a download link to that video; simply click the download link to save the video. Two great scripts for downloading embedded videos are Apple Trailer Download and Download Video (though I use this version of Download Video). Note that you must first install Greasemonkey before you can install these scripts. You can find a lot more useful Greasemonkey scripts on my Mozilla Firefox Tips and Tricks page, as well as a massive collection of contributed scripts on Userscripts.org.

The primary limitation with the extension/script method is that it's often limited to particular sites. Apple Trailer Download, for example, is limited to just movie trailers hosted on the Apple Movie Trailers website. Download Video and VideoDownloader both support a number of different sites (including the most popular, YouTube and Google Video), but are still limited to only sites they "know" about. I'll follow up this article with more advanced techniques and suggestions that should help you save just about any embedded video.

Universal Extractor Translation Updates

Submitted by jbreland on Fri, 03/23/2007 - 09:30

I've added several new language files for Universal Extractor since the 1.5 release, including (as of this post): Korean, Portuguese, Portuguese (Brazilian), Romanian, and Turkish. These files can be downloaded from the Universal Extractor page and added to your current copy of Universal Extractor if you'd like, and they will be included in the next release.

I'd also like to point out that all 18 (and counting!) language files have been voluntarily translated and contributed by UniExtract users that simply wanted to use the program in their native language. I'd like to send out a big thanks to all contributors for improving Universal Extractor by making it much more accessible to foreign users. Thanks!

If a translation doesn't currently exist for your own language, and you have a good understanding of English, consider contributing a translation yourself. The process itself is very straightforward; simply copy English.ini and follow the instructions at the top of the file to translate the English strings into your native language. If you're gracious enough to contribute the new language file for all other UniExtract users, I'll be happy to add your name to the list of contributors, as well as link back to your own website (if you'd like).

The only "catch" is that I'd ask you to keep the file updated as I release newer versions of Universal Extractor. These updates typically involve only minor changes or additions, and are only necessary as often as I release new versions (which, honestly, is not that often). I can only speak English myself, so I rely entirely on the generosity of UniExtract users to maintain the language files.

Thanks again to everyone that has contributed a language file to Universal Extractor.

Status of E-mails Regarding LegRoom Software

Submitted by jbreland on Wed, 03/21/2007 - 23:07

Several people have e-mailed me regarding a couple bugs in my programs, as well as to submit new translation files for Universal Extractor. I'm behind on a lot of my e-mail right now due to some other priorities, so please forgive the delay in response. I'm reading each and every e-mail that comes my way, and will reply as soon as I get the chance.

For everyone else that may have found a bug or wants to suggest a new feature, please consider using the new LegRoom Support Forum instead. I won't necessarily be able to reply any faster, but the information you provide will at least be visible to all other users, which could help them if they're having a similar problem or were thinking about requesting a similar feature. Plus, it helps cut down on my e-mail. :-)

I'm going to try to knock out a few UniExtract-related e-mails before going to bed tonight.

Small update for Firefox Tips and Tricks

Submitted by jbreland on Tue, 03/20/2007 - 13:43

Update: 03/21/2007 14:41
In a rather ironic mistake, I accidentally specified a bad link to the Firefox Tips and Tricks page. Oops. :-) That's been corrected.

I just noticed that the links to several of my modified Greasemonkey scripts were not updated to reflect the new site layout. I fixed these links on the Mozilla Firefox Tips and Tricks page so that they now point to the correct location.

Sorry for the inconvenience.

Firefox and Thunderbird Tips and Tricks pages updated

Submitted by jbreland on Wed, 03/14/2007 - 02:01

I've updated my Firefox and Thunderbird Tips and Tricks pages. I actually updated them a couple weeks ago during the website migration, but I was limiting myself to news posts only about the migration itself at the time. So, if you've already checked it out recently, there's no need to do so again.

For those of you that have not check it out recently, the main changes involved updating the extension and Greasemonkey script lists, as well as posting the latest copies of my user.js and prefs.js files. This update is current as of Firefox 2.0.0.2 and Thunderbird 1.5.0.10.

Two very cool applications - cmdTotal and TrID

Submitted by jbreland on Sun, 02/11/2007 - 03:28

I'm currently working on the next version of Universal Extractor. During my research into what features should be included in the new release, I came across a couple of new applications that will provide a substantial benefit to Universal Extractor.

In no particular order, the first is cmdTotal. This tiny assembly program "is a generic approach to make it possible to use any Total Commander WCX packer plugin from a command line." This has been something of a personal holy grail for me, as there are a number of extremely useful plugins for Total Commander that could greatly benefit Universal Extractor, but in the past could never be used because Total Commander itself is a shareware product. cmdTotal changes that by letting my use these plugins directly from Universal Extractor. Thanks to cmdTotal and the associated plugins, I've been able to add support for five entirely new formats, as well as expand support for six existing formats. I'd like to thank Adam at KaKeeware for making this possible, as well as the personal assistance he provided to help make it work correctly with Universal Extractor.

Up next is TrID. TrID is a "utility designed to identify file types from their binary signatures." The ability to identify filetypes based on their signature rather than relying on their extension has been another long standing item on my todo list. I had looked at a few such applications in the past, as well as toyed with the idea of writing my own, but nothing was quite appropriate for integration with Universal Extractor. When I came across TrID, however, I instantly realized I had found what I was looking for. This program integrates very easily with Universal Extractor, recognizes a large number a filetypes by default (currently 2377), and is very extensible. I've been able to add definitions for several additional filetypes to TrID, as well as tune existing definitions to better suit my requirements. This has allowed me to greatly increase the reliability and robustness of Universal Extractor's filetype detection, and instantly expands its file support by adding recognition of any file that matches a particular supported signature regardless of extension. For example, I was recently asked to add support for OpenOffice documents, which use the ZIP format for storage. I tested a few samples with the 1.5 beta version, and it worked right away without the need for any code changes! I'd like to thank Marko Pontello for writing this immensely useful application, and for giving me permission to include it with Universal Extractor.

Thanks again guys!

Here are the full links to the product pages. Check them out!

cmdTotal - http://www.kakeeware.com/i_cmdtotal.php
TrID - http://mark0.net/soft-trid-e.html

Firefox 2.0 Feed Preview Behavior (and closing windows)

Submitted by jbreland on Fri, 01/05/2007 - 13:20

For those of you that may be unaware, Firefox includes new Feed Preview capabilities in version 2.0. I discussed it briefly in a previous post:

A major issue that I have with RSS Preview is that Firefox will display this preview page even if the webmaster has already written an XSL transform to display the feed in human-readable form. I find this very frustrating, as I spent a lot of time styling the RSS feed for my site, making sure the look and feel matches that of the rest of the site, takes advantage of certain RSS elements available on my site that may not be available on others, etc. However, Firefox 2.0 ignores all of this and instead displays the feed using its own preview style. While this is a great feature for sites that only display raw XML, I strongly feel that Firefox should respect the webmaster's design if he's taken the time to create and specify a particular style/transform for the feed. At the very least there should be an option for users to enable the built-in preview style for all feeds rather than just raw feeds, with it set to only use the preview style for raw feeds by default.

As stated above, I think this is a great feature for sites that simply display raw XML output - I much prefer Firefox's Feed Preview page over that, as I'm sure most other Firefox users do. However, I still have a major problem with the fact that it will always override an RSS feed with this preview page, regardless of whether or not the developer has supplied his own style or extra content for the page. It's nice to know that I'm not alone on this, as evidenced by this 60+ post thread on the MozillaZine forums. Sadly, though, it seems that the core developers responsible for this change (ie, the ones that "matter") feel that their way is the way it should be done, users be damned. It's actually rather fascinating - read through that thread and count up how many people posted their objections, vs. how many people think it's a good idea. Then read through this bug report and do the same (also count the number of duplicates). Then read through this newsgroup thread and do the same. Anyone see a pattern?

I "solved" this problem for the feed on my own site with this lovely workaround added to the top of my feed:

This is a waste of space and bandwidth in order to appease Firefox 2.0's and Internet Explorer 7's feed sniffing.
By adding this extra and completely unnecessary text to the top of my feed, Firefox and IE7 will display the feed using
my own XSL stylesheet, as it should to begin with, rather than using it's built-in Feed Preview functionality.
You can thank the fine folks at Microsoft and the Mozilla Corporation for for the brain-dead implementation of what should be a very useful feature.

Thanks, Mozilla. Thanks, Microsoft. The reason I'm posting about this again today is because I recently came across some comments that seemed very familiar in VMware's RSS feed:

This is 512 bytes of nonsense, since the Firefox 2 developers, in one of the strangest decisions ever, decided they would obsolete XML styles by overriding them without permission. Furthermore, the developers appear to be disinterested in fixing this. Therefore, we use the unofficial workaround, which includes filling up the first 512 bytes of a document so that the sniffer doesn't encounter the RSS tag. I really enjoy using Firefox, but this particular behavior really annoys me! Anyway, since I'm almost at 512 characters, I'm going to ramble on for another minute in this comment, and then, without further adue, present you with a valid XML feed.

Thanks, Mozilla. In all seriousness, I truly appreciate the user-centric focus you take with your browser. The fact that I have a custom-made Get Firefox logo on my navbar, which is the one and only image/banner/link on my site that even remotely resembles an advertisement, should alone speak volumes of this. However, when such a large number of your own users come forward to ask that you fix an issue - not even remove it, just make it optional - please consider actually listening to what they say rather than responding with the same "our way is better than yours" comments over and over.

And while we're on the topic, please, for the love of all that is holy, fix this damn Tab/Window close bug. Once again, with so many of your own users reporting it as a problem (again, count the number of pro vs. con comments, as well as the number of duplicate bugs posted), consider the fact that the few of you who implemented this change just may be wrong. And once again, people are simple asking for an option here - not to completely do away with it, since some people seem to prefer this behavior, but make it optional for those that don't. At the very least, consider using the patch that I've already written.

Ok, that's enough ranting for now. I feel much better. :-)

Is Firefox/Google Spying on Your News Feeds? (Update)

Submitted by jbreland on Wed, 10/25/2006 - 11:52

Update - 11/02/2006:
There have been a couple developments since I first published this story. I wanted to put in a quick update to let everyone know what's been happening.

First of all, I'm very pleased to report that this was, in fact, an oversight on the part of the Firefox developers, and they are currently working on a solution to prevent this behavior.

Secondly, it seems that the same issue occurs with Yahoo in addition to Google. This was reported by one of the visitors to this site (thanks, dj), and later confirmed by the German news site Heise Online (here's an English translation of the article).

I'd like to thank the Firefox developers for taking such quick action to fix this issue, as well as those of you that helped raise awareness of it.

----------

I've been testing out the latest release candidate for Firefox 2.0 (which has since been officially released). One of the new features in Firefox 2.0 is ?Previewing and subscribing to Web feeds", which allows users to (according to the release notes):

...decide how to handle Web feeds, either subscribing to them via a Web service or in a standalone RSS reader, or adding them as Live Bookmarks. My Yahoo!, Bloglines and Google Reader come pre-loaded as Web service options, but users can add any Web service that handles RSS feeds.

The nice thing about this feature is the ability to preview feeds. When the user clicks on an RSS or Atom link, Firefox renders the feed in a human-readable format, rather than simply displaying the raw XML as it had in the past*. At the top of the feed page, the user has the choice of subscribing to the feed through different services, as described in the above quote. This is where our story gets interesting.

Before continuing, though, let me provide a brief bit of background on my browsing habits. I always browse the web with Firefox set to prompt me to accept any new cookies. I also use a custom installation package that presets my preferences, so the cookie settings are active on initial load of the browser. This is the reason I noticed this issue to begin with, as I'll explain later.

So, I have Firefox installed and configured, and I begin testing out the new RSS features. The first time I hit a feed (in this case, my own LegRoom.net feed) I was prompted to accept a cookie from fusion.google.com. I didn't think much of it at first and instinctively denied it, but then I noticed the same prompt after a reinstall, and then again on each other feed I visited. This was clearly being triggered by Firefox itself and not by the feed website.

I couldn't find any explanation for this behavior, so at this point I did what any good little geek would do: I fired up a copy of Wireshark (formerly Ethereal) and started sniffing network traffic. After a bit of analysis, I found that immediately after every feed page is loaded, Firefox makes call to Google. Specifically, this is the HTTP data that is exchanged (from Wireshark):

No.  Time      Source   Destination   Protocol  Info
32   3.456833  x.x.x.x  72.14.203.99  HTTP      GET /favicon.ico HTTP/1.1
Hypertext Transfer Protocol
   GET /favicon.ico HTTP/1.1\r\n
      Request Method: GET
      Request URI: /favicon.ico
      Request Version: HTTP/1.1
   Host: fusion.google.com\r\n
   User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0\r\n
   Accept: image/png,*/*;q=0.5\r\n
   Accept-Language: en-us,en;q=0.5\r\n
   Accept-Encoding: gzip,deflate\r\n
   Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n
   Keep-Alive: 300\r\n
   Connection: keep-alive\r\n
   Referer: http://www.legroom.net/backend.php\r\n

No.  Time      Source        Destination  Protocol  Info
36   3.587770  72.14.203.99  x.x.x.x      HTTP      HTTP/1.1 302 Found (text/html)
Hypertext Transfer Protocol
   HTTP/1.1 302 Found\r\n
      Request Version: HTTP/1.1
      Response Code: 302
   Location: http://www.google.com/favicon.ico\r\n
   Set-Cookie: PREF=ID=7e3b29ec472e1e47:TM=1161728606:LM=1161728606:S=4O7AZZG6VrvWd_V5; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com\r\n
   Content-Type: text/html\r\n
   Server: igfe\r\n
   Transfer-Encoding: chunked\r\n
   Content-Encoding: gzip\r\n
   Date: Tue, 24 Oct 2006 22:23:26 GMT\r\n
   Cache-Control: private, x-gzip-ok=""\r\n
   \r\n
   HTTP chunked response
      Data chunk (197 octets)
         Chunk size: 197 octets
         Data (197 bytes)
         Chunk boundary
      Data chunk (10 octets)
         Chunk size: 10 octets
         Data (10 bytes)
         Chunk boundary
   Content-encoded entity body (gzip): 207 bytes -> 230 bytes
Line-based text data: text/html
   <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
   <TITLE>302 Moved</TITLE></HEAD><BODY>
   <H1>302 Moved</H1>
   The document has moved
   <A HREF="http://www.google.com/favicon.ico">here</A>.
   </BODY></HTML>

Based on this information, it appears that Firefox is contact Google in order to download the icon used in the Subscription menu select box on the feed preview page. In the process, Google sets a cookie.

Ok, on the surface this sounds innocent enough, but the more I thought about it the more it bothered me. For starters, why would Firefox have to download the favicon Google to begin with? There are three other icons in the same Subscription menu: Live Bookmarks, Bloglines, and My Yahoo. Firefox is obviously able to load local favicons for those services, so why would it possibly need to download Google's favicon from Google's servers? From a technical perspective, it's both wasteful and inefficient.

Second, why is it necessary to include all of the extra data to grab the favicon? Eg, why does Google have to be told which feed I'm browsing (the referer attribute) or what my exact user browser version is (the User-Agent attribute)? The favicon could just as easily be downloaded without providing this information.

Third, why does Google have to set a cookie when providing the favicon? Cookies are used for tracking and session management. In this case, all I'm doing is downloading a graphic. That's it. There is no session management to perform. Again, from a technical perspective this is unnecessary, wasteful, and inefficient.

So at this point, in the best case Google knows your browser version, the page you were visiting, the time you visited, and your IP address for correlation. Now let's examine the cookie. Notice that it's a root domain cookie (.google.com) and not something separate for fusion.google.com. Notice also that it doesn't expire until 2038. Assuming you accept the cookie, which almost everyone will (explained below), Google can also correlate your feed views with all other Google services through the .google.com cookie. I cannot think of any technical need for this, either.

With all of that said, let me stress that I'm not trying to sound any conspiracy theories here. It may very well be some technical limitation or a simple oversight. After all, Google already knows what you search for, what and who you e-mail, who you chat with and what you chat about, who you socialize with, what your social life looks like, what files are stored on your computer, what documents and spreadsheets you work on, what you blog about, what pictures you share, what you shop for, what newsgroups you read, what current events you keep up with, how you run your website, what stocks you monitor, what books you like to read, and, of course, what newsfeeds you read.

Considering all of the above, is there really much benefit to tracking your feed usage through Firefox? To be honest, I just don't know. However, given the fact that Google is very much in the business of collecting data from its users, and that Google has a very well-known relationship with the Mozilla Foundation/Corporation, I feel that this was done intentionally. Furthermore, this tracking is always enabled by default, and there's no way of stopping it.

As I mentioned a couple times above, the only reason I noticed this issue is because I use a customized installer for Firefox that, among other things, begins blocking cookies on initial load. This is important because the default Firefox start page is a Firefox-branded Google search page. So, the first time Firefox is launched it will contact Google and accept Google's cookie (default behavior). Even if the first thing you do after launching Firefox for the first time is to adjust your cookie preferences to prompt you before accepting anything, the Google cookie has already been accepted. Again, I don't think this is part of some conspiracy, but it does compound the issue by making detection of the call to Google's servers that much more difficult.

My biggest question at this point is simply, "Why?". Does anyone know why Firefox does this? Has anyone seen any other reference to this, or acknowledgment of this behavior? I find it very odd that Firefox 2.0 loads the Google favicon, and only the Google favicon, remotely for the Feed Preview page, when the favicons for three other services displayed in the same page are loaded locally. I can think of no technical reason for this, and the only non-technical reason that comes to mind is that Google wants to track your newsfeed habits, and the Firefox made it happen.

What do you think? Am I being overly paranoid here (it's certainly been known to happen)? Feel free to leave your comments below, or just send me an e-mail. (Unfortunately, due to issues with comment spam, I had to disable anonymous comments. You will need to sign in to post a comment.)

-----

*A major issue that I have with RSS Preview is that Firefox will display this preview page even if the webmaster has already written an XSL transform to display the feed in human-readable form. I find this very frustrating, as I spent a lot of time styling the RSS feed for my site, making sure the look and feel matches that of the rest of the site, takes advantage of certain RSS elements available on my site that may not be available on others, etc. However, Firefox 2.0 ignores all of this and instead displays the feed using its own preview style. While this is a great feature for sites that only display raw XML, I strongly feel that Firefox should respect the webmaster's design if he's taken the time to create and specify a particular style/transform for the feed. At the very least there should be an option for users to enable the built-in preview style for all feeds rather than just raw feeds, with it set to only use the preview style for raw feeds by default.

Future versions of Eudora to be based on Thunderbird?

Submitted by jbreland on Wed, 10/11/2006 - 16:13

Wow, I didn't see this one coming (though it's very welcome news):

...future versions of Eudora? will be based upon the same technology platform as the open source Mozilla Thunderbird? email program. Future versions of Eudora will be free and open source, while retaining Eudora's uniquely rich feature set and productivity enhancements. QUALCOMM and Mozilla will each participate in, and continue to foster development communities based around the open source Mozilla project, with a view to enhancing the capabilities and ease of use of both Eudora and Thunderbird.

Qualcomm press release:

http://www.qualcomm.com/press/releases/2006/061011_project_collaboration_mozilla.html

Also worth checking out is the Penelope project page, which is the official Mozilla project that will be dedicated to this effort:

http://wiki.mozilla.org/Penelope