ImageMagick memory problems

I noticed my Windows development machine had slowed to a crawl recently, and I assumed it was something to do with the new plugins to convert images that are in development. Turns out it was, but not in an obvious way. Because the plugin requires ImageMagick, any EPrints process will load ImageMagick; this includes the scheduled task to run the indexer which I had installed and promptly forgotten about.

It looks like that because I had installed ImageMagick after I created this task, its environment didn't include the necessary paths for the Perl module to load correctly. It could find the Perl component, but not the XS part. Because of the way it autoloads unknown functions, this was causing an infinite recursion every time the indexer ran, which Perl eventually caught but not before it had used 2GB of my swap space and all my physical memory.

Recreating the task (in fact, just changing a property and changing it back) seems to have got rid of the problem. Maybe tasks store the path from the environment when they are created. This just proves why scheduling the indexer has to be done carefully; the Unix version has a complicated script to check that everything's running successfully, which I'm hoping Task Scheduler can duplicate under Windows.

On the positive side, the new plugins to convert images using the Perl API look to be working fine. Next stop, GhostScript—and the Windows version is getting closer to feature complete relative to Unix.

New release, indexer progress

A new Windows package has now been released. This release fixes a few bugs and has a new graphical installer. Download it from http://files.eprints.org/279/.

Good progress has been made with the indexer. Metadata can now be indexed periodically, and so can full text of plain ASCII documents thanks to a bug fix. I need to write some file format plugins to convert other formats, which might involve a change in the way plugins are configured in the core. The next release (soon I hope) will have the indexer included; until then I'll post a script which can be configured manually.

New release coming

I'm preparing a new package for release, hopefully within the next week. There are compatibility bugs in the core which I'd like to have fixed first, in case they sneak into future releases; I can't commit into the core code so I need to wait until another developer gets back.

In the meantime I'm testing the indexer. No full text yet, but it works for metadata.

Indexing

I made a start on the indexer a while ago, and while it could do a single pass I haven't yet looked at how to make it a background task. It's similar to subscriptions, which are done with Windows Task Scheduler, but the Unix indexer script takes great care to interrupt the indexer if it crashes. I need to investigate how Task Scheduler works to see if it can do this.

On top of this is the problem of indexing full text. Most of the formats the Unix version can understand are converted using Unix specific tools; these need to be converted to pure Perl, Windows specific, or portable tools. I already found an module which should do the trick for HTML, and GhostScript will probably work for PS and PDF.