Multisearch Log

These are the log files from 2 - 6 June 2008. For additional log files, please return to the main log page.

Friday, 6 June
Nimbus and Pileus had some issues, and I found again that two instances of Tomcat were running on BOTH of them! I shut them down and restarted one instance, but Pileus is still loading very slowly. The reason this is an issue is that the way Multisearch works is that it loads the Configuration file first, which involves checking to ensure all the servers are valid URIs... which, of course, means it is taking over 30 minutes to do this on Pileus currently, when before it would only take about 2 minutes.

Snowy still doesn't think I have a home directory. So, I started working on the code for selecting the servers provided by Chris's lists.

Snowy came back up around 4pm, and I have the basic code for the dynamic and static hosts figured out.

Thursday, 5 June
I spent most of the day putting backends on Pileus, and now there are 333 live. I tried to load some onto Nimbus, but it has very limited space. Since it's the server Multisearch is currently running off of, I didn't want to load it with backends. What if I need to add more whizzbangs to Nimbus? I already know I have to add some processing items to run our TREC code...

Unfortunately, now Pileus is taking a long time to respond and load. I'm not sure if this is a Tomcat problem or if it is something else... Maybe 333 backends was too much for it? I can't really tell. It might also just be more than one instance of Tomcat is running. I'm looking up the commands for that now.

ps aux | grep cat
            Shows the process of Tomcat running

kill [#pid#]       or kill -9 [#pid#]
            Kills process with the pid # listed

It turns out Pileus DID have two instances of Tomcat running, but getting it down to one didn't speed it up much. I've dropped a note to Greg about it.

I tried to load some backends onto Nimbus, and, sure enough, it's full! So I went over to load them into Snowy. I was working and it froze up, and I got this error when I tried to log in later:

Could not chdir to home directory /home/mccormic: No such file or directory
/usr/bin/X11/xauth: error in locking authority file /home/mccormic/.Xauthority

I dropped Greg a note about this. Since it was 9pm and I had busted Pileus, filled Nimbus, and broken Snowy, I thought it might be time to head home...

Wednesday, 4 June
Since I have Multisearch up and running, I'm focusing now on putting up all the different backends on the machines. Right now I'm looking at 1,000, which I think should be entirely possible. I just need to set to it.

I'm a little worried about space. I'm not sure Pileus is happy with all the space I'm using on it. I'm hoping new Snowy – now that Java is installed – will be the major backend support because it is huge.

Tuesday, 3 June
My work computer was down due to a file system issue, so I worked primarily from home for the day. I had trouble figuring out the problem, because there were multiple issues with updating Lucene.

Two versions of Lucene were being used in the servlet. Since Tomcat needs to put all jar files in $CATALINA_HOME/common/lib, and Axis (which is what OGSA-DAI is built off of) needs jar files in $CATALINA_HOME/webapps/axis/WEB-INF/lib, the two versions of Lucene did not agree.

All jars in $CATALINA_HOME/common/lib are also in the CLASSPATH, so the new code changes to ARSCLuceneSearchActivity.java would compile properly. However, the backends are Axis services and use the code provided in $CATALINA_HOME/webapps/axis/WEB-INF/lib, so the .jar file had to be dropped in both places.

Two versions of ARSC's code also existed in the servlet. There are two jar files provided by Multisearch – ARSC.jar (renamed LuceneAccessor.jar) and LuceneService.jar. Again, both of these are in two places

Tomcat: $CATALINA_HOME/common/lib
Axis: $CATALINA_HOME/webapps/axis/WEB-INF/lib

The CLASSPATH points to Tomcat, but the actual services use Axis. After compiling and recompiling the code, I realized that the older .jar files were being used by Axis (which showed up in the log files).

To check for multiple jar entires, the find command was very helpful.

find . - name “*lucene*jar”

Jarring files needs to be in the correct subdirectory. If its part of a package, the subdirectories need to be involved.

jar cvf Name.jar -C foo/ .

This command does not include “foo” in the jar, making it hard to find the package when called.

Lucene's Field is deprecated, and “contents” not “content”. The symantics of Lucene 2.2.0 were had to find because there is a newer form of Lucene, but having “content” instead of “contents” is a major issue that will not return results.

Front-end and Backend both had to match. When I checked the log files initially, the changes to Pileus were not showing up. This is because I hadn't updated Nimbus (the front-end). I'm not sure how much impact this had, but it helped the log files.

You cannot send value=”” over the wire in XML. There is a variable index=”” in the XML for searching with Lucene, since it requires an index. The new indices don't have the extra path (and it would be a waste of time to add it), so I wanted to send a blank index over. The XML doesn't let you do this—it will set the index to null or pull a space out of the node.

Solution: index=”DEFAULT” added.

When this is taken up on a backend, it sets the extra subdirectory for the index to “”, which has worked on the one I'm using.

Monday, 2 June
I've been working on getting Multisearch working with Lucene 2.2.0. I'm working with the indices Fallen has given me, and I think I've made significant headway. Via the logs, I've narrowed it down to several possibilities:

1. While the old code still compiles, there is some deprecated function that no longer acts as it should.
2. There could be some issues of having the CLASSPATH contain both jars.
3. Multisearch isn't sending the packaged information over the wire.
4. Lucene 2.2.0 uses different semantics than older Lucene 1.4.3 (unlikely).
5. OGSA-DAI 2.2 cannot work with Lucene 2.2.0 because it was made with 1.4.3.

I'm pursing all of these options, but it looks like I'll have to use Tintin (Old Snowy) or Pileus and just change the Lucene .jar files to see what happens.