The hires cdf site has been running fairly stably for a month or so now. The new version allows multiple data quantities to be requested at once, and also has features which facilitate debugging of the IDL scripts and of the data pipeline process. I modified the hiresd process to be more robust, so it can be restarted automatically in case of a crash, and so it does not use up idle cpu time. It has been running ok from my workspace, so I have now put it back into integration.
In the process of learing the system for myself, I created a document to help me find my way around all the different files involved. You may find it useful for reference:
/disks/plasma2/www/fast/hires/config/hires_roadmap.doc
I put this file as well as the hires_cdf.cfg file in the config directory into SCCS.
Here is a detailed summary of the changes to hiresd and hires_cdf:
sleep when waiting for sighup to read request queue, instead of running in a tight loop.
save log of processed requests in processed.db when they are removed from the queue.db file. records "success" or "trouble" for each request, as well as the PID of the hires_cdf process for identifying debugging files.
'-trouble' command line option allows email to be sent to webmaster in case of trouble generation a cdf for a request. currently the lzpw crontab lists only kenb to receive these messages, but it can be changes to FASTMaster or anything else.
include reference to hires_cdf PID in emails for debugging purposes:
hires_cdf now saves all IDL and SDT outputfiles with PID appended.
use file locking (flock) to synchronize que file access and database server access. This allows process to block, rather then wait in a tight loop.
if hiresd process poined to by hiresd.pid file is dead, start up a new process, otherwise, quit: no need to delete hiresd.pid file to get hiresd process running again after hiresd crashes.
don't use an at job to delete CDF files: a cron job running as lzpw now deletes files older than 7 days: so if same file is requested twice, it won't get deleted prematurely for the second person requesting it.
As for the request.cgi script, I checked the original version into SCCS in the fastcgi/hires directory, and then made the following changes in the new version:
(I used perl 5 features to make my modifications. (sorry, all the books I found on PERL were for Perl 5))
When a user requests multiple data quantities for a single orbit or timespan in a single request, they are put on the queue as a separate request line for each data type. -- so hiresd didn't have to be modified in the way it reads requests from the queue.
I made minor changes to the hires_cdf.cfg file -- to allow for specifying a minimum timespan as well as a maximum time span, and I changed the units to minutes.
I fixed some quirks I noticed in the user interface.
It might be nice if the time input format could be made more tolerant. There is a date module for Perl which can parse any date/time format you give it, but I didn't do this yet.
whichfiles still could be extended or replaced to give more acccurate guage of whether data is available for burst data and for quantities which depend on more than one apid.
I think that a nice way to take this web site to the next stage would be to provide a web tool that allows the public (ie our ever-growing network of scientists around the world) to browse the database for when the data types they want are available. We could make a new table in sybase (or in mySql), run queries on the fast archive database to fill it with summary data which can be browsed efficiently, rather than running whichfiles one orbit at a time. Then we can create a web interface to this table, and when the user finds data they want, the web interface could generate a request for hiresd to process.
I'm going to send out a more detailed spec for this idea to McFadden... he seems interested. I think mySql might be the way to go for powering this web interface, if we already have it installed and figured out.