Datamgr is a GUI program which was designed to allow users of SDT to select data from the FAST data archive and then retrieve that data for analysis in an SDT plot window. When datamgr is launched from an SDT plot window, the user can simply enter an orbit number or timespan, download the required files, if necessary, and then return to SDT, all at the push of a button. Datamgr can also be used as a stand alone program to make canned queries for FAST data.
Yes and No.
As for the No:
Many of the features of
datamgr are not available from the command line:
Datamgr
was designed to automatically interact with SDT to give it the data
it needs, but allow the user to manually override the automatic
selections, if necessary. Datamgr also automatically
searches and maintains a network of data dirctories.
As for the
Yes:
There are command line tools which can perform some of the
functions which datamgr performs. They can be strung
together in such a way that it is possible to perform batch
processing of FAST data analysis. There are also utilities for
managing the data directories. The available utilities are:
whichfiles -- given an apid list and an orbit number or timespan (or more detailed query specifications) will return a list of corresponding FAST lzp files, according to the database. All queries possible in the datamgr forms are possible with whichfiles. However, whichfiles cannot return timespan information.
get_apids -- a utility built upon whichfiles which is useful for batch processing at UCSOC, but was not designed to be used from CoI sites, since it assumes that all FAST data is available online. (For this reason, get_apids must be run on the jukebox host machine.) Some remote users have gotten around this limitation by using get_apids to copy files to an ftp staging area, from which they can retrieve their data. Given a list of apids and an orbit number or timespan, get_apids will retrieve the necessary lzp files from the FAST archive. Get_apids is used by the systems which generate the summary plots and daily CDF files, as well as the website which generates high resolution CDF files.
update_filelist -- lets you run datamgr's update_filelist function from the command line. Useful if you have scripts which delete files from datamgr data directories, bypassing datamgr's 'Delete Files' button.
trimFileList -- obsolete utility replaced by update_filelist and update_vollist.
"Revised FAST Database Design" (28 February 1994) -- I need to update this, to eliminate what has not been impremented, and to give more details about what is actually there. Should also give information about indexes.
"FAST Database Tables" diagram -- viewable from
datamgr.
The source is actually
src/database/fast_archive.fig in the Code Manager system.
This gets translated into Tcl with figToTcl.awk, and
then the results are hand edited into the fast_archive.tcl
file.
"FAST Online Files Tables" diagram.
This is
only available as src/database/fast_online.fig in the
Code Manager system.
Sybooks -- full Sybase documentation available on line through this utility.
Sybase Backups are documented in ~sybase/admin/sybase_backup+recovery.doc
The raw Sybase partitions on the four internal disks on juneau are mapped out in the Star Office spreadsheet: ~sybase/sybaseRawPart.sdc. Handy for working out new partition schemes, and for calculating the proper inputs for diskinit, based on the partition sizes given by format.
FAST Database Overview document by George Kaplan. This document is an essential introduction to the database update system, and some Sybase system administration issues. Some of the information is out of date, but the overall design is still the same. The backup system, for example, has been automated and revamped.
Path:
/disks/juneau/home/gckaplan/software/fast/gckaplan/src/database/database.overview.doc
To
print, use troff -ms database.overview.doc | dpost | lp
For
plain text, use nroff -ms database.overview.doc |
more
Another legacy document from George Kaplan, is his transition document: /disks/juneau/home/gckaplan/fast_tasks/transition/gck.current.tasks.txt
FAST Software releases are available in the juneau.ssl.berkeley.edu ftp site in the pub/software_releases directory.
A tar file containing the binary executable files for
'Datamgr 2001 beta' is available for download:
datamgr_2001.tar. Untar it in your $FASTBIN directory.
NOTE: first save your old datamgr related files: after downloading datamgr_2001.tar, make an archive of all your old datamgr-related executables by running
cd $FASTBIN
tar cf datamgr_save `tar tf datamgr_2001.tar`
New man pages are available for download: datamgr_2001_man.tar. You must untar it in your $FASTHOME directory.
A completely new FAST software release is in preparation, to be available mid-November 2001.
There are three versions of datamgr to contend with, two of them are currently widely used: for convience, we'll call them 'Datamgr Cassic' and 'Datamgr 2000'. As of November 2001, there is a new release -- call it 'Datamgr 2001' -- which fixes the glaring problems with 'Datamgr 2000'.
Users of 'Datamgr Classic' are Co-Investigators who have felt no compelling need to upgrade, or have held back because of horror stories about 'Datamgr 2000'.
Users of 'Datamgr 2000' are the intrepid Co-Investigators who upgraded, as well as anyone who installed and is using the Cluster release of SDT, made in early 2001.
'Datamgr 2001' actually has an 'About' command under the 'Datamgr' menu which gives version information.
With 'Datamgr 2000', users no longer had control over the filelist used by datamgr to keep track of files online. 'Datamgr 2001' fixes this problem. Here is a list of all the fixes/additions:
Except for the last item, all the changes are fixes problems with 'Datamgr 2000', which was first released in late 1999, and which went out with the cluster release.
If you have already upgraded to 'Datamgr 2000', you don't need to make any changes to your configration. If you are upgrading from the original 'Datamgr classic', you need to follow the detailed instructions are in the README.datamgr_remote_files file, steps 2) through 5). Ignore steps 1) and 6), which are specific to installing the 'Datamgr 2000' binaries. For these steps, follow the installation instructions in this FAQ, above.
Run update_filelist from the command line. If you don't have it, get it from juneau's ftp site.
Update Filelist in the 'Datamgr 2000' version only checks the files returned as online in the current query result file list. To get the functionality of Update Filelist in the old datamgr, you need to get a hold of the update_filelist command line utility (included in the 'Datamgr 2001' distribution). In 'Datamgr 2001', all files returned by the query as being on line are automatically checked for existance and basic integrity before they are listed as being on line. (Currently datamgr only checks that they are not full of zero bytes. It would be nice to also check the file lengths, but this is not done yet)
Yes. Add the following line to your Datamgr.conf file or
~/.datamgr.conf file:
set RSH /usr/local/bin/ssh
(assuming
your ssh is installed in /usr/local/bin)
The next step is to get access permission to UCSOC. To do this, you create a public key using ssh-keygen, -- when it asks for a passphrase, just press return to enter an empty passphrase -- and send us the public key file it generates for you (~/.ssh/identity.pub) to us as an email attachment, so we can add your key to the .ssh/authorized_keys file for the jbuser account on our end. Using an empty passphrase allows you to log into the jbuser account on the UCSOC machines without being prompted for a password. This is necessary because there is no way for datamgr to pass your passphrase to ssh.
If you are using OpenSSH, you need to to do the following to make sure that you are connecting with rsa1 protocol:
generate your key with
ssh-keygen -t rsa1
and in your Datamgr.conf file or
~/.datamgr.conf file, you need to specify that you will connect using ssh 1 protocol:
set RSH "/usr/local/bin/ssh -1"
It is important to verify that your ssh configuration works from the command line before you try using datamgr. Try running the command
ssh -l jbuser juneau.ssl.berkeley.edu ls
and verify that it works without prompting you for input.
If you are using OpenSSH, then verify your configuration
by running
ssh -1 -l jbuser juneau.ssl.berkeley.edu ls
In the "Datamgr" menu, select the "SQL Query" checkbox. Datamgr's form interface will be repaced with a text widget which displays the SQL code corresponding to the current entries in the form.
You can select "SQL" instead of the default "Form" from the pop-up menu at the right hand side of the "Orbit/Attitude Criteria" and "Mode Period" panels.
In older versions of datamgr, altitude is not listed in the "orbit/attitude criteria" menu. This is a bug which is fixed in the latest version. In older versions, you can use the SQL text widget instead of the form widget for the Orbit/Attitude panel
By default, datamgr makes a query for timespans as well as files. If you are just interested in getting data to load into SDT, you may want to disable timespan queries by unchecking the "Query Result: Time Spans" checkbox above the timespan result list. On the other hand, if you want to make a survey to find when a certain type of data is available, then you can de-select the "Query Result: Files" checkbox. The resulting timespan list can be saved and/or loaded into the timespan list query widget with the "Set Timespan List" button. To save the list, right click in the Time Span list box, and choose "Save ...". To load into the "timespan list" widget for "Query Limits: ", right click in the list box and select "Load ...".
Press Shift-F1. It is called the tw window. It was designed for development and debugging, and it gives you access to datamgr's internals. It even keeps command history like tcsh!
In the tw window (press Shift-F1 to get it), enter the command
list_files location_string
location_string
is a string
made up of the location codes of the files in the file list that
you want to locate. For example, if you just want to locate
online files, then you would run
list_files OD
In the latest versions of datamgr ('Datamgr 2000' and later) cdromCopyTool will come up and list the names of all the volumes you need. Tip: you can have as many cdromCopyTools as you like, so you can find all the volumes you need, then go get them all from the CDROM library, and then come back and run each cdromCopyTool in turn.
Be aware that if datamgr did not start up when you selected
"Data Manager", it is possibly because SDT's fast_decom
sub-process has failed. In this case, it is necessary to restart
SDT, before you will be able to display any FAST data at all. Run
ps -ef | grep fast_decom to check if the fast_decom process is
running.
If fast_decom is running, then you can start
datamgr from the command line and then select "Attach to SDT
Window" from the "File" menu. Select the number (0,
1, 2 or 3) which corresponds to the number in the title of the SDT
Plot Window you selected "Data Manager" from.
Sometimes, even though there is no datamgr present, there remains a 'zombie' record of a datamgr running in the X windows property which is used for communication between SDT and datamgr. Unfortunately, this bug is difficult to reproduce intentionally, and therefore difficult to debug. If this happens to you, the problem is known to go away by itself after some time -- not always a good solution! To fix it right away, try quitting SDT, starting up datamgr from the command line, and then run "Attach to SDT Window" before you start SDT again. If this still gives an an error, you may have no choice but to exit your desktop session and log in again to solve the problem.
Most likely, the data actually is online, but the fast_online database is corrupted. It is necessary for the DBA at the UCSOC to fix the database.
Instruction for the DBA: Check which CDROM volumes are corrupted by running the datamgr query which shows files to be available only on CDROM, and pressing 'Get Files' to bring up the CdromCopyTool, which will tell you the names of the CDROM volumes which are affected. Then, from the UNIX command line, give the command
update_filelist -pseudoraid /disks/FAST_CD_*/VOLNAME
where VOLNAME is the name of one of the cdrom volumes which is not entered correctly into the database. This command locates VOLNAME in the online storage disks, and updates the database. The -pseudoraid option is necessary, because by default update_filelist will not work on the pseudoraid system, in order to prevent users from inadvertenly launching a 3+ hour update process.
This command can fail for two reasons:
First, if the CDROM actaully does not exist on line, you will get the
error:
update_filelist: No match.
Second: if the volume is completly absent from the database, (not likely
unless you just restored it to be online) it
will be necessary to run
trimFileList -u /disks/FAST_CD_*/VOLNAME
to add the CDROM volume to the database.
There is a problem with the jukebox software: sometimes it gets into a state where files retrieved from the jukebox come out completely filled with zeroed out bytes (null characters), although they have the proper file lengths. Also, there is a problem with SDT, where it crashes if it gets passed one of these bad files. My solution in the latest datamgr release is to have datamgr check for these bad files and remove them, so it doesn't pass them on to SDT, but until the jukebox software gets reset, you will have to get them from somewhere else! Usually this problem affects some but not all of the volumes or readers in the jukebox.
Having these bad files lying around the system is a bad thing, since with the old versions of datamgr, they are indistinguishable from good files. So you can even get this error when accessing files in the online data directories, when using the old datamgr, if any bad files have been downloaded and left lying around. If you don't want to download the new datamgr, then at least get the update_filelist utility. When you run update_filelist, it will automatically delete these files and remove them from the filelist.
You need to have permission to log in as jbuser. For the best way to do this, see Can datamgr use ssh to download UCSOC data?
There is no Pmode 0 nor Fmode 0. When fastcount cannot determine the particle mode or the fields mode from the telemetry, it sets Pmode or Fmode to 0, respectively. When the counts file gets processed into the database, these 0 entries should properly be recorded as NULL entries for Pmode or Fmode in the mode_periods table. Anyone want to fix counts_sqlgen and then fix the database?
The mode entries in the mode_periods table come from the counts files generated by fastcount from the telemetry. The same period of time is usually counted and processed into the database in multiple contacts -- this is because some APIDs may get downlinked in a later contact than the survey data, and the data in the housekeeping APIDs may be downlinked in
There is some attitude data in the database, up until October 14, 1997. So far, there is only level 0 attitude data, from the FDF files. Level 0 is good enough to know which way the spacecraft is pointing relative to the sun and the orbit plane, but is not good enough for accurate spin fitting. The IDL procedure update_fa_fdf does the job of adding this data to the database, but it never got put into the dbupdate script to be called every time we get a new vector (daily).
As things stand now, all the level 0 data could stand to be reprocessed -- a job which would take about 55 hours to run. But first, the update_fa_fdf procedure needs to be rewritten to interpret the angles lambda and phi correctly: sometimes the FDF files use negative angles like -90, and sometimes as 270. Furthermore, the linear interpolation which is currently used does not work for real transitions from 360 to 0 degrees, even if the input angles from the FDF files are normalized. Once this is done, update_fa_fdf should be incorporated into the dbupdate process.
There are no routines yet for getting level 1 attitude [the UCLA attitude solutions] into the database.
Z_sun is the angle in degrees between the +Z axis and the sun. The FDF files define Lambda and Phi as the OCS Attitude. They apparently define the attitude vector in a spherical coordinate system. Lambda apparently is the angle between spin axis and the orbit plane. Who knows how phi is defined? Careful: Sometimes lambda is -90, and sometimes it is 270. Easy for humans to understand, but not so easy for comupters.
Usually, nothing! I don't worry about the occaisional glitches with the drive which cause some dumps to fail. I try to look into DBCC errors and persistant errors with dumping the fast_archive database. The latter could be due to running out of room on the dump tapes.
Check that DS3 DAT tapes were used. DS2 tapes hold only ¼ as much data, and should no longer be used.
You should use the sybtape utility:
rsh -l sybase juneau
sybtape -x 514
This utility changes the name of the tape's log file in the ~sybase/admin/dat_tapes directory and does whatever else is necessary...
The device configuration is designed for ease of
upgrade (space for two separate sybase installations) and to be
able to keep running if any one of the four (4) sybase disks has to
be taken completely off line. Sybase runs on four of the internal
hard disks in juneau: c2t0 - c2t3.
So, even though tempdb
does not need to be archived because it contains critical data, it
is necessary to have tempdb mirrored so we can keep running in the
event that one side of the mirror must go down.
To expand
tempdb, we might expand into the reserved space on the t1 and t2
disks.
Last time, we mirrored the database onto external disks, then replaced the internal disks, installed the new version of the server on the internal disks, and restored the databases from dump tapes into the upgraded database. We had two sybase servers running at the same time, on separate ports. The old version remained up and running throughout the process, until the new server was ready to go on line.
It would be possible to expand disk space without using external disks, since we are currently fully mirrored, so that the database can stay up and running while any single one of its data disks is off line. This makes it possible to disable all mirrors on one disk, swap it out, and then re-mirror. Note, however, that to actually extend the size of a sybase device, it is necessary to use dump and restore. To do this, unmirror all partitions, set up a non-production server on the empty set of partitions, and set up the devices with the new, expanded sizes. The non-production server should run on a different port, configured in the interfaces file. Do the databases from the dump tape into the offline server. Once the ofline server is working, the other server can be brought down, and the new one brought up on the standard FAST Sybase port. Then the partitions can be re-mirrored.
More info about simultaneous servers: compare RUN_FAST_DB* in
/disks/fast/sybase/install with RUN_new_fastdb*, which were used to
start up the non-production server. You can create an alternate
FAST_Sybase start script to run the new_fastdb server. See also the
/disks/fast/sybase/interfaces file, which still has entries for a
new_fastdb server. The name of the server can be specified with the
DSQUERY environment, or with the -S option when using isql in order
to connect with the proper server.
If new raw devices will
be used with sybase, then it is necessary to set the permissions
for them. This is done in the /etc/init.d/FAST_Sybase start
script.
Here's the plan to upgrade and expand data disk
space. Refer to the Sybase Raw Partition Spreadsheet for details on
the partitioning scheme.
I would unmirror all the devices off the the disk with the /mydisks/sybase partition (/dev/rdsk/c2t3d0s..), replace that (t3) disk with a larger disk and format it with larger partitions for extradata and extraindex, also making sure that the master and sybsystemprocs devices will be big enough for the new version of Sybase server, and that the /mydisks/sybase partition is big enough for the installation itself. We may want to think of a way to expand the tempdb device.
During the upgrade process, the production database server can continue to run from /mydisks/sybase_bak. I would unmirror the rest of the devices on the production database so it runs only on the t0, t1 and t2 disks. I would leave auditing running on the production database.
I would install the latest stable release of the server (ASE 12.5?) on /mydisks/sybase, and bring it up as new_fastdb. I would configure it for larger extradata and extraindex devices on the t3 disk -- all the other devices would remain the same size (unless we are expanding master, sybsystemprocs or tempdb) -- it would set up its devices on partitions on the t1, t2, and t3 disks. Then I would dump the 11.9.2 production database and restore it into the ASE 12.5 server.
Once the database is upgraded and working in the ASE 12 server, I would make that server the production server. I would take down the 11.9.2 server and set up auditing for the ASE 12.5 server on the sybsec and auditdev_01 and auditdev_02 devices, which would then be free. Remember that the /disks/fast/sybase symbolic link will have to be changed to refer to /mydisks/sybase.
Finally, I would bring juneau down once again, swap out the disk with the /mydisks/sybase_bak partition and increase the size of the extradata and extraindex partions on it. I would then re-mirror all the devices to reproduce the same configuration as in the original Sybase Raw Partition Spreadsheet. I would place a copy of the new Sybase server installation in the /mydisks/sybase_bak directory, and continue to run the production server from /mydisks/sybase.
Call 1-800-8-SYBASE (1-800-879-2273)
Calls must be
initiated by a Sybase support contact. Besides Ken Bromund, Tim
Quinn is also a support contact.
Our customer number is
22866-93
We are running Adapdive Server Enterprise 11.9.2 and
Open Client 11.1.1
I have been very satisfied with Sybase
support.
Sybase support has to be renewed at the end of
every year!
Last year, the contact for support renewals was:
Sybase, Inc.
ATTN: Christine Cunningham
Fax:
510 922-4772
Phone: 510 922-7645
Other contacts:
Jane Schoenfeld is our rep for the UC sybase contract at the office of the President, she can always give you the latest info about whom to contact at Sybase:
Jane Schoenfeld
510-987-0683
jane.schoenfeld@ucop.edu
The last message I got from her about our rep at
Sybase was:
Our Sybase rep has come full circle. We have a "new"
rep back in MA. Here's the contact info:
Mike DiFilippo
Telesales Representative
(w) 978.287.2673
(f)
978.369.3592
http://www.sybase.com
Michael.DiFilippo@sybase.com
trimFileList is designed for adding/removing entire data directories,which can be CDROM volumes, which look just like user data directories to datamgr. Once a volume has been added, it does not look into its actual contents again.
Implementation detail: A volume is considered 'present' if there is an entry for it in OnlineLocations AND there is at least one corresponding file entry in OnlineFiles, (where Location = LocId) So, a volume might be present in OnlineLocations, but trimFileList can work to add the underlying lzp files if there were none already in the OnlineFiles table. Conversely, if there were only a partial list of files in OnlineFiles, trimFileList will not add the missing ones
update_filelist was designed to be a command-line utility which gives the functionality of the 'Update Filelist' command in the original 'datamgr classic' program. Because it looks at the filelist database (to those in the know, that means the fast_online..OnlineLocations table) and finds all directories on the local domain, update_filelist does not work to add a new data directory to the database. Rather, for each directory the database knows about, it synchronises the contents of the database for those directories with what is actually on line. It also can check for and remove bad files in the process. When it is run from juneau, the 'DOMAIN_HOST' which is datamgr's entry point to our domain from the outside, it also sets the OnHost flag. This flags entries like '/mydisks/scratch/fastdata' (accessible only from the local machine, and not juneau) so that datamgr will know they are not visible from remote sites. Users should properly use DATA_DIR names like '/fastdata/hostname' (which is the name by which the same disk is visible from both juneau and the local host), so that other datamgr users can share their data files.
update_vollist was designed to replace trimFileList for adding/removing CDROM volumes from the database. This is a new functionality of datamgr which I am not forcing on anyone, since it has not been tested by real datamgr users other than myself, (but I have done some testing myself.)
Check out the hires_roadmap file.
Whichfiles is not always the best tool for data availablity questions, since it was designed to return return any data which matches the query criteria, but not to let you know if a full data set is available or not. Perhaps the availablity column of the hires_cdf.cfg file could be used to specify a different tool to use to determine availability, when necessary, e.g. for burst data, or DSP.
hires_roadmap is at /disks/plasma2/www/fast/hires/config/hires_roadmap.doc
Check out this document: /disks/plasma2/www/fast/hires/config/hires_cdf_status.html