THEMIS Tohban FAQ
Q: The data I'm expecting to see isn't in the system, or looks incorrect. What do I do next, and who should I contact?
A: There are a number of reasons your data might not be showing up yet.
1) Perhaps it hasn't yet been received from the probe. Probe B and probe C are now at the Moon, so there may be delay between when the data is collected and when it's received at Berkeley for processing. Even then, sometimes passes can't be supported as planned, and some data might have to wait until the next perigee pass before it's dumped. Some passes are for commanding and housekeeping data dumps only, so the fact that a pass occurred and data was received does not necessarily imply science data was dumped.
How to check: The THEMIS web site has several tools for viewing pass schedules and determining what data was received. http://themis.ssl.berkeley.edu/status.shtml has links to several useful tools: Probe File Recovery from Ground Stations This page gives a contact schedule, with an indication of whether data was received from each pass. If you click on a highlighed entry, more information will pop up telling you which virtual channels (VCs or VC files) were dumped on that pass. If VC2 or VC3 files are shown as "Normal Files", it means at least some data was recovered. If they show up under "Problem Files", it could simply mean that those channels weren't dumped (so we received 0-length files), or there may have been some other processing problem. Virtual Channel (VC) File Statistics This link takes you to a table listing each VC file processed so far, with some statistics showing how much data was recovered, and the data collection times covered by that file. Using these tools, you should be able to "drill down" and figure out whether the data you're interested in has been received from the probe yet. If not, your next course of action should be to contact someone in the THEMIS MOC. They may be able to explain why an expected data dump did not occur, and hopefully tell you when the next dump opportunity is.
2) If the data has been received from the probe, perhaps the automated processing hasn't picked it up yet. This shouldn't delay the data product production by more than a few hours. If the pass ended within the past few hours, you probably just need to wait until the automated processing kicks in.
3) If we've received the telemetry, and enough time has passed that the processing should have happened, and your data hasn't shown up yet, you'll need to do a little more digging to find out how much of the processing was performed. When the THEMIS SOC receives probe VC files from the ground station supporting a pass, there are three stages of processing to produce the Level 2 products. First the VC files are processed to extract the Level 0 packet data. Then, the Level 0 packets are processed to Level 1 CDF files containing uncalibrated data in instrument coordinates. Finally, the Level 1 CDFs are calibrated and transformed into geophysically relevant coordinates, yielding Level 2 CDFs and overview plots. The Level 0 products are organized on our file servers by probe, UTC date, and apid (data type).
The URL http://themis.ssl.berkeley.edu/themisdata/PROBE/l0/YYYY/MM/DD should give you a directory listing showing the packet files processed so far for that probe and date, including modification dates and file sizes. The data type is indicated in each filename by the apid, a 3-digit hex number just after the "l0" in the file name. Apids in the 300-3ff range are BAU housekeeping. Apids 404, 405, and 406 are IDPU housekeeping. Apids 410 and above are science data.
Each Tohban should be familiar enough with THEMIS data types to know which apids are required to produce a given L1 or L2 CDF or overview plot type:
Apid Description L1 CDF type
0x300..0x3FF BAU housekeeping BAU
0x404, 0x406 IDPU housekeeping HSK
0x405 FGM Engineering FGM (8 Hz, least sensitive range)
0x410 IDPU Spin Fits FIT
0x440 EFI Filter Bank FBK
0x441 EFI Voltages V1-V6 (A) VAF (Fast Survey)
0x442 EFI Voltages V1-V6 (B) VBF (Fast Survey)
0x443 EFI E-Fields EFF (Fast Survey)
0x444 SCM Fast Survey SCF
0x445 EFI Voltages V1-V6 (A) VAP (Particle Burst)
0x446 EFI Voltages V1-V6 (B) VBP (Particle Burst)
0x447 EFI E-Fields EFP (Particle Burst)
0x448 SCM Particle Burst SCP
0x449 EFI Voltages V1-V6 (A) VAW (Wave Burst)
0x44a EFI Voltages V1-V6 (B) VBW (Wave Burst)
0x44b EFI E-Fields EFW (Wave Burst)
0x44c SCM Wave Burst SCW
0x44d FFT Particle Burst FFP_16, FFP_32, FFP_64
0x44e FFT Wave Burst FFW_16, FFW_32, FFW_64
0x451 Trigger data TRG 0x453 ESA/SST moments MOM
0x454 ESA Ion (eif) ESA Full Distributions
0x455 ESA Ion (eir) ESA Reduced Distributions
0x456 ESA Ion (eib) ESA Burst Distributions
0x457 ESA electron (eef) ESA Full Distributions
0x458 ESA electron (eer) ESA Reduced Distributions
0x459 ESA electron (eeb) ESA Burst Distributions
0x45a SST Ion (sif) SST Full Distributions
0x45b SST Ion (sir) SST Reduced Distributions
0x45c SST Ion (sib) SST Burst Distributions
0x45d SST electron (sef) SST Full Distributions
0x45e SST electron (ser) SST Reduced Distributions
0x45f SST electron (seb) SST Burst Distributions
0x460 FGM Low Resolution FGM
0x461 FGM High Resolution FGM
If the packet files you're looking for are not on the server, but should be (based on the probe configuration, pass schedule, and VC file statistics), you should notify Jim Lewis (jwl@ssl.berkeley.edu) and Tim Quinn (teq@ssl.berkeley.edu). Please provide the following information: Probe, date, and apid of L0 packet data you're looking for Name of the VC file(s) you think should have contained this data (obtained from the various status pages)
4. If L0 packets you're interested in are present, try looking for the L1 CDF files. These are organized by probe, year, and data type, with each yearly directory containing all 24-hour CDF files for that data type. The URL format is: http://themis.ssl.berkeley.edu/themisdata/PROBE/l1/DATATYPE/YEAR If the L0 packets are present, but the L1 CDF is not, or if the CDF size is too small for the amount of L0 packet data, you should notify Jim Lewis (jwl@ssl.berkeley.edu). Please provide the following information: Probe, date, data type of L1 CDF of interest If the L1 CDF is present, but something looks wrong with the contents, please describe in as much detail as possible what you're seeing, and what you expected to see. Keep in mind that the programmers may not be able to look at a plot and immediately see what's wrong with it -- plots are helpful and we'd like to see them, but please don't skip the verbal description of what the problem is!
5) If the L1 CDF looks OK, but the L2 CDF or overview plot is missing or incorrect, try generating an overview plot from the IDL command line: To get the single-spacecraft overview plot:
IDL> thm_gen_overplot, probe='b', date='2007-12-11'
Set the /makepng keyword to get output files. This will give the full day file, and the six-hour ones. To get FGM tohban plots:
IDL> thm_fgm_overviews, '2007-12-11', /nopng
For the tohban plots, png files are done as a default. /nopng turns this off. Also, for these the date is not a keyword. ESA, SST are the same as FGM:
IDL> thm_esa_overviews, '2007-12-11', /nopng
IDL> thm_sst_overviews, '2007-12-11', /nopng
For the ESA you get burst, reduced and full plots. For SST you get full and reduced plots. For the new memory plots, the date is a keyword input.
IDL> thm_memory_plots, date = '2007-12-11', /nopng
If the plots that you obtain differ from those that are online, then the next set of online plots will probably be correct -- the overview plots are created every 4 hours. If the plot that you create is the same as the online plot, then the L2 data is truly not in the database. If there is L1 data and not L2 data, check the time on the L1 files. The L2 process runs at 0600 local time. If the L1 files were created after 0600 today, then there wil not be a corresponding L2 file, but there should be one tomorrow. If you need to see the L2 data earlier, then contact jimm@ssl.berkeley.edu, and he can run the L2 generation process and create the needed files.
6) If the L2 CDFs or standard overview plots look OK, but the TDAS client side software is doing something unexpected with them, perhaps you've discovered a bug in TDAS. In that case, please use the error reporting form available from thm_gui, or contact David King (dking@ssl.berkeley.edu) directly, providing a detailed description of the problem, a TDAS history file, plots, or error messages illustrating the problem.
