Argos Realtime Scripts Flow Diagram

graph TD do_argos-->argos buoys.xml-->argos argos-->set_up argos-->make_empty_matfile argos-->proc_argos_hex imet_rec.xml-->go argos-->go argos-->do_plot_deployment argos-->do_plot_QC argos-->gps_throughput argos-->print_GPS argos-->plot_GPS m_map-->plot_GPS go-->process_imet go-->extract_argos_data extract_argos_data-->cksum go-->get_posit go-->print_MET process_imet-->ndbc_xml_out process_imet-->vect_2md process_imet-->precip2prate1hr print_MET-->MET_data.txt print_MET-->clean_data gps_throughput-->MET_throughput.png print_GPS-->Argos_GPS.txt plot_GPS-->Argos_GPS.png ndbc_xml_out-->ftp_ndbc.sh ndbc_xml_out-->rh2dewpoint ndbc_xml_out-->argos.xml argos.xml-->ftp_ndbc.sh plot_RAW.xml-->do_plot_deployment plot_QC.xml-->do_plot_QC do_plot_deployment-->plot_throughput do_plot_deployment-->get_plot_pars do_plot_QC-->get_plot_pars do_plot_QC-->plot_MET plot_MET-->slabel plot_MET-->suptitle plot_MET-->MET.png plot_public.xml-->do_plot_QC style buoys.xml fill:#6c5; style imet_rec.xml fill:#6c5; style plot_QC.xml fill:#6c5; style plot_public.xml fill:#6c5; style plot_RAW.xml fill:#6c5; style MET_data.txt fill:#39c; style MET_throughput.png fill:#39c; style Argos_GPS.txt fill:#39c; style Argos_GPS.png fill:#39c; style MET.png fill:#39c; style argos.xml fill:#39c; click do_argos callback "Testing comment"

argos.m

This is the primary argos script. It is responsible for identifying buoys which have Argos instruments, checking whether there are new files to be processed, and calling other scripts necessary to process messages, and print and plot the output.

Argos messages will be processed for buoys which have an tag defined in buoys.xml, and are operational, in-prep, or recovered. Argos messages are first parsed using a script proc_argos_hex.m which separates the data into two structures: argrec (GPS information) and drec (MET data). The original .dat message file is archived in /argos/buoy/dat/, argrec and drec are saved in a /argos/buoy/mat_raw/. Next, argos.m calls go.m, a subroutine which handles buoy system level processing. New data is appended to current data which is saved in a .mat file located in the buoy's deployment directory. Tabulated data is also printed out to .txt files, and raw MET, QC'd MET, and gps plots are made and sent to the real-time webserver.

cksum.m

Performs a cyclic redundancy check on the 256 bits of binary encoded MET data. It uses the Bradley method (June 1983), converted to MATLAB by Jon Ware 5/15/2000.

It takes two parameters:

  1. fr - a 256 bit binary character string
  2. n - number of bits to check?

crcfix.m

This is an experimental script which uses a brute force strategy to attempt to recover MET data which failed the cyclic redundancy check. In theory, all single bit errors should be detectable by a CRC check. Therefore, by flipping one bit at a time, single-bit errors are fixed. It expects a 256 bit binary character string as a parameter, and zero if unsuccessful, or the position of the incorrect bit if successful.

clean_data.m

graph LR qc.xml --> clean_data b["mybuoy[]"] --> clean_data clean_data --> c["sys[]"] c-->clean_data style qc.xml fill:#6c5

This script is responsible for most of Argos data's quality control. sys is a structure containing one buoy system's MET variables. qc_file is an .xml document which contains information about acceptable values for each variable. mybuoy is a structure which contains all of the meta-data for a specific buoy. It is extracted from /realtime/config/buoys.xml. It also requires a MATLAB "structures" set of routines which is included in /matlabtools/structures/. The modified sys structure is returned upon completion of the script.

For buoys with a deployment date, clean_data trims off all records prior to the deployment. Each variable is tested for values which are beyond the minimum and maximum acceptable values as defined in qc_file. Data which fails the min/max test is replaced with NaN, and will be omitted from plots. Each variable is also tested for stuck values. A threshold is defined in the qc_file using the tag. If the most recent records have the same variable value, then an email alert is sent to all recipients listed in of buoys.xml. A blanking test is also implemented which allows a user to manually clear out records (values will be set to NaN) for a specific variable given and tags. A value of zero will disable blanking for that variable.

do_argos.m

Calls argos.m, the main argos script. It is responsible for catching errors, and exiting gracefully back to the shell.

do_plot_deployment.m

After data has been processed for a deployment, this script will create full and short ts plot of imet argos data. Called by argos.m

do_plot_QC.m

Script is located in the MET directory. do_plot_QC creates all of the QCd MET plots. It loads in the QC data from the .mat files located in the deployment directory. The order and format of each plot is set up by calling get_plot_pars() which reads in the info from a .xml file. It calls plot_MET which handles the actual graphics calls. Plot titles and time range are also defined in do_plot_QC.

extract_argos_data.m

graph LR imet_rec.xml --> extract_argos_data argos_RAW.mat --> extract_argos_data extract_argos_data --> argos_PROCESSED.mat extract_argos_data --> cksum style argos_RAW.mat fill:#39c style argos_PROCESSED.mat fill:#39c style imet_rec.xml fill:#6c5

First, this script tests whether an Argos raw data file contains any records with ptts which match the currently deployed bouy. It then performs a cyclic redundancy check(CRC) on MET data string to ensure data integrity. If successful, this script decodes each variable from binary to integer, and applies appropriate scaling and biasing to the variables as defined in imet_rec.xml. Variables are saved to a .mat file within the buoy's deployment directory with variable names such as A2 AT B2 BP BV C2 CO ...

  1. Scans all records in Argos raw .mat file for ptts which match currently deployed buoy. Note that buoys with different deployments may share the same Argos raw .mat file
  2. Performs a cyclic redundancy check(CRC) on transmitted MET binary string to ensure data integrity. Discards data if there is a checksum error.
  3. Decodes binary MET data into variables according to bit precision, offset, and scaling as defined in imet_rec.xml
  4. Saves variables to a Argos processed .mat file which resides inside the buoy's deployment directory.

getUTCmday.m

Print coordinated universal time in number format rounded to nearest day. Used by do_plot_deployment.m. For example, it may return '736430'. UTC is a successor to Greenwich Mean Time. It doesn't observe daylight savings time.

getUTCstr.m

Print coordinated universal time in string format including month,day,year,hour, and minute. For example, it may print 'Apr 14 2016 13:51'. Used by plot_imet_argos.m. UTC is a successor to Greenwich Mean Time. It doesn't observe daylight savings time. get_plot_pars.m

get_posit.m

graph LR get_posit --> sys.mat sys.mat --> get_posit argos_RAW.mat-->get_posit style sys.mat fill:#39c style argos_RAW.mat fill:#39c

Processes Argos Doppler Position data (rough GPS-like information accurate to about 1000m) for one specific buoy, one specific system by scanning for the appropriate ptt identifier in the Argos raw .mat file. proc_argos_hex() needs to run first in order to generate the raw argos.mat file. The script will append new position data to current position data saved in the sys.mat file located within the buoy's deployment directory.

go.m

Process and print out data for one specific instrument:

  1. Loads in data from sys.mat, creates new sys.mat if it doesn't exist.
  2. Creates deployment directory if necessary.
  3. Process data for one specific instrument
    1. Calls proc_argos_hex.m to create .mat file if it doesn't exist
    2. Calls extract_argos_data.m to decode hex information in .dat file
    3. Calls process_imet.m

ime_rec.xml

Defines bit precision, scaling, and offset values necessary to decode each Argos MET variable.

make_empty_matfile.m (UNDER CONSTRUCTION)

Creates an empty MET structure with a field for each MET variable. This is called the first time a buoy's argos system is processed.

ndbc_xml_out.m

graph LR rh2dewpoint --> ndbc_xml_out ndbc_xml_out --> ftp_ndbc.sh ftp_ndbc.sh --> NDBC ndbc_xml_out --> rh2dewpoint ndbc_xml_out --> argos.xml argos.xml-->ftp_ndbc.sh style argos.xml fill:#39c

This script will process one specific buoy, one specific system. It translates Argos MET information into an .xml format, performs QC by calling clean_data.m, and sends the output to NDBC via a ftp script. The script also derives dewpoint from ATMP and RH MET variables. Note that NaN values are replaced by -999 per NDBC's request.

plot_MET.m

This script is located in the MET directory. plot_MET is called by both do_plot_deployment() and do_plot_QC(). It creates a plot of multiple variables (using subplots), from multiple MET systems.

proc_argos_hex.m

graph LR argos.dat --> proc_argos_hex proc_argos_hex --> argos_RAW.mat style argos.dat fill:#39c style argos_RAW.mat fill:#39c

Reads in an argos .dat file, parses out position and MET data into two separate structures, along with some basic meta information. These structures are saved into a .mat file: argos_RAW.mat

process_imet.m

graph LR argos_RAW.mat --> process_imet GSW --> process_imet process_imet --> sys.mat process_imet --> ndbc_xml_out process_imet --> vect2md process_imet --> precip2prate1hr style argos_RAW.mat fill:#39c style sys.mat fill:#39c

Reads in data from raw Argos .mat file, derives a few additional variables (Salinity, wind speed, wind direction, cumulative precipitation), and appends data into sys.mat structure. process_imet() is also responsible for calling ndbc_xml_out() script which forwards information to NDBC. Magnetic wind correction takes place in this script. This function requires GSW MATLab toolkit.

rh2dewpoint.m

This function calculates the dew point given relative humidity(%) and air temperature(degrees Celsius).

set_up.m

This MATLab script defines the directory structure used by the Argos scripts.