SAFARI WireWalker Data Processing

Purpose

This directory contains the SAFARI WireWalker processing pipeline for both Level-2 quality-controlled observations and Level-3 gridded products, along with raw inputs, metadata definitions, and diagnostic plots.

Processing: 6 October 2026 — Benjamin Greenwood.

Processing assistance: OpenAI Codex was used to assist with development and refinement of processing code, quality-control workflows, and diagnostic plots. Final methods and outputs remain subject to scientific review.

Real-time telemetry: A real-time WireWalker data stream was also received through the Certus 9770 modem. The Campbell CR6 and Certus data-return streams are described in the Raw data section; they are complementary because Certus provides near-real-time return while the CR6 files provide the recovered record set.

Products

The Level-2 pipeline decodes Campbell CR6 records containing RBR CTD measurements, applies pressure-aware adaptive envelope and time-domain spike QC, and writes a compressed CF-1.13 NetCDF4 product. The Level-3 pipeline selects QC-approved upward profiles, averages them into one-meter depth bins at two-hour and four-hour resolutions, and provides separate bounded two-dimensional interpolation products that fill only within 10 meters vertically and five time bins on either side.

How to run

Install the required Python libraries with python3 -m pip install numpy matplotlib netCDF4 gsw pdfkit. The README PDF also requires the system wkhtmltopdf executable used by pdfkit. From this directory, run python3 L2_process.py to regenerate the L2 NetCDF file, python3 L3_process.py to regenerate the L3 NetCDF file, and python3 L3_plots.py --save to create the L3 diagnostic PNG files. Run python3 L2_plots.py to create L2 diagnostic PNG files, or add --show for interactive display. Run python3 publish_readme.py to rebuild this HTML and PDF README.

Directory structure and files

SAFARI_Wirewalker/
  raw/
    wirewalker*.dat
  L2_process.py
  L2_quality_control.py
  L2_netCDF.py
  L2_variables.json
  L2_globals.json
  L2_plots.py
  L3_process.py
  L3_netCDF.py
  L3_variables.json
  L3_globals.json
  L3_plots.py
  publish_readme.py
  plots/
  SAFARI_wirewalker_L2.nc
  SAFARI_wirewalker_L3.nc
  README.html
  README.pdf

The raw directory contains the WireWalker data files. L2_process.py decodes binary packets and coordinates the L2 pipeline. L2_quality_control.py computes QC flags and up/down profile IDs. L2_netCDF.py assembles the L2 product from JSON metadata. L2_variables.json and L2_globals.json define L2 NetCDF metadata. L3_process.py averages QC-approved upward profiles into 2-hour and 4-hour by 1 m time-depth bins and creates bounded interpolated variants. L3_netCDF.py writes the L3 product from L3_variables.json and L3_globals.json.

Raw data

Raw data files

The raw directory contains the Campbell CR6 WireWalker files used to create the L2 product.

FileSizeFirst CR6 timestampLast CR6 timestamp
wirewalker0.dat0.18 MB2025-10-15 15:40:00Z2025-11-09 20:23:00Z
wirewalker1.dat0.73 MB2025-11-10 00:13:00Z2025-11-10 18:00:10Z
wirewalker2.dat32.41 MB2025-11-10 18:03:30Z2025-12-20 08:59:50Z
wirewalker3.dat32.88 MB2025-12-20 09:13:20Z2026-01-28 07:13:00Z
wirewalker4.dat8.15 MB2026-01-28 07:16:20Z2026-02-17 22:02:20Z
wirewalker5.dat8.24 MB2026-02-27 14:03:30Z2026-03-10 01:21:30Z

WireWalker packet format

Each Campbell CR6 record contains WireWalker controller fields and a 1000-byte RBR buffer. The buffer can contain up to 125 RBR measurements. RBR timestamps are offset from Campbell record timestamps by one index. The decoder compares consecutive buffers because size_rbr was not logged accurately; changed bytes identify the number of new 8-byte RBR samples.

Campbell CR6 CSV record
timestamp, record number, controller fields, 14-byte WireWalker header, and a 1000-byte RBR buffer containing up to 125 RBR measurements
↓
WireWalker header: 14 bytes, little-endian byte order
ww_time
4 bytes
rbr_time
4 bytes
size_rbr
2 bytes
serial
1 byte
packet
1 byte
voltage
1 byte
current
1 byte
↓
RBR data buffer: 1000 bytes, up to 125 repeated 8-byte samples, little-endian byte order
time elapsed
1 byte
pressure
3 bytes
conductivity
2 bytes
temperature
2 bytes
time elapsed
1 byte
pressure
3 bytes
conductivity
2 bytes
temperature
2 bytes
time elapsed
1 byte
pressure
3 bytes
conductivity
2 bytes
temperature
2 bytes

RBR timestamps are offset from Campbell record timestamps by one index.

Multi-byte WireWalker and RBR integer fields are stored little-endian: the least-significant byte comes first. The 4-byte elapsed-time field and 2-byte conductivity and temperature fields are decoded as little-endian integers; the 3-byte pressure field is assembled least-significant byte first.

Each RBR sample stores elapsed time in the low byte of the raw value and pressure in the upper three bytes. The size_rbr field was not logged reliably. The parser compares the previous and current 1000-byte buffers, finds the changed byte range, and converts it into the number of new 8-byte RBR samples.

Campbell CR6 and Certus data return

A real-time data stream was received through the Certus 9770 modem in addition to the recovered Campbell CR6 record set. Certus data return was assessed by counting unique RBR timestamps, so repeated telemetry packets are not counted more than once. The Certus stream provided near-real-time observations, while the recovered CR6 files provide the complete record set available after recovery.

L2 processing

Inputs and processing

L2_process.py decodes the raw binary records and coordinates the L2 pipeline. L2_quality_control.py computes pressure, temperature, conductivity, salinity, and up/down profile quality flags. L2 observations retain the native time sampling.

Output and metadata

SAFARI_wirewalker_L2.nc is a CF-1.13 NetCDF-4 product assembled by L2_netCDF.py from the JSON metadata definitions. The inventory below is read from the generated file each time this README is published.

Dimensions and storage

record_cr6=22842 time_rbr=2063569 platform=1. The product is NetCDF-4 with enabled for data variables (zlib, level 4, shuffle).

Variables

NameTypeDimensionsUnitsLong name / flags
latitudefloat32platformdegree_northSAFARI surface mooring latitude
longitudefloat32platformdegree_eastSAFARI surface mooring longitude
record_cr6uint32record_cr6Campbell CR6 datalogger record number
record_timestamp_cr6uint32record_cr6seconds since 1970-01-01 00:00:00ZCampbell CR6 record reception time
record_timestamp_controlleruint32record_cr6seconds since 1970-01-01 00:00:00ZWireWalker controller record timestamp
controller_currentfloat32record_cr6mAWireWalker controller current draw
controller_voltagefloat32record_cr6voltsWireWalker controller supply voltage
rbr_recorduint32time_rbrCampbell CR6 record number associated with each RBR sample
time_rbruint32time_rbrseconds since 1970-01-01 00:00:00ZRBR Concerto CTD measurement time
up_profilefloat32time_rbrUpward WireWalker profile identifier
down_profilefloat32time_rbrDownward WireWalker profile identifier
rbr_pressurefloat32time_rbrdbarRBR Concerto CTD absolute pressure
rbr_pressure_quality_flaguint8time_rbrRBR Concerto CTD pressure quality-control flag
flags [0, 1]; bad good
rbr_depthfloat32time_rbrmRBR Concerto CTD depth, positive downward
rbr_conductivityfloat32time_rbrmS cm-1RBR Concerto CTD electrical conductivity
rbr_conductivity_quality_flaguint8time_rbrRBR Concerto CTD conductivity quality-control flag
flags [0, 1]; bad good
rbr_temperaturefloat32time_rbrdegree_CelsiusRBR Concerto CTD in-situ temperature
rbr_temperature_quality_flaguint8time_rbrRBR Concerto CTD temperature quality-control flag
flags [0, 1]; bad good
rbr_practical_salinityfloat32time_rbr1RBR Concerto CTD practical salinity derived from conductivity, temperature, and pressure
rbr_practical_salinity_quality_flaguint8time_rbrRBR Concerto CTD practical salinity quality-control flag
flags [0, 1]; bad good

Global attributes

titleSAFARI 2025 WireWalker Level-2 CTD observations and quality-control flags
summaryThe ONR SAFARI Departmental Research Initiative aims to improve our understanding of how air/sea interactions impact the timing, intensity, and impacts of atmospheric river events in the eastern Pacific Ocean. A joint Scripps Oceanographic and Woods Hole Oceanographic partnership has developed a capacity to observe surface fluxes and ocean boundary layer response through previous ONR support. The system combines an air/sea flux buoy (WHOI) with a WireWalker ocean-wave-powered profiling vehicle (SIO). For SAFARI, in consultation with the SAFARI DRI science team, a mooring location in the central mid-latitude Pacific (33°25′N, 158°W) was chosen to provide information for numerical models and to investigate basic questions regarding the influence of air/sea fluxes on atmospheric processes in the critical upstream region. This Level-2 dataset contains decoded WireWalker profile observations from an attached RBR Concerto CTD, including pressure, depth, temperature, conductivity, practical salinity, engineering measurements, and quality-control flags. The WireWalker was deployed alongside the surface mooring on 2025-11-21 at 00:40Z and recovered on 2026-09-08. Communication with the WireWalker was lost on 2026-03-09; consequently, the dataset may not represent the complete deployment period.
ConventionsCF-1.13
netcdf_version4.9.2
date_created2026-10-05T22:51:00Z
time_coverage_start2025-11-21T00:41:06Z
time_coverage_end2026-05-10T04:36:44Z
time_coverage_durationP170D
time_coverage_resolutionP2S
geospatial_vertical_min8.810438055890591
geospatial_vertical_max303.47246063603524
geospatial_vertical_unitsmeters
geospatial_vertical_positivedown
licenseCC BY-NC 4.0

QC flags use uint8 values with 0=bad and 1=good. Data-variable valid_min and valid_max attributes are calculated from QC-approved values; the vertical geospatial limits are likewise based on pressure-QC-approved depths.

Quality control

Pressure prerequisite

Absolute pressure is screened first using the configured pressure range. Samples failing pressure QC are excluded from temperature, conductivity, and salinity QC.

Adaptive envelopes

Temperature, conductivity, and practical salinity use adaptive centered time-window envelopes. Candidate windows are 3, 8, and 16 hours. The shortest candidate window containing the equivalent of at least 1,000 measurements per hour is selected locally; longer windows are used during sparse periods. Broad seed limits exclude gross failures before percentile estimation.

VariableSeed rangeLower quantileUpper quantileFixed margin
Temperature9.5 to 22.0 °C0.0050.990±0.07 °C
Conductivity35.0 to 52.0 mS cm-10.0060.990±0.2 mS cm-1
Practical salinity33.5 to 35.00.0150.992±0.018

The resulting bounds are used for the gross envelope check. Measurements inside the envelope are then evaluated by the time-domain WireWalker spike tests described below.

Spike tests

Samples inside the envelope are tested with the causal WireWalker second-difference method and a bidirectional local-linear prediction test. Temperature, conductivity, and salinity use separate thresholds.

Derived salinity

Practical salinity must pass its own envelope and spike tests and also requires both temperature and conductivity to pass QC.

L2 plots

The 24-hour QC and depth plots show blue kept observations and red removed observations. The full L2 time-series and depth plots show only QC-kept observations while retaining estimated envelopes where applicable. The direction plot selects a 12-hour window with more complete up/down profiles; up profiles are black and down profiles are purple. Engineering points use red centered one-hour means for voltage and current.

24-hour QC time series
Full-period QC time series
Up/down profile direction time series: selected 12-hour window
24-hour depth profiles
Full-period QC-kept depth profiles
Engineering time series

L3 processing

Inputs and grid design

L3_process.py selects QC-approved upward L2 profiles and averages temperature, conductivity, and practical salinity into one-meter depth bins at two-hour and four-hour resolutions. The 2-hour bins preserve more short-term variability, while the 4-hour bins combine more profiles and generally provide more complete coverage.

The 1 m grid matches the typical sub-meter to approximately 1–2 m L2 vertical spacing, with a median spacing near 0.7 m. It is a grid interval rather than a claim of 1 m effective resolution; per-bin sample counts should be consulted.

Interpolation

The interpolated products use bounded local two-dimensional interpolation within five time bins on either side—up to 10 hours for the 2-hour product and up to 20 hours for the 4-hour product—and within 10 m vertically. Interpolation is restricted to overlapping sampled depth ranges, so large February–May gaps are not bridged and profiles are not extended above or below their sampled limits.

L3 2-hour binned sample counts
L3 4-hour binned sample counts

Output and NetCDF metadata

L3_netCDF.py assembles the compressed L3 CF-1.13 NetCDF-4 product from L3_variables.json and L3_globals.json. It includes binned and interpolated temperature, conductivity, practical salinity, sample counts, density, and mixed-layer-depth variables for both temporal resolutions.

Dimensions and storage

time_2hr=2042 time_4hr=1021 depth_l3=290 platform=1. The product is NetCDF-4 with enabled for data variables (zlib, level 4, shuffle).

Variables

NameTypeDimensionsUnitsLong name / flags
latitudefloat32platformdegree_northSAFARI surface mooring latitude
longitudefloat32platformdegree_eastSAFARI surface mooring longitude
time_2hrint64time_2hrseconds since 1970-01-01 00:00:00Zcenter time of 2-hour L3 averaging interval
time_4hrint64time_4hrseconds since 1970-01-01 00:00:00Zcenter time of 4-hour L3 averaging interval
depth_l3float32depth_l3mcenter depth of 1 m L3 averaging bin
temperature_2hr_binnedfloat32time_2hr, depth_l3degree_Celsius2-hour mean QC-approved upward-profile in-situ temperature
temperature_2hr_binned_interpolatedfloat32time_2hr, depth_l3degree_Celsius2-hour mean QC-approved upward-profile in-situ temperature with limited 2D interpolation
temperature_4hr_binnedfloat32time_4hr, depth_l3degree_Celsius4-hour mean QC-approved upward-profile in-situ temperature
temperature_4hr_binned_interpolatedfloat32time_4hr, depth_l3degree_Celsius4-hour mean QC-approved upward-profile in-situ temperature with limited 2D interpolation
conductivity_2hr_binnedfloat32time_2hr, depth_l3mS cm-12-hour mean QC-approved upward-profile electrical conductivity
conductivity_2hr_binned_interpolatedfloat32time_2hr, depth_l3mS cm-12-hour mean QC-approved upward-profile electrical conductivity with limited 2D interpolation
conductivity_4hr_binnedfloat32time_4hr, depth_l3mS cm-14-hour mean QC-approved upward-profile electrical conductivity
conductivity_4hr_binned_interpolatedfloat32time_4hr, depth_l3mS cm-14-hour mean QC-approved upward-profile electrical conductivity with limited 2D interpolation
practical_salinity_2hr_binnedfloat32time_2hr, depth_l312-hour mean QC-approved upward-profile practical salinity
practical_salinity_2hr_binned_interpolatedfloat32time_2hr, depth_l312-hour mean QC-approved upward-profile practical salinity with limited 2D interpolation
practical_salinity_4hr_binnedfloat32time_4hr, depth_l314-hour mean QC-approved upward-profile practical salinity
practical_salinity_4hr_binned_interpolatedfloat32time_4hr, depth_l314-hour mean QC-approved upward-profile practical salinity with limited 2D interpolation
temperature_2hr_countuint32time_2hr, depth_l31number of QC-approved upward-profile temperature samples in each 2-hour bin
temperature_4hr_countuint32time_4hr, depth_l31number of QC-approved upward-profile temperature samples in each 4-hour bin
conductivity_2hr_countuint32time_2hr, depth_l31number of QC-approved upward-profile conductivity samples in each 2-hour bin
conductivity_4hr_countuint32time_4hr, depth_l31number of QC-approved upward-profile conductivity samples in each 4-hour bin
practical_salinity_2hr_countuint32time_2hr, depth_l31number of QC-approved upward-profile salinity samples in each 2-hour bin
practical_salinity_4hr_countuint32time_4hr, depth_l31number of QC-approved upward-profile salinity samples in each 4-hour bin
mixed_layer_depth_2hr_binnedfloat32time_2hrmestimated mixed-layer depth from 2-hour binned profiles relative to the shallowest sampled bin
mixed_layer_depth_2hr_binned_interpolatedfloat32time_2hrmestimated mixed-layer depth from 2-hour interpolated profiles relative to the shallowest sampled bin
mixed_layer_depth_4hr_binnedfloat32time_4hrmestimated mixed-layer depth from 4-hour binned profiles relative to the shallowest sampled bin
mixed_layer_depth_4hr_binned_interpolatedfloat32time_4hrmestimated mixed-layer depth from 4-hour interpolated profiles relative to the shallowest sampled bin

Global attributes

titleSAFARI 2025 WireWalker Level-3 gridded upward-profile CTD observations
summaryThis Level-3 dataset contains quality-controlled upward WireWalker profiles from the attached RBR Concerto CTD, averaged into one-meter depth bins at both two-hour and four-hour temporal resolutions. For each resolution, the product includes the original binned observations, a separate two-dimensional interpolation product limited to 10 m vertically and five time bins on either side, and per-bin sample counts. Variables include temperature, electrical conductivity, and practical salinity.
ConventionsCF-1.13
netcdf_version4.9.2
date_created2026-10-06T13:35:50Z
time_coverage_start2025-11-21T01:00:00Z
time_coverage_end2026-05-10T03:00:00Z
time_coverage_durationP170D
time_coverage_resolutionPT2H
geospatial_vertical_min8.5
geospatial_vertical_max297.5
geospatial_vertical_unitsmeters
geospatial_vertical_positivedown
licenseCC BY-NC 4.0

QC flags use uint8 values with 0=bad and 1=good. Data-variable valid_min and valid_max attributes are calculated from QC-approved values; the vertical geospatial limits are likewise based on pressure-QC-approved depths.

Mixed-layer estimate

The L3 mixed-layer estimate uses TEOS-10 density from gsw. It identifies the dominant 5 m-smoothed density-gradient peak between 30 and 150 m, applies a minimum gradient threshold, and uses a gap-aware centered 24-hour temporal filter plus short-gap filling.

Plots

Binned and interpolated plots show temperature and practical salinity. Interpolated products include temperature and salinity contours. Separate interpolated density plots show temperature, density, and the mixed-layer estimate. The plots end on 10 March 2026, include daily minor time ticks, and show the NetCDF filename and creation date in the upper-right. Count plots show temperature and salinity coverage using fifteen positive-count bins with zero-count cells clear.

L3 2-hour binned temperature and practical-salinity time-depth averages
L3 2-hour interpolated temperature and practical-salinity time-depth averages
L3 4-hour binned temperature and practical-salinity time-depth averages
L3 4-hour interpolated temperature and practical-salinity time-depth averages
L3 2-hour interpolated temperature, density, and mixed layer
L3 4-hour interpolated temperature, density, and mixed layer