Skip to content

iowarp/clio-data

v1.0.0BSD-3-Clause

The clio-data agent's plugin: its mission, assess-data-quality (what a variable states, checks, comparisons, visual checks) and convert-scientific-data (the HDF5/ADIOS2 to Parquet dtype map, skip-or-ask rules, and a bundled converter that reads back and verifies rows, nulls, NaNs, dtypes and per-column checksums).

GitHub
skills

convert-scientific-data

Convert HDF5 (including netCDF-4) datasets or ADIOS2 BP4/BP5 variables to Parquet with an explicit dtype map, decide when a field is skipped and when to ask, and prove the result with row, null, NaN, dtype and per-column checksum checks. Bundles convert_to_parquet.py, which writes, reads back and verifies.

Read SKILL.md at the source

Pinned to revision 458d7da9d653, so it is the text this page describes rather than whatever the author pushed since.

Files

Every link opens the file at its source, pinned to the revision this page describes.