uw-ssec/zarr-chunk-optimization
Benchmarking and optimization tools for Zarr chunking strategies on cloud object stores. Measures performance across access patterns and generates recommendations based on Nguyen et al. (2023) methodology.
Identifies, formalizes, and prioritizes data access patterns for multi-dimensional Zarr datasets, translating user workflow descriptions into weighted, benchmark-ready pattern definitions with xarray operation mappings. Use when defining or optimizing access patterns before benchmarking chunk configurations, when translating informal user workflow descriptions ("I make maps", "I need time series") into xarray operations, when assigning weights to mixed access patterns for a shared dataset, or when diagnosing slow reads caused by a mismatch between chunk layout and access pattern.
Benchmarks and optimizes Zarr chunking strategies for multi-dimensional scientific datasets, measuring wall-clock time, peak memory, and I/O metrics across spatial, time-series, and spectral access patterns following Nguyen et al. (2023) methodology. Use when the user asks about Zarr chunk sizing, storage layout optimization, benchmarking array access patterns, or choosing chunk configurations for scientific data on S3, GCS, or local storage.
Generates structured benchmark reports with configuration comparison tables, performance bias analysis, ranked recommendations, and trade-off explanations from Zarr chunking benchmark results. Use when the user has Zarr chunking benchmark data to analyze, needs to compare chunk configurations, wants performance recommendations for Zarr storage, or needs to present chunking strategy results to stakeholders.
Safely applies chunking configurations to Zarr datasets with validation, progress reporting, memory-bounded execution, and rollback safety. Supports local, S3, and GCS storage backends. Use when the user needs to rechunk a .zarr dataset, change chunk sizes, optimize Zarr storage layout, or apply a chunk configuration identified by benchmarking.
Generates synthetic Zarr datasets with configurable dimensions, shapes, data types, and compression for controlled chunking benchmarks. Supports local and cloud storage backends (S3, GCS). Use when the user needs to create test Zarr datasets for benchmarking, generate synthetic array data for performance testing, or create reproducible benchmark inputs without access to production data.