Comet_maths | CoMet Toolkit

CoMet Toolkit vs. Atmospheric Weather Balloons

Tue, 13 May 2025 00:00:00 +0000

🎈 Background

Global Climate Observing System (GCOS) ensures access to long-term climate observations. Within GCOS, the GCOS Reference Upper-Air Network (GRUAN) plays a key role by providing reference-quality atmospheric balloon data.

GRUAN data includes measurements such as temperature, humidity and pressure. This data is traceable to SI standards, and published with detailed uncertainty and metadata. At present, GRUAN does not report detailed error correlation across its data products.

This article outlines a study which has successfully deployed CoMet Toolkit to acquire covariance information of GRUAN data.

🗸 Aims

Investigate GRUAN data product uncertainty components.
Obtain covariance matrices for temperature (T) and relative humidity (RH).
Assess how different data processing strategies impact covariance.

📋 Data

GRUAN data is gathered via radiosondes mounted on weather balloons, recording vertical atmospheric profiles at high resolution (1–2 seconds per point) reaching altitudes of 30–35 km.

We selected two radiosonde launches from Lindenberg, Germany (GRUAN’s lead centre) on January 1st, 2024 at 00:00 and 12:00 UTC. The data is available in NetCDF format, and includes full metadata and GRUAN-validated uncertainty components.

Each balloon ascent collects thousands of measurements. To reduce the number of data points, this data is typically layered. For instance, 137-layer ECMWF pressure coordinate system, is commonly used in atmospheric modelling.

🗸 Uncertainties

For each measurement point, GRUAN reports the combined uncertainty as well as three uncertainty components:

Uncorrelated: random, no relationship between points exists.
Spatially Correlated: data within a single vertical profile is spatially related
Temporally Correlated: data across different soundings and/or sites is temporally related (e.g., due to shared calibration).

GRUAN assumes full (r = 1) correlation for spatial and temporal components—an important simplification carried into the analysis.

☄️CoMet Setup

Covariance and correlation matrices were calculated using CoMet.

obsarray: used to re-structure weather balloon measurements, their uncertainty, and metadata (units, dimensions)
comet_maths: performed the mathematical operations to compute covariance matrices, correlation information

🗂️ Results

Using CoMet, raw covariance matrices were computed from the unprocessed midday profile.

Temperature and RH matrices show strong diagonals (variance) and structured off-diagonal patterns (systematic uncertainty).
The RH covariance scale is notably higher, especially due to sensitivity to solar radiation during daytime launches.

In layered heatmaps:

White regions indicate missing data beyond balloon burst.
Mean-layered data shows reduced variance (weaker diagonal).
Subsampled data retains more variance but still shows strong off-diagonal structure due to full assumed correlation.

Notably, the Upper Troposphere–Lower Stratosphere (UTLS) region exhibits distinct covariance patterns.

🗸 Layered Relative Humidity Covariance

Layered RH matrices differ significantly:

Covariance values are up to 100× greater than for temperature.
Heatmaps show a checkerboard pattern, indicating high inter-layer correlation.
Averaging reduces the variance, and temporal correlations dominate the structure, reflecting RH’s rapid environmental sensitivity.

These findings underscore the need to treat RH and temperature differently in uncertainty-aware models.

🗸 Summary of Findings

This project:

Developed and validated a method for computing covariance matrices from GRUAN RS-41 data using CoMet,
Tested multiple data processing strategies (mean, subsample, integration),
Provided practical recommendations through a GRUAN Technical Note.

The results help researchers better account for correlated uncertainty when working with radiosonde data.

🗸 Future Work

Planned next steps include:

Incorporating Tier 2 uncertainty sources,
Implementing partial correlation handling,
Expanding to other variables (e.g., pressure, wind, radiation).

These developments will enhance climate models’ treatment of uncertainty and ensure even greater accuracy in long-term climate monitoring.

CoMet v1 release

Mon, 15 Apr 2024 00:00:00 +0000

🎉 CoMet Toolkit v1.0 Released!

We’re excited to announce the release of version 1.0 — the first stable release of the CoMet Toolkit!

This milestone includes three core Python packages:

All three are now available on GitHub and can be installed directly via Python package index (pip):

- pip install comet_maths
- pip install punpy
- pip install obsarray

📋 These tools have been validated and tested, and are accompanied by full documentation:

Further updates and improvements will continue to be released on both pip and GitHub as the toolkit evolves!

Getting started with CoMet

Thu, 21 Mar 2024 00:00:00 +0000

Welcome 👋

In this brief guide we will walk you through the basic steps and prerequisites to get started with CoMet.

1.💡 Get familiar with the toolkit and its capabilities.

All the relevant information regarding the aims and functionality of CoMet Toolkit is outlined in the About Section.

But, in a nutshell,

CoMet stands for Community Metrology Toolkit and it a set of software tools that handle, process, and store measurement data uncertainties and error-correlation information.

It accounts for case- and source-specific characteristics of the measurements, and can be used to quantify uncertainties and the uncertainty budget, create digital effects tables, and overall validate measurements.

At this time, there are three individual tools:

1. obsarray
2. punpy
3. comet_maths

but more modules are planned to be developed and included in the future. For more detail, refer to the Tools Section.

2. 🗃️ Characterise the data/measurements that require the uncertainty propagation.

The main purpose of these tools, is to propagate uncertainties. To do that, you must have an overall understanding of the type of data/measurements you are working with.

For a general approach on determining an uncertainty budget, we refer to the 5-step QA4EO approach. See this page in our theory section, or the QA4EO process document.

To help you identify all the relevant information from your dataset, we have compiled a list of relevant questions and tips.

🗸 General

❔ What kind of data do you have?
❔ Does it require any pre-processing or filtering?
❔ How many datapoints do you have? Is the data memory-heavy?

🗸 Quantifying uncertainties on input quantities

❔ Can you list all the input quantities of your measurements?
❔ Can you identify all the error sources?
There are three types of errors, each with their own characteristics:
1. Random
2. Systematic
3. Structured

❕ Typically, each of the input quantities will be affected by one or more error effect!

🗸 Defining measurement function

❔ What is the analytic expression (i.e. measurement function) of your data?
❔ Do you have a more complex processing chain using external software?
❔ Can your measurement function be written as a Python function that has the input quantities as arguments, and returns the measurand?

Read more about the importance and functionality of measurement functions in our page on propagating uncertainties through a measurement function.

🗸 Determining error correlation

Once you have identified the various errors and their types, that are present in your measurements, it’s important to consider how these values and errors correlate with one another.

As defined by this FIDUCEO article on “The origin of error correlation”,

Correlation is a statistical measure of how two, or more, variables vary together.

To learn more about error correlation structures and examples in the context of Earth Observations, refer our page on error correlation and how to store it.

3. 🧾 Identify similarities between your specific requirements and the available examples.

Look through the available examples and documentation.
Plan out how the toolkit can be applied to your specific case study.
❔ Which tools and in what order will you use?

4. 🖥️ Install the tools

All the available tools are available on GitHub and installable via pip:

- pip install comet_maths
- pip install punpy
- pip install obsarray

Installing punpy will automatically install comet-maths and obsarray.

5. ✔️ Perform the uncertainty estimation and interpret the results.

After defining a measurement function, installing and importing all the relevant packages and data, it’s time to benefit from the power of CoMet!

🗸 Method breakdown

A general overview of the various capabilities and methods are compiled bellow.

store uncertainty and error correlation information
1. machine readable digital effects tables
2. UNC specification
Propagate uncertainties
1. 🎲 Monte Carlo (MC)
2. ⚖️ Law of Propagation of Uncertainty (LPU)
Interpolate data & uncertainties
1. Linear
2. Quadratic
3. Cubic
4. Gaussian Process Regression (GPR)
5. Extrapolate data

Several of the methods listed above apply to more than one of the applications. For more information refer to examples.

6. 📈 Advanced use.

In this section, we have highlighted certain tips for advanced use of the toolkit.

🗸 Managing memory and runtime

Certain products may have large RAM requirements, and the MC approach that is often used in CoMet can increase the RAM and runtime requirements by one or more orders of magnitude.

There are ways to manage memory and runtime, as described in the punpy documentation.

For example, often storing the error correlation between all the measurements along all dimensions in a dataset is often prohibitively memory intensive. Instead, it is usually possible to store the error correlation separately between different dimensions.

E.g. the HYPERNETS L2A surface reflectance data, has a wavelength and series dimension for which the error correlation are stored separately.
When propagating this information, using error correlation dictionaries can be useful, (see e.g. the end of this jupyter notebook example).

Examples

Tue, 19 Mar 2024 00:00:00 +0000

Below, we have compiled a list of relevant examples and linked their corresponding jupyter notebooks with detailed commentary. All the available tools can be used on their own or in conjunction with other modules. For additional information regarding individual packages, refer to the signposted sources throughout the website.

📦 Punpy as a Standalone Package

🗸 General use cases

An example showcasing the capabilitites of the punpy package can be found here.

This jupyter notebook covers the following concepts:

Calibration of L0 data to L1
Propagation of various types of uncertainties: - uncorrelated (random) uncertainties - fully correlated (systematic) uncertainties - uncertainties associated with structured errors
Correlation along one, two, or more dimensions of a variable
Multidimensional input quantities, where a certain correlation structure is known along one dimension, while the other dimensions are random or systematic.

🗸 Punpy vs. NIST

We also have compiled some validation examples here, where the punpy results are compared against the NIST uncertainty machine.

Here, we have replicated the following examples available on the NIST uncertainty machine user manual.

End-gauge calibration
Dynamic viscosity
Resistance
Stefan-Boltzmann constant
Voltage reflection coefficient

❕ All the obtained results are fully consistent with the results of the NIST uncertainty machine.

📋 Digital Effects Tables (DET)

🗸 Defining DET

A notebook containing examples that define digital effects tables is available here.

It covers the following concepts:

How obsarray can be used as a templater for efficiently making xarray datasets (both with and without uncertainties)?
How, using obsarray’s special variable types (uncertainties and flags), datasets including detailed uncertainty and covariance information as well as quality flags can be created?
An example for a digital effects table quantifying the uncertainties and error-correlation of the gas temperature, pressure, and the number of moles. Here, the uncertainties can be efficiently and easily propagated through a measurement function using punpy (learn more).

🗸 Utilising obsarray & punpy

An example showcasing the application of obsarray can be found here.
An example of using punpy with digital effects tables created with obsarray is explained here.

This notebook outlines how digital effects tables that are created with obsarray, can be propagated through a measurement function using punpy.

At first, we calculate the uncertainties in a volume of gas, using the ideal gas law and a digital effects table.
Then we quantify the uncertainties and error-correlation of the gas temperature, pressure and amount of substance.

☄️ Comet_maths interpolation

🗸 How to interpolate data with uncertainties?

A jupyter notebook for interpolation with uncertainties can be found here.

This example covers the following concepts:

Interpolation
1. Linear
2. Quadratic
3. Cubic
Unknown input uncertainties (e.g. model uncertainties)
Known measurment uncertainties
Monte Carlo uncertainty propagation
Extrapolation
1D interpolation along high-resolution example

☄️ Curepy retrieval

🗸 How to obtain uncertainties for inverse problems?

A jupyter notebook for retrievals with uncertainties can be found here.

This example allows you to get familiar with the curepy tool, and covers the following concepts:

Propagate uncertainties through a basic inverse problem (harmonisation of two sensors).
Pass all relevant inputs to curepy
1. Measurements
2. Measurement function
3. Prior
4. Ancillary params
Run curepy retrieval to obtain state vector with uncertainties and error correlation
1. MCMC
2. Optimal Estimation

🗂️ Project specific examples

In this section, we have compiled a list of external projects and examples that have utilised the CoMet Toolkit.

🗸 HYPERNETS example

🛰️ LANDHYPERNET flags and uncertainty propagation (through band integration over S2 SRF) is available here.