Uncertainty | CoMet Toolkit

Introduction to uncertainties

Sun, 15 Jun 2025 00:00:00 +0000

Navigating the world of uncertainties can be quite tricky, as often terms such as errors and uncertainties are used interchangeably, even though they do not describe the same thing.

In this brief introduction to uncertainties and their propagation, we will highlight the most important concepts and signpost you to useful resources.

By providing learning materials covering relevant topics at varying levels of depth, we want to ensure that you can fully understand and therefore implement the capabilities of this toolkit.

❔ Why do we care about uncertainties?

Uncertainties are a vital aspect of the measurement science, as they provide credibility and trust in measured data.

To ensure that everyone is on the same page, a great place to start is this Measurement Good Practice Guide put together by the National Physical Labpratory (NPL).

It covers all the relevant terminology, best practises, and examples for understanding the uncertainty of measurements.

As mentioned at the beginning of this article, we can confirm that uncertainties and errors are, in fact, not the same thing:

Uncertainty is a quantification of the doubt about the measurement result.

Error is the difference between the measured value and the “true value” of the thing being measured.

📜 Metrology & the Guide to Uncertainties in Measurements

Metrology, the science of measurement, is the discipline that maintains the SI and the associated system of measurement, ensuring measurements are stable over time and measurement standards equivalent worldwide. These properties are supported by the principles of metrological traceability: uncertainty analysis and comparison.

The Guide to the expression of Uncertainty in Measurement or GUM for short, is the ultimate resource to refer to, when you are evaluating measurement uncertainties.

All the official documentation of GUM as well as a guide to the International Vocabulary of Metrology (VIM) are available here.

🏆 Best practice framework for EO

To ensure credible and reliable interpretation of environmental observations from satellites and in-situ measurements, Committee on Earth Observation Satellites (CEOS) has established and endorced the Quality Assurance framework for Earth Observation (QA4EO).

This framework requires that associated uncertainty information is provided for all measurements. Additionally, it highlights the importance of understanding the error-covariances in the data.

Approaches defined within QA4EO enable the EO community to develop quantitative characterisation of uncertainty in EO data.

QA4EO developed a 5-steps approach to do an uncertainty budget. These 5 steps are described on this CoMet page and in the QA4EO process document.

However, practically implementing these methods is not trivial and can be time consuming.

As a way to facilitate this, the CoMet Toolkit was developed as a means to store and propagate uncertainty and error-correlation information.

✔️Other useful resources for Uncertainty in Earth Observations (EO)

A great resource that describes all the various aspects of uncertainty propagation for satellite EO data is FIDUCEO.

This project provides a comprehensive guide to understanding and implementing:

QA4EO uncertainty budget

Sat, 14 Jun 2025 00:00:00 +0000

QA4EO is a best practice framework, based on the principles of metrology, to establish The Global Earth Observation System of Systems — based on coordinated and harmonised processes and activities that enable interoperability. QA4EO was established and endorsed by the Committee on Earth Observation Satellites (CEOS). QA4EO ensures credible and reliable interpretation of environmental observations from satellites and in-situ measurements by requiring that associated uncertainty information is provided. A set of guidelines was developed on how to apply the QA4EO principles to generate metrologically-rigorous data products and to perform uncertainty analysis.

There are five steps towards a metrological uncertainty analysis. These steps are described in detail in the QA4EO Process document which provides Step-by-step guidance on implementing a metrological approach to uncertainty analysis. The 5 steps are: • Step 1: Define the measurand and measurement model • Step 2: Establish the traceability with a diagram (see example at the top of this page) • Step 3: Evaluate each source of uncertainty and fill out an effects table • Step 4: Calculate the data product and uncertainties • Step 5: Record information about the uncertainty analysis for long term data preservation purposes (implicit above) and summarise for today’s users

The CoMet toolkit is particularly useful for steps 4, (see our page on propagating uncertainties through a measurement function) and step 5 (see our page on error correlation and how to store it).

As discussed in the QA4EO Process document, Effects Tables (step 3) are a useful way to record and report the information required to fully parameterise an error-correlation effect. However, to use this information in a processing chain, it must be provided digitally. The CoMet Toolkit defines a mechanism for this, with a metadata standard (see UNC specification) that enables the creation of Digital Effects Tables stored in NetCDF files. In this way, uncertainty information can be written, read,

Punpy interfaces with obsarray to make uncertainty propagation as efficient and easy to use as possible. The digital effects tables produced with obsarray can be propagated through measurement functions using punpy, without the need for providing additional information. The data has thus been encoded with all relevant error-covariance information, though users do not need to interact with it. Together these tools enable both the experienced and inexperienced user to efficiently include uncertainties throughout their data processing, and thus make their datasets more reliable and interpretable. Optional keywords provide the user with the flexibility to deal with all kinds of complex use cases. For further info, we refer to the punpy and obsarray documentation.

Propagating uncertainties through a processing chain

Fri, 13 Jun 2025 00:00:00 +0000

When determining the value of a given measurand (i.e. the quantity intended to be measured), some processing typically needs to be performed to calculate the measurand based on some input quantities. In some cases, this processing is relatively simple, and can be done with a simple analytical function. In other cases, a full processing chain is needed, which can contain a whole range of computational processing steps. In metrology (GUM), this relationship between the measurand and the input quantities is refered to as a measurement model or a measurement function. For some further information, we refer to this FIDUCEO tutorial.

In order to calculate the uncertainty on the measurand, one needs to propagate uncertainties on the input quantities through the measurement model. There are various methods that can be used for this. Two commonly used ones are the Monte Carlo (MC) method and the Law of Propagation of Uncertainties. For further detail on these methods, see the GUM and its supplements, the QA4EO metrology document, or the punpy ATBD.

The CoMet toolkit implements both the Monte Carlo and the Law of Propagation of Uncertainty methodologies for uncertainty propagation. The CoMet toolkit interface to both of these methodlogies is exactly the same, so the user can easily switch between them and compare the results. There are two ways the input data and their uncertainties can be ingested in the CoMet toolkit. Either they can be ingested manually as numpy arrays, or they can be ingested using `Digital Effects Tables’, defined with obsarray.

In order to propagate uncertainties using the CoMet toolkit, one needs to be able to write the measurement function as a Python function which takes the input quantities as arguments and returns the measurand. Inside this measurement function, there can be a varying range of complexity. It could be a simple analytical function, or a full processing chain, including calls to other external software. The CoMet toolkit treats this measurement function somewhat as a blackbox and simply modifies the inputs, and analyses the outputs.

Error correlation and how to store it

Thu, 12 Jun 2025 00:00:00 +0000

See this FIDUCEO tutorial for further information on how to evaluate error-correlation information.

CoMet Toolkit vs. Atmospheric Weather Balloons

Tue, 13 May 2025 00:00:00 +0000

🎈 Background

Global Climate Observing System (GCOS) ensures access to long-term climate observations. Within GCOS, the GCOS Reference Upper-Air Network (GRUAN) plays a key role by providing reference-quality atmospheric balloon data.

GRUAN data includes measurements such as temperature, humidity and pressure. This data is traceable to SI standards, and published with detailed uncertainty and metadata. At present, GRUAN does not report detailed error correlation across its data products.

This article outlines a study which has successfully deployed CoMet Toolkit to acquire covariance information of GRUAN data.

🗸 Aims

Investigate GRUAN data product uncertainty components.
Obtain covariance matrices for temperature (T) and relative humidity (RH).
Assess how different data processing strategies impact covariance.

📋 Data

GRUAN data is gathered via radiosondes mounted on weather balloons, recording vertical atmospheric profiles at high resolution (1–2 seconds per point) reaching altitudes of 30–35 km.

We selected two radiosonde launches from Lindenberg, Germany (GRUAN’s lead centre) on January 1st, 2024 at 00:00 and 12:00 UTC. The data is available in NetCDF format, and includes full metadata and GRUAN-validated uncertainty components.

Each balloon ascent collects thousands of measurements. To reduce the number of data points, this data is typically layered. For instance, 137-layer ECMWF pressure coordinate system, is commonly used in atmospheric modelling.

🗸 Uncertainties

For each measurement point, GRUAN reports the combined uncertainty as well as three uncertainty components:

Uncorrelated: random, no relationship between points exists.
Spatially Correlated: data within a single vertical profile is spatially related
Temporally Correlated: data across different soundings and/or sites is temporally related (e.g., due to shared calibration).

GRUAN assumes full (r = 1) correlation for spatial and temporal components—an important simplification carried into the analysis.

☄️CoMet Setup

Covariance and correlation matrices were calculated using CoMet.

obsarray: used to re-structure weather balloon measurements, their uncertainty, and metadata (units, dimensions)
comet_maths: performed the mathematical operations to compute covariance matrices, correlation information

🗂️ Results

Using CoMet, raw covariance matrices were computed from the unprocessed midday profile.

Temperature and RH matrices show strong diagonals (variance) and structured off-diagonal patterns (systematic uncertainty).
The RH covariance scale is notably higher, especially due to sensitivity to solar radiation during daytime launches.

In layered heatmaps:

White regions indicate missing data beyond balloon burst.
Mean-layered data shows reduced variance (weaker diagonal).
Subsampled data retains more variance but still shows strong off-diagonal structure due to full assumed correlation.

Notably, the Upper Troposphere–Lower Stratosphere (UTLS) region exhibits distinct covariance patterns.

🗸 Layered Relative Humidity Covariance

Layered RH matrices differ significantly:

Covariance values are up to 100× greater than for temperature.
Heatmaps show a checkerboard pattern, indicating high inter-layer correlation.
Averaging reduces the variance, and temporal correlations dominate the structure, reflecting RH’s rapid environmental sensitivity.

These findings underscore the need to treat RH and temperature differently in uncertainty-aware models.

🗸 Summary of Findings

This project:

Developed and validated a method for computing covariance matrices from GRUAN RS-41 data using CoMet,
Tested multiple data processing strategies (mean, subsample, integration),
Provided practical recommendations through a GRUAN Technical Note.

The results help researchers better account for correlated uncertainty when working with radiosonde data.

🗸 Future Work

Planned next steps include:

Incorporating Tier 2 uncertainty sources,
Implementing partial correlation handling,
Expanding to other variables (e.g., pressure, wind, radiation).

These developments will enhance climate models’ treatment of uncertainty and ensure even greater accuracy in long-term climate monitoring.

CoMet Toolkit used in HYPERNETS Processor

Fri, 14 Jun 2024 00:00:00 +0000

🎈 Background

The LANDHYPERNET and WATERHYPERNET networks (which together make up the HYPERNETS network) consist of a set of autonomous hyperspectral spectroradiometers (HYPSTAR®) acquiring fiducial reference measurements of surface reflectance at various sites covering a wide range of surface types (both land and water) for use in satellite Earth observation validation and remote sensing applications.

The HYPERNETS_PROCESSOR is a Python software package to process the HYPERNETS in-situ hyperspectral raw data to reflectance and other variables.

🗸 Aims

In order to achieve fiducial reference measurement quality, and be optimally useful as a satellite validation reference, uncertainties need to be propagated through each step of the HYPERNETS processing chain, taking into account temporal and spectral error-covariance.

📋 Data

The HYPERNETS instruments measure downwelling hemispherical irradiance, and upwelling radiance. The raw data is provided as digital numbers, and combined with gains and non-linearity coefficients from Lab calibrations. The irradiance and radiance data are combined to calculate reflectance. These data are stored in different product files from L0-L2B.

🗸 Uncertainties

Three uncertainty contributions are tracked throughout the processing.

Random uncertainty: Uncertainty component arising from the noise in the measurements, which does not have any error-correlation between different wavelengths or different repeated measurements (scans/series/sequences).
Systematic independent uncertainty: Uncertainty component combining a range of different uncertainty contributions in the calibration. Only the components for which the errors are not correlated between radiance and irradiance are included.
Systematic uncertainty correlated between radiance and irradiance: Uncertainty component combining a range of different uncertainty contributions in the calibration. Only the components for which the errors are correlated between radiance and irradiance are included. This error-correlation means this component will become negligible when taking the ratio of radiance and irradiance (i.e., in the L2A reflectance products), which is why we separate it from the systematic independent uncertainty.

The two systematic uncertainty components are both based on lab calibrations. Since the same lab calibration is used within the HYPERNETS_ PROCESSOR for repeated measurements (scans/series/ sequences), the errors in the systematic uncertainty components are assumed to be fully systematic (errorcorrelation made up of ones) with respect to different scans/series/sequences. With respect to wavelength, we combine the different error-correlations of the different contributions and calculate a custom error-correlation matrix between the different wavelengths.

CoMet Setup

The HYPERNETS_PROCESSOR uses a Monte Carlo (MC) approach, to propagate uncertainties and error-correlations between product levels. This MC approach is implemented using punpy.

The HYPERNETS products themselves are stored as digital effects tables using obsarray.

Results

An example of the propagated uncertainties on the HYPERNETS radiances is shown below.

These uncertainties can be propagated throughout further processing chains (see e.g. this example)

Summary of Findings

The HYPERNETS_processor is operationally producing product files stored as digital effects tables. Uniquely for this type of processing, multiple different types of uncertainty (including error-correlations) are propagated through each of the processing levels.

These uncertainties have been very useful for the validation of satellite data (e.g. FLEX, CHIME, ENMAP, S2, …).

Future Work

There are a number of uncertainty components that are still missing from the HYPERNETS uncertainty budget (currently a placeholder uncertainty of 2% is added instead). Dedicated studies will be performed to better quantify each of the missing components.