Handling of normalised (scaled) data derived from variables with CF standard names where units aren't known/useful

[lhmarsden]

Maybe there is already a solution for this, but...

I’d like to raise a question about how to handle cases where scientific data (with a well-defined CF standard name and physical units) are stored in a normalised or scaled form for visualisation or distribution, rather than as physically meaningful values.

Example use case
A dataset contains toa_outgoing_radiance_per_unit_wavelength (W m-2 sr-1 m-1). For distribution, the values are min–max normalised to the 16-bit integer range [0, 65535]. The resulting stored data are dimensionless “digital numbers” (DN).

Problem
The CF standard name toa_outgoing_radiance_per_unit_wavelength implies canonical units of radiance. If the stored values are instead dimensionless, then:

• Using the same standard_name would be misleading, because the units are no longer compatible.
• Leaving off the standard_name means we lose the formal link to the underlying quantity.

Questions

1. Should CF define a way to indicate that a variable is a normalised form of another variable with a known standard name?
   • e.g. an attribute like normalised_of = "toa_outgoing_radiance_per_unit_wavelength".
2. Does this require proposing new “normalised_…” standard names for every existing standard name? This seems impractical, as the combinations would be large.
3. Is there scope for a different, more generic CF solution?

[DocOtak]

Take a look at the packed data section of the CF Convention. As I think this is what you might be describing. The CF standard doesn't seem to have the formula, but it's:

actual value = (stored value * scale_factor) + add_offset

This comes out of the netCDF user guide (NUG).

[lhmarsden]

I think this is different. scale_factor and add_offset are usually used to save storage space. They are used when the absolute, physical value is known an recoverable.

In some cases, like in my example, the absolute values are not important. I don't even know what they are. The data are used for visualisation purposes, and what is important is how large or small values are relative to each other, not what their absolute values are. So they are normalised.

[JonathanGregory]

Dear Luke @lhmarsden

You're right that scale_factor and add_offset are used for compression, to save storage although, as Barna @DocOtak says, a normalisation in your sense is the same sort of computation. Another way to interpret a scaling is that you're changing the unit, e.g. from W m-2 to 0.005344 W m-2, so that 350.22 W m-2 is stored as 65535 units. With that interpretation, a normalisation isn't making the quantity dimensionless; it's not different from changing a distance from m to mm.

However, the crucial point you make is that you don't know the scale factor, or equivalently that you don't know the units. But in that case it's impossible to compare data variables from different datasets, isn't it, even if you know they're the same quantity. The first purpose of CF is to provide metadata sufficient for deciding whether data from different sources is comparable. Although this sort of normalisation is useful, is it within the scope of CF to describe such normalised data?

Best wishes

Jonathan

[lhmarsden]

Thanks @JonathanGregory for your insight.

I am not sure whether this falls under the scope of CF or not. I suppose for most variables one should always provide the data with units.

The variable "toa_outgoing_radiance_per_unit_wavelength" is in my case (and I think in most cases) used for visualisation where the absolute values, or units, are not really important to most users - according to the people who provided me with the data, at least. Perhaps this is outside of the scope of CF, but we do have standard_names for:

• red_component_of_toa_outgoing_shortwave_flux_expressed_as_rgb_color
• green_component_of_toa_outgoing_shortwave_flux_expressed_as_rgb_color
• blue_component_of_toa_outgoing_shortwave_flux_expressed_as_rgb_color

"toa_outgoing_radiance_per_unit_wavelength" is very closely related to these. In fact, these are all computed from "toa_outgoing_radiance_per_unit_wavelength".

So perhaps a new standard_name is more appropriate in this case?

If you agree I can propose a new standard_name discussion.

[JonathanGregory]

Yes, you're right, @lhmarsden, it is similar to what we agreed for the RGB components, where I didn't feel the same discomfort because by saying expressed_as we acknowledged that it was no longer the original physical quantity. Yes, I think a new standard name for your particular use-case would be a good way to deal with it. Of course, if this subsequently turns out to be a large and general need, we can think about another way later.