CF Conventions
Mapping CF conventions onto Zarr #461
Replies: 10 comments 51 replies
|
In my view, making the CF Metadata Conventions available in Zarr stores is not a major departure from the current CF. The Zarr data model is very similar to the netCDF enhanced data model, and with very few exceptions, a dataset stored in a netCDF file can be converted to a Zarr dataset using the same encoding of attributes, variables, and groups. There are arguablymore differences between netCDF-3 and netCDF-4 than there are between netCDF4 and Zarr. Remember that netCDF is an API, not a file format, and the canonical implementation (netCDF-C) can read and write netCDF-3, netCDF-4, and Zarr2 datasets. There are many such Zarr versions of CF-netCDF datasets already in existence. I'm not sure what we are being invited discuss - is it "What should a Zarr convention that re-encodes parts of CF look like?", or maybe I've misunderstood? Thanks, |
|
I'm going to pull back from this thread -- if it's actually about how to get the OGC process unstuck with regards to geozarr... it's very much a different issue than I thought we were discussing. |
|
Hi Ethan, I suppose my reference to compression is incorrect: |
|
Hi Dave @dblodgett-usgs - The conclusion from last weeks GeoZarr/CF meeting, as I recall it, was that the CF community needed to discuss the various ways CF could be mapped onto Zarr. So, while this issue sprang out of GeoZarr/CF discussions, I don't think it is limited to the GeoZarr effort to encode CF. Though it is also, as Patrick mentions, the main CF discussion on GeoZarr (at least for now). |
|
Thanks Ethan. I see. I would have benefited from knowing that strategic detail more clearly. I have an unanswered question over on the geozarr-spec that leaves me pretty confused by the whole thing. Happy to get together to talk about it sometime if you'd like. I'll reiterate what I said up above -- the amount of CF-like data encoded in zarr in the wild means that much of the decision has been made already unless you want to orphan a ton of good data. I think the most productive thing at this point is to embrace the progress made to date, enumerate patterns in the wild, and iterate with the zarr developers' community around the commonly used implementations of CF-like data. If folks would be interested in starting that process, -- I'd be keen to work on it. I've been vocal about the fact that I think the move to create OGC SWG was premature. I would very much like to keep any work on this at a lower level or in something like an OGC interoperability experiment before taking it to the formality of a SWG. I was planning to reinvigorate work on this soon https://github.com/DOI-USGS/rnz as well as get pizzarr on CRAN. I see now that @pvanlaake has an R package that implements zarr 3 on Cran. So maybe some synergies there. |
|
Hi Dave, the CF-GeoZarr discussion has been announced multiple times over the past year so it's unfortunate that you were not up-to-date on these recent developments. But this discussion also builds on other CF Roadmap issues, such as the possibility of supporting CF on other storage formats.
This is an unfair characterization of this discussion in general and the suggested options in particular. Nobody is arguing to tell people not to use The bigger question for me is this: is an identical implementation of CF in Zarr the optimal solution for the CF/Zarr community over the long term? My personal opinion is that there are several parts of the CF conventions which are arguably due for a critical review. While that is not so easy for CF-on-netCDF due to the policy of backwards compatibility, starting with a new encoding of the same semantics for a new format does open up that possibility. I hope we may consider and discuss that issue because clinging on to the current CF encoding is not going to help CF over the coming decades. With regards to the process that you outline, that is what this discussion is supposed to be doing. It may be more practical to spin off a new discussion if you think that would be helpful. The idea was to do the initial brainstorming in this community, come to some consensus, and then reach out to the GeoZarr community for feedback. Should you have another approach in mind, then please go ahead. Happy to discuss R coding with you. Perhaps open an issue in the Best regards, |
Maybe I am missing something here, but isn't this a structurally new capability in relation to netCDF, and thus CF? If this is true, are there any desirable OGC/ISO (etc.) functionality that depend or would be substantially facilitated by this -- and is maybe outside what CF currently handles? |
|
Before I think further, could I clarify if you are suggesting that alternative Zarr-specific encodings (e.g. using nested attributes instead of container variables) are created as part of the actual CF standard? Thanks, |
|
In my previous comment I am suggesting effectively two things: (1) that the semantics of the conventions are decoupled from details of implementing the semantics in netCDF (such that it will become easier to support other storage formats for CF); and (2) that for each storage format an encoding is used that is most appropriate for that environment. |
Agreed -- but I don't think those are questions for the CF committee -- that is, efficiency isn't our concern, that's the concern of library implementers and users. That being said -- if there IS a way to increase efficiency that requires an extension / change of the CF Data Model, then we might want to consider making that change. Might that be the case here? It seems David has suggested that:
but also:
So I'm a bit confused -- is it a matter of whether the nesting of attributes is an "encoding" or an actual part of the data model? e.g. when you read a Zarr dataset with dataset attributes with s CF compliant reader -- should it flatten those for you? In this case though: IIUC, if nested attributes become part of the Data Model, then you would no longer be able to store an CF compliant dataset in netCDF?? Or would the netcdf libs need to grow a way to "flatten" and "unflatten" attributes? |
This happened a long time ago -in the CF data model. "Encoding independent" does not mean "can be applied to any encoding". It means that any encoding that preserves the logical structure can be used. Chris is right, and I was wrong to suggest how we could include of nested attributes in the the CF data mode (I as not advocating it - just imagining the mechanics). Nested attributes imply a logical structure, and for that the data model has its "constructs". The data model already does things very differently to the netCDF encoding (e.g. formula_terms attributes map to Coordinate Reference constructs), and recognising that nested attributes are just one way of encoding a construct falls into that category. |
This is exactly my point. Where you say "logical structure", I say "semantics", but otherwise we are saying the same thing. Whether or not that uses nested attributes is immaterial. Breaking down the process of ingesting a CF-compliant data set looks somewhat like this:
basic storage format/API = netCDF to date, perhaps native Zarr can be added, maybe Parquet and others at a later stage. From the Zarr perspective the question is then, can we develop a native Zarr CF encoding (as opposed to the CF-netCDF encoding possibly stored in Zarr) and make that available to the The hard part would be translating the CF encoding to the model and/or the tool, but - as Chris already noted - that is not the concern of the CF community: if it is useful then it will be used. |
|
IIUC, Zarr implements the HDF5 data model [*]. (at least it started that way) And netCDF4 is a restricted subset of HDF 5. Which means that mapping CF to Zarr could be as "simple" as mapping it to a strict subset of Zarr, like is done for HDF5. Which seems to be what ncZarr is -- so isn't that the easiest first step? Someone may want to use Zarr features that aren't netCDF compatible -- if so, then that requires some extension to CF, but in that case, I think it should be done in a broader way -- e.g. a way that supports other more expansive data models, such as well HDF5, and ???) NOTE: folks are using xarray as a file-format-independent array management system, and xarray implements the netcdf4 data model (I'm not sure how strictly), and to some extent the CF data model. And folks are using it with Zarr and CF compliant datasets just fine right now. NOTE: I see this for nczarr (as implimented by the netCDF C lib:
That really surprises me -- why in the world would the string type not be included? that seems like it would be native to zarr. But oh well. [*] this sure makes it look like Zarr is HDF (and be extension, netcdf) compatible. |
|
Zarr does not implement any specific data model. It is a very straightforward n-dimensional data cube, persisted as an array in a store that can have a hierarchy of groups, and with both arrays and groups having metadata. In that sense it more closely resembles the netCDF classic data model: just data structure and very basic semantics. COARDS defined coordinate variables, and CF still uses exactly that definition. HDF5 has the closely related concept of dimension scales. These approaches give meaning to the data arrangement: from a bare-bones Cartesian indexing space to a coordinate system that typically (but not necessarily) includes a spatio-temporal domain. But your point is otherwise valid. |
Or library :- ) I agree with both of these. However, as much as we all want standards to be independent from implementation, there is the fact that a standard without an implementation is useless, and that, in practice, implementations tend to drive standards, rather than the other way around. Which I think was the point about xarray -- not that we focus on a particular language or library (though we sure have a history of that with netcdf-c :-) ) -- but that xarray is widely used, and is driving the state of practice forward, whether we like it or not. That is, xarray already provides a way (however imperfect) to use CF with Zarr, and i can tell you that a lot of users don't think twice about the fact that technically there is no standard for CF in Zarr .... The good news is that xarray does (Mostly?) implement the netcdf (with some CF extensions) data model, so it works pretty well.
Is that still true, AI tells me it's no longer the case, but it can hallucinate ... But in any case, that's irrelevant to CF -- that's a "which library works well for my workflow" problem, not a CF problem. Anyway -- back to the point about relying on particular libraries -- ncZarr is a particular library, and it deserves special attention, as it comes from Unidate, and is integrated with netcdf-C -- but it shouldn't drive our discussion of CF. We've (that is, David :-) ) have made a lot of progress with decoupling CF from netCDF with the development of the CF Data Model. If the CF Data Model can be represented in Zarr, then we're done, yes? I think the real question at hand is what to do about features that Zarr (or any other file format) has that netCDF doesn't have. (Is that what the nested attributes issue is about??). Do we extend the CF data model is a way that can not be supported by netCDF? THAT would be a real change. I think of a Data Model as a specification of what exactly can be stored -- actual data, metadata, relationships between data, etc. if multiple file formats (encodings) can be store that data -- that is, if you can losslessly convert between them, then there's nothing new that has to be done. And I think that's the case with Zarr and netCDF -- they both can fully store the CF data model.
Am I missing something? My analogy is image storage -- a core data model for images is an array of pixels -- maybe 24 bit RGB or 32 bit RGBA. Any file format/ encoding that can store an array of 24 or 32 bit values is compatible -- no one has to define a new Data Model when someone comes up with a fancier compression scheme to store an image. IN that analogy, isn't Zarr simply a new encoding? |
|
We're ahead of schedule :-) That's all out now in the current release (v3.19) |
|
@davidhassell -- in cf-python, do you encode CF constructs in zarr in a way that's generally compatible with xarray and the netcdf API? Scanning the code a bit it's not entirely clear to me so I figured I'd just ask. |
|
Hi Dave (@dblodgett-usgs), When cf-python reads and writes Zarr it assumes the CF-netCDF API - i.e. attributes have the same meaning as they would in a netCDF-3 or netCDF-4 dataset. E.g. a It can read Zarr 2 (xarray-style) and Zarr 3, but only writes Zarr 3. It uses the python A reason why the code might seem to be a little opaque is because it really doesn't have much in the way of Zarr-specific code to get this to work. cf-pthon's data representation is, 100%, the CF data model, so its internal functionality is encoding independent. This is why If it finds nested attributes, they are happily read in, and written out again, but are not parsed or given any special treatment, just in the same way that an attribute called [*] As an aside, it took me a while to get this to work, because the Zarr dataset I was editing to put in a grid mapping dictionary was storing the same attributes in two different places! One with the directory of the variable itself, and a copy in the dataset's top-level directory. I edited the |
|
Interesting. I'll have to ponder that a bit further, but it sounds like, in essence, you are ending up with nczarr? Is the output from cf-python compatible with xarray and other implementations that attempt to read CF constructs from zarr? |
Yes, I think so.
Yes |
|
This might be somewhat tangential, but it might be of interest in this discussion to look at these two talks by Mark Kittisopikul (in chronological order or presentation)
Efficient Chunk Access across HDF5, Zarr, and TIFF”
https://www.youtube.com/watch?v=IEAcCmcOdJs
Using a HDF5 File as a Zarr v3 Shard
https://www.youtube.com/watch?v=c4b_yfIeHJc
…-Roy
On Feb 28, 2026, at 1:20 PM, Patrick ***@***.***> wrote:
It means that any encoding that preserves the logical structure can be used.
This is exactly my point. Where you say "logical structure", I say "semantics", but otherwise we are saying the same thing. Whether or not that uses nested attributes is immaterial.
Breaking down the process of ingesting a CF-compliant data set looks somewhat like this:
basic storage format/API -> CF encoding -> model -> tool -> user/application
basic storage format/API = netCDF to date, perhaps native Zarr can be added, maybe Parquet and others at a later stage.
CF encoding = CF semantics (everywhere the same!) structured to be optimal for the basic storage format/API.
model = CF data model as in Appendix I and as used in cf-python; I know of no other implementation; xarray and (my very own) ncdfCF use an implicit, specific data model.
tool = cf-python, xarray, ncdfCF, ... make CF data sets easily accesible with a published interface that end-users or application builders can use.
From the Zarr perspective the question is then, can we develop a native Zarr CF encoding (as opposed to the CF-netCDF encoding possibly stored in Zarr) and make that available to the model -> tool -> user/application level? In other words, could there be a Zarr-specific CF encoding which can then be ingested by the CF data model and cf-python and its user base, as well as to other tool chains?
The hard part would be translating the CF encoding to the model and/or the tool, but - as Chris already noted - that is not the concern of the CF community: if it is useful then it will be used.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
|
Staying on the tangent: we have engineered |
|
A typical use of Zarr data stores is to publish data over the internet, e.g. from a web server of through a cloud service. Zarr is increasingly favoured over netCDF (such as through a THREDDS server or OPeNDAP) because it scales better with increasing data request loads. Efficiency is key. Consider a bog-standard CMIP6 data set with a single data variable having T-Y-X dimensions and shape Reading from such a data variable encoded in the standard CF-netCDF style requires the user (or rather the tool that the user is using, such as Reading from the same data variable but encoded using a Zarr convention like 373KB may not sound like a major hurdle and the user may not even notice the additional download but for the data publisher, perhaps serving up thousands of data requests per day, that may be a different story. There is more to it than download size alone. The standard CF-netCDF encoding requires (at least) 21 HTTPS GET requests: one for the Something to consider, it would seem. |
|
I don't think what you say is true for most interesting cases at scale. Firstly, data is compressed, we always need a chunk index to go from array space to chunk space as chunks vary in size. Secondly, if, as is mostly the case, we used different logical datasets for each scientific variable, we have to read metadata for each one ... From the server point of view, it has to server up, for each variable, metadata, chunk indices and chunks. The first two are small beer compared to the chunks themselves. If folks only ever read one chunk of data maybe you'd start to care about the load associated with the first two, but that's not a prevalent use case - at least for climate data. It also turns out that we can write NetCDF to be as efficient as Zarr, but mostly we haven't, but we do (now) know how (more on that elsewhere)... The other big problem client side is that you can't properly parallelise things because of the (lack of) thread safety in the libraries based on the HDF5 library ... but we now have a Python solution to that which is thread safe. So most of the reasons why Zarr might be preferred can be alleviated. That's not to say people shouldn't use Zarr, but the reasons why one might use Zarr are not quite as obvious as they were ... |
|
I slightly drifted off topic. I think in practice the worst case reads with properly formatted data are for n-dimensional data in anetcdf, one file open, one metadata read, n-coordinate variable reads (likely one read each given sensible behaviour), one chunk index read ... then the chunks themselves. As I say, most interesting (and load inducing) use cases involve multiple chunks ... |
|
I have to admit to not knowing the details of zarr layout, but I can't see it being any worse?? |
|
I was explicitly not talking about the actual data, that is the same whichever format we look at. I was referring specifically to the metadata (which is what the CF encoding is all about). In many cases the data variable is much bigger than the metadata, granted, but that does not mean that we can forget about the metadata. And, yes, it can get much worse. ERA5-land on ECMWF has moved over to the ARCO format, optimized for extracting time series of data. NetCDF is only used to serve up the data to the users. In the ERA5-land data request form you can specify only a single location and a temporal domain over the entire data set, 1950 - now. If you select the entire time series you get (today) an 8MB download. The data variable uses the NC_FLOAT data type and uses up 2.55MB. The "valid_time" dimension has a length of 667,536 and uses the NC_INT64 data type that is commonly used for the DateTime class in Python, which is 5.1MB of data. So fully 2/3 of the download volume is metadata. |
|
Dear CF-folks, OK, I'll make a suggestion here. I think that some minor zarr encoding notes in the CF specification would be consistent with what the geozarr SWG is asking for. I don't think it would be absurd to try and progress key things as incremental changes to the spec that help focus the community of zarr implementers. If this proposal is worth pursuing, we would start the process of deliberating individual issues related to zarr encoding of cf. The conversation around CF-in-Zarr has gone in fits and starts and keeps stalling because individual issues haven't been isolated and dealt with separately. A strong separation of concerns would let the community make visible, durable progress on each in relative isolation instead of relitigating things over and over. I want to argue for rallying around the things that have been solved in this space and working toward conclusion on clearly defined and cohesive topics requiring some design consensus. The proposal: adopt a clean separation of concerns — this framework I lay out below, or an iteration on it. Treat the closed concerns as foundations to build on that are recognized stable aspects of CF in Zarr. Work the open concerns as parallel tracks where the technical solutions are largely in hand. NOTE: I don't mean to open debate on these items here -- the descriptions are to characterize each concern only. This comment is intended to propose that we break these out as independent issues to be worked incrementally and in relative isolation. Six concerns are actually in play. Two have reached working consensus already — recognizing the solved bits is itself valuable: Array Dimension Identity. The longstanding Missing Data Semantics. The conceptual distinction between Zarr's storage-level Four concerns are genuinely open, but each is self-contained enough to be tackled independently: Coordinate Role Identification. CF has clear and well-understood conventions for identifying dimension coordinates, auxiliary coordinates, and scalar coordinates. The gap is that Zarr has no structural location to express these roles. Tools that are not CF-aware cannot identify coordinate types without implementing CF's inference logic, which is a meaningful barrier to interoperability. The fix is small and self-contained: a Zarr convention adding an optional coordinate role attribute would close this without touching anything else in this list. Attribute Type Fidelity. NetCDF's typed attribute model — Axis Order Declaration. This is the most dangerous open problem because it fails silently. CF's CRS Representation Authority. This is where the most work remains, but it helps to be clear about what is and isn't in dispute. There is broad cross-community consensus — across PROJ 6+, pyproj, STAC, and the emerging GeoZarr Is this the right seperation of concerns? How else might we break up the issues to allow a more natural and cohesive incrementalism? Regards, Dave p.s. Pondering what it would look like to call out zarr encoding rules in the CF spec -- I was reminded that things like the "NUG" are formative documents for CF. I think a valid stance the CF community could take is that they expect something akin to the NUG (the ZUG?) to outline the encoding rules. If there's no appetite for solving the problems above, we could just say that, like the NUG, the ZUG is out of scope for CF. |
|
@davidhassell -- I'm glad to hear you agree on the strategy. The "case-by-case" nature of encodings leaking into CF is something we need to understand as clearly as possible. A critical review of the existing spec on this front would be instructive. My take on the geozarr conversation is that herding multiple groups of cats with significant collections of data they want to maintain backward compatibility with is just a bridge too far. Either way, I think there is a reasonable way forward within CF that may be of utility. Yes, I agree that the details of Array Dimension Identity are not fully worked out. But the core solution is clear. (trying not to play that discussion out here!) Respectfully, @pvanlaake, it is simply not an option to start fresh at this juncture. We need to be creative and engineer our way forward starting from what we have (which is a lot). There will be significant room for creativity and evolution, but it has to stem from current practice. In terms of sequencing, I think tackling this starting from fundamentals and working up in a bit of a dependency tree is the obvious way forward. That would be:
These can, to some degree, all be worked in parallel but I think it would be wise to get one or two settled to build a core assumption set and build some momentum to the broader strategy before getting the community wrapped around too many axles!! Regards, Dave |
|
Dear CF Folks -- I've just finished a proposed convention I've called "NZUG" that I would like to propose as addressing 1, 2, and part of 3 in my comment just above. https://github.com/zarr-conventions/nzug Auxiliary coordinates, axis order, and CRS representation are not addressed by design. I'll drop reference to this in the newer discussion on this topic -- not sure any significant follow up here is needed, but I wanted to close the loop. Regards, Dave |
|
Awesome! This is great! I suspect it could be used as "template" for defining the underlying data model / encoding required for CF, as we've been talking about in another discussion. |
|
Thanks, Dave. I though that the dimension-order thing was a solved problem in CF. It says (https://cfconventions.org/Data/cf-conventions/cf-conventions-1.13/cf-conventions.html#use-of-the-crs-well-known-text-format): It is the responsibility of the data producer to ensure that the list is consistent with the CRS WKT definition of CS AXIS, with the correct number of entries in the correct order (note: this is not a conformance requirement as CF conformance is not dependent on CRS WKT parsing). For example, a file has two coordinate variables, lon and lat, and a grid mapping variable crs with an associated crs_wkt attribute; the WKT definition defines the AXIS order as ["latitude", "longitude"]. The grid_mapping attribute is thus given a value crs:lat lon to define that where coordinate pairs are required, these shall be ordered (lat, lon), to be consistent with the provided crs_wkt string (and not order inverted). A 2-D array of (lat, lon) tuples can then be explicitly derived from the combination of the lat and lon variables. But it's possible I've misunderstood the issue! |
|
Hi David,
is the way it reads in the spec. This is saying that "axis order may be declared in a way that it is ambiguous to some readers" So solved in the sense that we know what the answer is but not solved in the sense that we've committed to the solution. This is an issue that will hang out there for a long time because we have two entrenched camps. One, broadly speaking the SFSQL / EWKT world of PostGIS (https://postgis.net/workshops/postgis-intro/geometries.html), who just say that x/y is canonical regardless of your CRS and the WKT2 / PROJJSON world of Proj6 where the CRS defines the axis order of the encoded coordinate pairs. I don't see the WKT world budging from insisting x/y order for lon/lat and the projjson world has very much decided that having flexible axis order such that the CRS spec can specify it is the only way forward. Those of us who have to deal with both worlds (PostGIS backed pygeoapi web services for example) just have to deal with the fall out. It's why hacks like "alwaysxy" have to exist in tools that obey axis order by default. https://pyproj4.github.io/pyproj/stable/api/transformer.html /rant Dave |
|
Hi Dave,
Of course it is an option. You apparently do not find it an attractive option, but that is a different issue. Ruling this out before the effort starts is a missed opportunity.
That we need to "start[...] from what we have" is a choice and it is a loaded choice. What we have is all tailored to netCDF and @davidhassell nicely summed up the scope of considering Zarr when he wrote:
That is a very narrow scope and not the topic of this discussion, really. David refers to the CF-netCDF encoding that we already have and incremental changes to that are nothing new, that is what we have been doing every day for many years. Is it really too big a leap of faith to take a step back and consider what CF would look like without netCDF? |
|
Did you mean to create a new thread in the discussion? This overall discussion has gotten very confusing with the various new threads that have cropped up. |
|
whether a new thread or not, let's all try to see each other's point of view. I'm getting nervous about the tone of some of the discussion. (And I, for one, am completely lost regarding the many facets of what it would take to move this forward.) |
|
On a couple levels @ChrisBarker-NOAA. Well done. @taylor13 -- I'm not quite sure who you are calling out here or if it's just a general comment. @pvanlaake -- I hesitate to respond in this new thread as it seemed like it was a response to the above discussion thread placed as a new thread in error. Please don't take my discourse here wrong. I see how it would be good to start from a more first principles place, but in my experience, doing that has not gone well unless there is a truly new paradigm without backward compatibility concerns. On many levels, it's important to recognize where (figurative) mass and gravity exist in the community and what momentum exists before making any major contributions or changes in this space. You've gotten a number of responses that are more or less... "say what?" in this thread as a result. I'll hold further comment here and perhaps join @ChrisBarker-NOAA in his fresh discussion. |
|
I'm on the same page as @dblodgett-usgs, here. CF progresses from the best-efforts of a finitely resourced, largely volunteer community. It has a general policy of considering concrete proposals for well-defined use-cases, which makes the most of its resources and reduces the chance of getting things wrong. If a specific use-case is presented in another discussion with a suggested course of action, I'd be happy to discuss it. David |
|
@pvanlaake: I think we need a new discussion for this topic. I've started one here: #463 If you (or anyone else) has a very different idea, then feel free to start your own discussion -- I'll probably close mine if it goes in another direction. |
|
@dblodgett-usgs: Thanks for nailing down the issues so clearly. A few thoughts on a couple points:
Honestly, this is a bit of a mess on CF with netCDF as well -- at least in the wild -- The overlap between I think this is also an example of a key problem -- a new "container" may not fully support all the CF concepts -- and a decision has to be made about what to do about that.
Hmm, it seems that for the most part, CF does not concern itself with data types -- e.g. a time coordinate can be stored with floats integers, etc ....
I would argue that that's a bug / missing feature in CRS parameterization. See the time example, we quite explicitly say that the choice of data type (or even units) for time says anything about the precision.
Any numeric metadata that give meaning to data type should probably be revisited.
Agreed -- though ideally CF would be inherently less tied to serialization details. |
CF does concern itself with data types - it generally doesn't say what the data type should be, but it does say that data must have a data type. E.g. encoding coordinate data as "start, stop, step" numbers with no data type in an Zarr attribute could be problematic, perhaps. Also note that storing binary numbers in decimal is generally lossy (apart from for dyadic fractions. My point (if any!) is to support the general observation that attribute data types is an area that will need careful attention in Zarr. |
that's what I meant :-) But this is another example of why it would be good to clearly define the data model -- if CF requires that any value should be of a clearly defined data type, that needs to be written down somewhere. And if those data types are restricted to a certain subset, then all the more important to clearly document that.
I didn't think that was the case -- the other way around is -- you can't represent 1/10th exactly in a binary, but you can represent, any binary fraction as a decimal. At least that's what AI tells me.
Indeed -- and it's a bit messy -- JSON itself has only decimal numbers, with no defined limit to precision. It can make a distinction between integers and non-integers, but that's about it. However, JSON tools make other assumptions:
So yes, that does need to be addressed, though I'm not sure it's THAT different than even netCDF -- which has a clearly defined data type for attributes, but tools may or may not strictly respect them -- that is, whatever data type is used in the netCDF, the tools may well store them in a different type, and it may not round trip exactly. It will take some thought, but perhaps CF does not have to be more specific than "string", "integer", "float". Hmm.... NOTE: It might worth looking at what ncJSON does: |
Uh oh!
There was an error while loading. Please reload this page.
Topic for discussion
Over the last 2 ~ 3 years, the GeoZarr community has looked into data models in the larger n-dimensional data ecosystem with the aim to find commonalities which might serve as the basis for GeoZarr. Late last year it was determined that this is too onerous a process and not providing a clear path forward. Instead, GeoZarr is now inviting user groups to submit “conventions” built on top of Zarr to demonstrate working arrangements. This would lead, conceivably, to a variety of approaches, each one tailored to its own user base. It is expected that there will be a convergence over time to a common approach on attaching a geospatial reference frame to the Zarr data model.
For the CF community this is a very positive development because it means that we can define our own Zarr conventions to map CF semantics onto Zarr. Given that this is a new direction for GeoZarr, there is also very little established practice in the form other Zarr conventions that CF should align with (or, inversely, can build upon) so our hands are effectively free. By acting now, the CF community can propose an arrangement that is CF-compliant, clean, clear and efficient, rather than having to make CF fit with established Zarr conventions.
This new discussion builds on a number of previous discussions and announcements on GeoZarr here:
The purpose of this new discussion is to bring these different efforts together and respond to the new approach in the GeoZarr community to demonstrate working arrangements, and, not least, to make the CF Metadata Conventions available to native Zarr stores.
Zarr conventions
A Zarr convention is represented by a JSON sub-schema embedded in the
"attributes"section of azarr.jsonfile that is attached to every group and array in the Zarr store. Being JSON, the convention can have nested attributes. The task at hand is therefore to define one or more Zarr conventions that (jointly) comprise the CF conventions. Two draft examples, demonstrating two separate approaches to mapping CF onto Zarr, are available here:CF: A direct mapping of the current CF structure to Zarr: https://github.com/zarr-conventions/CF. This Zarr convention was suggested by the GeoZarr SWG as a starting point for enabling CF-compliant data sets in Zarr. It is not intended to be used as-is.cs: Restructuring CF semantics on coordinates in Zarr: https://github.com/R-CF/zarr_conventions_cs. This is a Zarr convention that I put together to demonstrate a modular approach, in this case for defining a coordinate system (a "domain" in the CF data model) aligned with ISO/OGC terminology. Additional Zarr conventions would cover other parts of CF.Another interesting example would be the Zarr convention for DGGS, which is used by HEALPix (CF-1.13): https://github.com/zarr-conventions/dggs. Other currently defined Zarr conventions are listed here: https://github.com/zarr-conventions.
Points for discussion
Making the CF Metadata Conventions available in Zarr stores is a major departure from the current CF. While the Zarr data model and netCDF have many things in common (groups, variables/arrays, attributes), there are also significant differences that need to be accounted for. Before delving into the technical details of defining one or more Zarr conventions for CF, a number of issues should be discussed and a consensus reached. The most fundamental of these, in my opinion, is the following:
The distinction between semantics and encoding is crucial here. As an example, in the current CF a coordinate variable is stored in a netCDF file as a variable; in the
csZarr convention above, regularly spaced coordinates are represented by an initial value and an increment in thecssub-schema. The encoding is different in both cases but the software reader will present both encodings as identical sequences of coordinates: the semantics are preserved.The first option, close alignment, is the approach in the
CFZarr convention linked to above. It has the benefit of being known to CF users and software developers. On the other hand, it perpetuates design decisions that may not be as relevant today as they were in the past. Some changes to the encoding will necessarily have to be made anyway because of the Zarr data model (no dimensions, for instance; no "NC_CHAR"!; compression is handled by Zarr).The second option, conserving CF semantics in an encoding that differs from the current encoding where necessary, as in the
csZarr convention above, means more up-front work for design and implementation and a new structure to be learned by CF users and software developers, but having the benefit of streamlining the encoding, better alignment with Zarr structures, possible alignment with standards like ISO/OGC, and more efficient storage.Many more arguments can be made for either position and I would like to invite the reader to contribute to the discussion here.
All reactions