Repository navigation
Allow a list of IDSs and occurrences in imas convert --ids, and deprecate --occurrence - #152
Conversation
imas convert could convert either every IDS in a Data Entry or a single one. --ids now also takes a comma-separated list, for example --ids core_profiles,equilibrium,summary; --occurrence applies to each IDS in the list. The names are checked against the target DD before the output Data Entry is created, so a misspelled name fails at the start instead of partway through a conversion. Closes iterorganization#144
--ids now takes IDS or IDS:OCCURRENCE entries, for example --ids summary,plasma_profiles:1,plasma_profiles:2, so several occurrences of an IDS can be converted together. An IDS given without an occurrence still converts all of its occurrences. --occurrence only made sense with a single IDS. It still applies to IDSs given without an occurrence, but logs a deprecation warning pointing to the new syntax, so it can be removed at the next major release. A malformed occurrence (core_profiles:x) is rejected before the output Data Entry is created.
maarten-ic
left a comment
There was a problem hiding this comment.
Some comments and suggestions to discuss:
| If not provided, all IDSs in the data entry are converted.", | ||
| help="Specify which IDSs to convert, as a comma-separated list of IDS or " | ||
| "IDS:OCCURRENCE, for example summary,plasma_profiles:1,plasma_profiles:2. " | ||
| "An IDS without an occurrence converts all its occurrences. " |
There was a problem hiding this comment.
In the IDS Path syntax, omitting the occurrence means you want the default occurrence (0), not all occurrences.
Maybe we'd need an explicit syntax for all occurrences, for example <ids_name>:*?
There was a problem hiding this comment.
Agreed, done: as in the IDS path syntax, an IDS without an occurrence now means occurrence 0, and IDS:* selects all occurrences, for example --ids summary,plasma_profiles:*. Leaving --ids out still converts every IDS and occurrence, as before.
One behaviour change to be aware of: on develop, --ids core_profiles converts all occurrences of core_profiles; with this PR it converts occurrence 0 only, and core_profiles:* gives the old behaviour. If you'd rather keep the old meaning for a bare name, it's a one-line change.
As in the IDS path syntax, an IDS given in --ids without an occurrence now means occurrence 0; IDS:* selects all occurrences. Leaving --ids out still converts every IDS and occurrence. An IDS or occurrence that is not in the input Data Entry is now reported before the output Data Entry is created, instead of failing partway through or converting nothing. The CLI docs no longer mention the deprecated --occurrence option.
|
|
||
| def _parse_ids_option( | ||
| ids: str, known_ids: list[str], dd_version: str | ||
| ) -> list[tuple[str, int | None]]: |
There was a problem hiding this comment.
I think this will fail on Python 3.9 and older unless we add from __future__ import annotations.
In pyproject.toml we indicate versions >=3.7 although we only test from 3.10 as part of our CI. Personally, I think we should still support 3.9 which is the standard version on RHEL 9 systems (although even 3.10 is now EOL).
There was a problem hiding this comment.
Good catch, done in ec31eb1: cli.py now has from __future__ import annotations, so the new hints are not evaluated at import time, and there is no other 3.10-only syntax in cli.py. I could not run the tests on 3.9 itself: imas-core 5.7.2 publishes wheels for cp310 to cp314 only, so pip cannot install it there. On 3.12, test_cli.py (15) and the full suite (1808 passed, 297 skipped) pass: https://www.kaggle.com/code/sankurkundu/notebook75525243ea
Add from __future__ import annotations so that the new type hints (int | str | None) are not evaluated at import time, which fails on Python 3.9 and older. Use str.isdecimal() rather than str.isdigit() for the occurrence, so that superscript digits such as "2" in "core_profiles:²" are rejected with a usage error instead of crashing int(). Describe convert as converting selected IDSs/occurrences.
…occurrence deprecation
Closes #144
As discussed in #144,
imas convert --idsnow takes a comma-separated list of IDS, IDS:OCCURRENCE or IDS:* (all occurrences):IDS:*selects all its occurrences. Leaving--idsout still converts every IDS and occurrence.--occurrenceis deprecated, as @maarten-ic suggested: it still applies to IDSs given without an occurrence, but logs a warning pointing to--ids IDS:OCCURRENCE, so it can be removed at the next major release.unknown IDS name(s) for DD version 4.1.0: core_profile;core_profiles:xorcore_profiles:²;not found in the input Data Entry: core_profiles:7, wall:0.On develop, a name that is not an IDS (a typo such as
--ids core_profile, or a list, which develop reads as one name) converts nothing and still exits with status 0.Behaviour change: on develop,
--ids core_profilesconverts all occurrences of core_profiles; with this PR it converts occurrence 0 only, andcore_profiles:*gives the old behaviour.cli.pyusesfrom __future__ import annotations, so the new type hints are not evaluated at import time on Python 3.9.Tests in
imas/test/test_cli.py:test_imas_convert_ids_list: only the listed IDSs are converted;test_imas_convert_unknown_ids: a misspelled name exits with status 2 and creates no output;test_imas_convert_ids_with_occurrences: a bare name converts occurrence 0,IDS:Noccurrence N, andIDS:*all of them;test_imas_convert_occurrence_option_is_deprecated:--occurrencelogs the warning and still applies to IDSs without an occurrence;test_imas_convert_invalid_occurrence::x,:-1,:²and an empty occurrence exit with status 2 and create no output;test_imas_convert_ids_missing_from_input: a missing occurrence, a missing IDS andIDS:*with nothing in the input exit with status 2 and create no output.All of
test_cli.py: 15 passed. Full test suite (pytest -n=auto, as in the CI): 1808 passed, 297 skipped. black (24.x, as inlinting.yml) and flake8 are clean. I could not run the tests on Python 3.9: imas-core 5.7.2 publishes wheels for cp310 to cp314 only.The CLI docs and
--helpdescribe the new form. I have not touched the changelog, since it is updated at release time; I'm happy to add an entry if you prefer.Run log: https://www.kaggle.com/code/sankurkundu/notebook75525243ea