Add NX docstring as attribute

[lukaspie]

@rettigl this is a way to add the NX docstrings to the HDF5 files as attributes. The idea is that you can pass the write-docs flag to the dataconverter and then docstrings are added. By default, this is turned off so as to not change any existing workflows. We can discuss if we want this to happen by default.

Here's an example using the xps reader:
output.nxs.zip

The implementation is relatively trivial. There are however two open questions for me

• How to handle inherited docs
  Usually, our appdefs have very sparse documentation since all the docstrings are in the base classes or in appdefs that are further up the chain. My understanding is that the docs are also extended (not overwritten) when you inherit from a class. So technically, one should concatenate the docs of all inherited nodes. However, this gets quite unwieldy and messy as the docs in the files get rather large and sometimes they might even contradict each other. My suggestion (implemented here) was that as we travel up the inherited nodes, as soon as there is a docstring we only use that one (and don't add any docs coming from even earlier nodes). But this is probably not strictly correct. Maybe @sanbrock can comment.

• Docs for NeXus attributes
  NeXus attributes are written as HDF5 attributes already. Since HDF5 attributes cannot have attributes themselves, the question is where to place the docs for these attributes? My solution here: write another attribute <attribute>__docs (e.g. entry/definition/version__docs) to the HDF5 file.

[lukaspie]

reminder to remove before merging

[sherjeelshabih]

• Docs for NeXus attributes
  NeXus attributes are written as HDF5 attributes already. Since HDF5 attributes cannot have attributes themselves, the question is where to place the docs for these attributes? My solution here: write another attribute <attribute>__docs (e.g. entry/definition/version__docs) to the HDF5 file.

Any practical solution works in my opinion. The only issue will be how to report/add this in the NXDL structure. It's a bit meta over the already meta attributes we have in the NXDL. Will something like adding this renameable field, FIELDNAME__docs, in NXobject work out nicely in the NXDL framework?

[lukaspie]

• Docs for NeXus attributes
  NeXus attributes are written as HDF5 attributes already. Since HDF5 attributes cannot have attributes themselves, the question is where to place the docs for these attributes? My solution here: write another attribute <attribute>__docs (e.g. entry/definition/version__docs) to the HDF5 file.

Any practical solution works in my opinion. The only issue will be how to report/add this in the NXDL structure. It's a bit meta over the already meta attributes we have in the NXDL. Will something like adding this renameable field, FIELDNAME__docs, in NXobject work out nicely in the NXDL framework?

I didn't go this far. For groups and fields, I basically added an attribute @docs, whereas for the attributes, I used an additional attribute <attribute>__docs. I guess the problem would be that all of these are undocumented..

[sherjeelshabih]

Ah alright. So it's just for the attributes that you add a suffix.

You're right. They will remain undocumented. Let's say to see how it goes in use for us we can leave it undocumented.

It will make it practically easier to understand Nexus files like this. It makes the Nexus files more self sufficient too. And it seems this is the best we can do without overcomplicating it.

[lukaspie]

Notes from TF meeting:

• Use single underscore for attributes, like for example data and data_units
• Can we render the docs a bit nicer? HTML syntax?
• Don't write the documentation to every single instance of the same concept, use internal / softlink
• External links to documentation? docs and docs_url
• @sanbrock: HDF5 contains data, not information. What happens when we want to reuse the data somewhere else? We could prevent this by writing a parallel structure with the docs and linking to that place.
  • option 1: add a warning if links are used and the docstrings are not actually correct?
  • option 2 (by @rettigl): if we write the docs, we are not using links, but rather we copy the data directly

[lukaspie]

@mkuehbach @rettigl I made some changes here, following the changes to the NexusNode object. In principle, I believe this would be ready. The bigger question is if we want to have such a feature. I think since it is optional and only turned on by passing a flag to pynx convert, it is relatively harmless and a nice feature to add. But happy to hear your opinions.

What changed (since rebase onto latest master)

• writer.py: --write-docs embeds NeXus concept docstrings as HDF5 string attributes (@docs, @docs_url, @<attr>_docs, @<attr>_docs_url) using NexusNode.get_link() for the URL and NexusNode.get_inheritance_concept_paths() for inheritance-aware doc text.
• cli.py / convert.py: --write-docs flag and --docs-format option (default = raw RST, plain = markup-stripped single-line, optimised for h5web).
• annotator.py: pynx read silently skips @docs/@docs_url/@*_docs/@*_docs_url attributes to keep output uncluttered.
• NXtest.nxdl.xml: added <doc> to the version attribute for test coverage.
• Tests: test_write_docs parametrized over both formats; plain asserts no embedded \n.
• Docs: new section in dataconverter-and-readers.md with usage, format table, and a !!! warning on link semantics (write-time and read-time). -> this addresses the tension raised by Sandor in Add NX docstring as attribute #415 (comment).

Not yet addressed / open design questions

1. Repeated concepts — every instance of the same concept gets its own copy of the docs string. The suggestion was to write docs once and softlink; not implemented (architecturally non-trivial, requires a parallel /.nexus_docs/ structure or concept-level deduplication).

2. Link/copy tension — --write-docs intentionally skips HDF5 link nodes (soft link, virtual dataset, external link) to avoid attaching the wrong concept's docs. The alternative (@rettigl: copy data instead of linking when --write-docs is active) was discussed but not implemented. Currently only documented with a warning.

3. HTML rendering — dropped: NXDL docs use Sphinx RST (:ref:, .. include::) that docutils cannot resolve standalone. Only default and plain are offered. plain is however good enough that in an h5web viewer one can read the docs sufficiently well: