Make the BSDF files of IGSDB available again

[christoph-maurer]

IGSDB was sharing the BSDF files of shading in the past. How difficult is it to share them again?

[danielmcquillen]

@christoph-maurer You can download a BSDF file from a product that has one on the IGSDB. For example:

https://igsdb-v2.herokuapp.com/products/accordia-classic-lf-toffee/

Go to the bottom of the page and look on the left. You should see:

[christoph-maurer]

Thank you, @danielmcquillen ! How difficult would it be to share the BSDF files via the GraphQL-API? The THERMX files would also be interesting if the required effort is very low.

[danielmcquillen]

@christoph-maurer I guess @simon-wacker would know best about the feasibility of incorporating into graphQL, but the URLs to those files are directly available if the user is logged in (as is the REST API itself).

You can see them in the above example I gave if you access the same product via the API:

https://igsdb-v2.herokuapp.com/api/v2/products/accordia-classic-lf-toffee/

My assumption is the graphQL database could return the same urls, just as it does a 'locator' for the product's API representation itself.

Any thoughts @simon-wacker ?

[simonwacker]

@danielmcquillen I just thought about how to serve those files. I suppose that THMX and BSDF are both optical data files. And that the field optical_data of optical_properties can be non-null even when those files are present. Is that correct?

1. If multiple product detail files are present are they in essence different representations of the same data?
2. If the optical_data field is non-null and the files are present, are the files in essence other representations of the same data?

If they represent essentially the same data, then the files could be returned as additional resources of the same optical data set, meaning that there would be only one OpticalData object and one UUID for this optical data set with multiple resources, in particular, a resource tree with more than one node, namely one node for each representation of the data. To construct this thing I'd need to fetch the product-detail-file records associated with the ProductMaterialized. I suppose I cannot do this with select_related because the product-detail-file records are related to Product and not to ProductMaterialized. Do you have another idea? (my knowledge of django is rather limited) For the meta information of the tree it is necessary to know how the data files were generated from the "raw" data in the field optical_data or how the data files relate to each other.

If they represent different data, then by the current design it becomes rather complicated because we would need one optical data UUID for each of the files and for the information of the optical_data field itself if present. I really hope this is not the case because in this case I would be rather lost in how to implement that properly.

[danielmcquillen]

@simon-wacker @christoph-maurer I might have to chat with @jcjonsson and @RDmitchell to fully answer you question, but I can provide some background on how the ProductDetailFile model is currently organised.

The ProductDetailFile is a model that stores one or more data-intensive documents for a product, where each document provides extra information about a product. It's generic in that it can store any kind of file, and the file_type field declares what kind of file it is. It's meant to grow over time as new document types are supported:

class ProductDetailFileType(Enum):
    GEN_BSDF_XML = "GEN_BSDF XML"
    BSDF_XML = "BSDF XML"
    THMX = "THMX"
    TOLERANCE = "Tolerance Report"
    OTHER = "Other"

So these formats might include optical data, or might include other things like structural information for a separate application (THMX for THERM).

In the traditional IGDB and Checkertool systems, products are submitted in a standardised "submission" text document with rows of measured spectral data. As you know, in the IGSDB that data gets stored in the OpticalData object. ProductDetailFiles come in when that format doesn't suffice, and more detailed formats are needed -- such as the BSDF format -- in Checkertool V2 and the IGSDB, those files are stored as ProductDetailFiles.

Therefore, I don't know if you can say that these files always "represent essentially the same information as the OpticalData" object. And since they might contain data not related to optical data, perhaps means that the ProductDetailFile table needs to be joined to the top-level Product table (although it could very well be I didn't model this well...please provide more feedback if you can).

You wrote

"For the meta information of the tree it is necessary to know how the data files were generated from the "raw" data in the field optical_data or how the data files relate to each other."

again the data files aren't generated from the raw data provided in a traditional submission file (rows of spectral data). I think they're generated via other processes. @jcjonsson would have to elaborate.

You wrote:

then by the current design it becomes rather complicated because we would need one optical data UUID for each of the files and for the information of the optical_data field itself if present.

... but since these files can represent different data types, some not optical data, is there another place in the GraphQL structure they could appear? We could add uuids to ProductDetailFile if that helps (including registering them with BED during Checkertool operations when they're first submitted).

Alternatively, if we qualified somehow which ProductDetailFileType are indeed optical data, or a subset of the data in the file is optical data (such as BSDF_XML and GEN_BSDF_XML), could we use a similar locator strategy to the one you and Christoph use in the current GraphQL strategy? ... i.e. include a list of urls in OpticalData object to relevant ProductDetailFiles, with each URL pointing to an endpoint in the REST API (which would just download the file)?

For example, we could add a method to the OpticalData model which produces a list of these urls if those optical data-specific files are present for the parent Product instance. You could then use that method when generating your GraphQl OpticalData instance (and maybe then don't need to worry about extra uuids?)

As I understand how these files are used, simulators would want the full file to use as part of an exercise, rather than need to review the data stored within. So in that case they'd probably be more useful as a direct download. @jcjonsson thoughts?

Apologies if I've misunderstood your question, or my ideas above don't really address the issue you described. Happy to keep discussing.

[christoph-maurer]

@danielmcquillen @simon-wacker Thank you for your thoughts!
@simon-wacker We can discuss it at our next in person meeting.
@danielmcquillen @jcjonsson Does the format of GEN_BSDF_XML and BSDF_XML differ or is the difference the method how they were created (maybe simulated vs. measured)?

[simonwacker]

@danielmcquillen Thank you for the detailed answer that helps a lot. I believe the best strategy to proceed is to add additional UUIDs to each ProductDetailFile. The reason why all those UUIDs are needed for one and the same IGSDB product, is that we designed the network in such a way that the metabase stores all meta information about products and that the product-data databases store actual simulation or measurement data that we categorized into optical, hygrothermal, calorimetric, and so forth. And the data is related to products via the product UUIDs managed by the metabase (to make it possible to fetch data of one and the same product from various product-data databases). And one entry in the list returned by allOpticalData is one data set and has its own UUID. Now, if there are various optical data sets for the same product in various formats, then each data set needs its own UUID (even though it refers to the same product via the componentId) and its own locator. In short, if I add UUIDs to ProductDetailFile, then it should work somewhat seamlessly (hopefully). Thanks for coming up with this simple solution.

[danielmcquillen]

@simon-wacker Ok sounds good. I will discuss with @RDmitchell and @jcjonsson the idea of adding uuid properties to the Product DetailFile model. As before, if we do this those IDs might change as we're in this strange period where Checkertool is still in development and we therefore periodically rebuild the IGSDB directly from the IGDB or CGDB.