[Proposed Feature] Analytics - Storage

[alukach]

Description of Feature:

Data providers should be able to see the volume of data they have stored both within and across products within Source Cooperative. Ideally, they should be able to see a trend of how this volume data changed.

What value is this feature adding to Source Cooperative?

This seems like it would be generally useful information, both for product owners and as system administrators. Particularly if we need to charge accounts for storage.

[jedsundwall]

Yes 100%. I'd like to get Anthony thinking of how to present this information on a product detail page. We have a somewhat spare "Product Details" box where this kind of thing should go, but it needs to be more dense.

I've also been thinking that we could do a better job of summarizing product inventories. E.g. we should be able to say that a product contains X number of objects, A of which are CSVs, B of which are JSON, etc… (I believe Tyler has asked for this too!). This is potentially really complex because I'm not sure how to describe a Zarr store in this way and it might be expensive to count these things up properly. The benefit here would be to help people find products that contain file types that they're familiar with. As I write this, I realize this is a separate issue, so let's start with byte volume and see if we can expand.

[alukach]

I'd like to get Anthony thinking of how to present this information on a product detail page. We have a somewhat spare "Product Details" box where this kind of thing should go, but it needs to be more dense.

I'm guessing you mean @LanesGood to think about this? Anyways, I suspect that it would be valuable to think about this alongside of #257.

[jedsundwall]

lol yes I meant @LanesGood !

[LanesGood]

Do organizations currently have a private provider view?

In terms of both org-wide and product-centric analytics on both storage and usage, another good comparison could be the "Insights" tab on Github, with easily accessible charts and figures. What seems useful here is also the adjustable time period, and from a dev perspective it seems like a lot of this is calculated on the fly, rather than needing to store this information in a database (at least that's my assumption based on the fact that queries seem to need to be first run when you visit the page).

Does Amazon also provide this info for S3 in an easily digestible way? Not looking to replicate or provide feature parity, but could be interesting to see what is common/expected in the space and what is already visible for users who may be using other cloud storage providers.

[alukach]

Do organizations currently have a private provider view?

Not exactly. It's more akin to the style of Github, where anyone can view github.com/source-cooperative but if you're authenticated and authorized, you'll see things not intended for the public such as private repos, buttons to access settings, etc.

However, I do think it's reasonable to assume that someday, the homepage of source.coop will be more of a dashboard "see all the things" view when a user is authenticated. However, that's probably scope creep for this exercise.

In terms of both org-wide and product-centric analytics on both storage and usage, another good comparison could be the "Insights" tab on Github, with easily accessible charts and figures.

I think this is a very good suggestion.

What seems useful here is also the adjustable time period, and from a dev perspective it seems like a lot of this is calculated on the fly, rather than needing to store this information in a database

The Workers Analytics Engine seems to be very capable of querying time spans on the fly. However, I think that's probably less important for this Storage Analytics feature than it is for the Usage Analytics feature (#257).

Does Amazon also provide this info for S3 in an easily digestible way? Not looking to replicate or provide feature parity, but could be interesting to see what is common/expected in the space and what is already visible for users who may be using other cloud storage providers.

Yes, within an S3 Bucket, you can click on the "Metrics" tab:

However, you'll notice that this is only for the entirety of the bucket. It's not very easy to understand which prefixes (equivalent to directories) are holding the most data. For that, S3 can dump regular S3 Inventory Reports that contain the filesize of every object within a bucket. A user could load those regularly delivered reports into something like AWS Athena to use SQL to query over the data to answer questions like "which prefixes are holding the most data" or get more nuanced trends for data. It will be important for us to agree upon where we should stop for fine-grained tracking of storage and to consider the costs/complexity associated with such tracking (not saying it actually is an issue to track the trend of file sizes down to every prefix / file, however we just need to be able to realistically calculate how expensive/difficult that will be)