[Proposed Feature] Analytics - Usage

[alukach]

Description of Feature

The data proxy is currently collecting analytics data via Worker Analytics Engine. We can surface this data to end-users.

For every request, we store:

• account_id
• product_id
• file_path
• method
• user_id (currently unused)
• country
• content_type
• bytes_sent
• status_code

Data is retained for 90 days.

Note

Analytics data can be sampled on both write & read, meaning that the values are close to but exactly match reality.

What value is this feature adding to Source Cooperative?

These statistics empower us to demonstrate access summaries on system, account, product, prefix, and file level.

We can display access data (e.g. download counts, byte volume) over time and/or client location.

This could be useful for data consumers to determine the popularity of a product against others. This could be useful for data providers to help understand how many people are using their products.

[alukach]

Some notes I wrote down while thinking about this:

Products

Note

Consider both product detail views and product list views.

• What's the latest/trend of count/volume of all files downloaded this past {time_interval}
• What were the most popular files this past {time_interval} by count/volume
• What's the download count/volume by country for this past {time_interval}
• How many times was this individual file downloaded this past {time_interval} (e.g. on file detail view)
• [FUTURE] Which users downloaded the most files from my product this past {time_interval} by count/volume

Account

• What were the most popular products from this account over this past {time_interval}
• What's the count / volume of files downloaded across all products this past {time_interval}
• What's the volume/trend of bytes downloaded across all products this past {time_interval}
• [FUTURE] What's the latest/trend of downloads by my account this past {time_interval} by count/volume

Admin

• Which accounts/products are distributing the most data?
• Which accounts/products are costing the most money?
• [FUTURE] Which users are downloading the most data?

Examples

NPMJS

https://www.npmjs.com/package/recharts

Simple Sparklines

Weekly Total + sparkline:

Past Weekly Totals when hovering sparkline:

Cloudflare

Metrics

Geo

[alukach]

output.mp4

We have a crude set of visuals deployed in https://source-cooperative-git-feat-analytics-radiantearth.vercel.app/fused/overture (#250)

[alukach]

Another thought: it's not unusual for us to see traffic patterns like this, where all is calm and then there's a huge spike in usage.

Source Coop systems admins should be able to understand what's causing this type of spike.

Perhaps breaking the bars up by org / product, location, ip address, etc.

[LanesGood]

@alukach @jedsundwall I've spent some time over the past weeks drafting a few ideas on the usage analytics for product pages. This is documented with some notes in Figma here:
https://www.figma.com/design/OR5NPTxiaqsRLX2gz0e9l6/Source-Coop?node-id=684-4268&t=xC3fC9dLqgelh2Ny-1

A few quick screens of the options I've explored:

Card	Analytics Tab	"Rings" card

[tylere]

Some Source data products have several datasets within the product, often within a specific folder/paths. Should we show analytics based on a folder/path, in addition to the overall analytics?

Some examples:

• Global Fields of The World (FTW) has:
  • raster prediction and confidence fields. vector field boundaries
  • the same data in multiple formats: cog vs. zarrs
• Overture Maps - Fused-partitioned has a folder for each release.
• Argentina National Census Data has folder for different years, and parquet and pmtiles formats

[jedsundwall]

I feel like we should be able to parse analytics based on any arbitrary prefix, but I want us to favor the {account_id}/{product_id} approach, so we should start with analytics at that level first.