ElasticSearch Unified Index

[learnitall]

Tracking creating a unified template for Elasticsearch indices, with proper datatypes and defaults for each of our fields. Essentially the question that we want to answer here is: how do we transform a BenchmarkResult dataclass into JSON?

[learnitall]

I think that this discussion becomes even more important with #298 and #296. Right now we are exporting a flat-JSON document, which is fine for now, but should we think about revisiting this since we are adding more fields?

Thoughts?
@whitleykeith
@jtaleric
@dry923

[whitleykeith]

Elasticsearch is built on top of Lucene, which stores data in an inverted index. Simply put, Lucene maps terms -> documents instead of documents -> terms. Lucene doesn't really support nesting as it only supports numeric, binary, and text fields. ES does a lot of optimizations on top of Lucene to support nesting but I'm not sure about the index/search performance of them as I've only ever used Solr. But in general I would default to not nesting given the base technology ES uses (to a point).

ES has some guidelines on doc structure which is generally pretty solid. I think we should start here and look at defining the common core fields for all benchmarks and then specific core fields for each benchmark.

I think of the common core fields as things like uuid, run_id, start_time, end_time, .... Things that every benchmark should have.

I think every common core field should be left at the document root with no nesting. Every doc should have these fields defined and we should reject docs that don't. After that, we should look at how we search the data and fields that are heavily used in queries should be moved to the root of the doc. For instance I think all the environment information of a run (i.e. cluster_name, platform, etc.) should be as close to the root of the doc as possible to make searches easier and more performant.

The counter to that is when fields may or may not exist doc to doc, or are specific to the environment/benchmark. For instance not every snafu run may be running in k8s, so cluster_name, etc. may not be collected. For those we don't necessarily want them to be flat because we want a uniform root doc structure for better readability and also indexing.

Given that I think we can afford to nest somewhat. I think a good starting schema would look something like this:

{
    "uuid": "str",
    "run_id": "str",
    "start_time": "datetime",
    "end_time": "datetime",
    "duration": "Number",
    "type": "string",
    "iteration": "Number",
    "kubernetes": {
        "cluster_version": "str"
    },
    "openshift": {
        "cluster_version": "str",
        "cluster_name": "str",
        "platform": "str",
        "network_type": "str"
    },
    "config": {
        "foo": "bar"
    },
    "data": {
        "foo": "bar"
    }

}

Where config and data would have benchmark-specific schemas. That way we nest fields that aren't always there while still not going too crazy and make it hard to query. Nesting config and data is fine because we don't really search on that, but rather retrieve those results from a query

[learnitall]

I like that structure, and I think you have a really good point about modeling after ECS. I think it might be worth doing something like this:

{
    "uuid": "str",
    "run_id": "str",
    "start_time": "datetime",
    "end_time": "datetime",
    "duration": "Number",
    "type": "string",
    "iteration": "Number",
    "environment": "flattened",
    "config": "flattened",
    "data": "flattened
}

If we create an environment key in the root and take advantage of the flattened data type then we don't have to worry so much about the structure of the document in order to have successful searches. At least, that's my understanding but I could be wrong.