[INF] Merlin Commons

[karlhigley]

Problem:

Merlin now has a bunch of libraries that need to interoperate smoothly, but a general lack of shared abstractions, conventions, and standards that would make that possible.

Goal:

• Build a solid foundation for the Merlin libraries via improvements in Core

New Functionality

• Core:
  • Shape in column schemas (for consistent tracking across libraries)
  • Cross-framework dtype translation (e.g. via Merlin dtypes)
  • Cross-framework data transfer via zero-copy protocols (for Columns and DictArrays -> Series and Dataframes)
  • Bespoke Merlin schema file format (i.e. a Protobuf schema for Merlin schema that isn't from Tensorflow Metadata)
• Corresponding updates in all downstream libraries

Constraints:

• All functionality entailed by this issue has to work in and be adoptable by all Merlin libraries

Starting Point:

• Proposals:
  • Shapes - https://docs.google.com/document/d/13WrHuCxspTDjv2NPQxWtzZqBoI75Np5_tOopJEWQbBg/edit#
  • Dtypes
  • Data transfer
  • Schema file format
  • [ARCH] Fixed vs Ragged #791
  • Overarching decisions to be made (Merlin commons ) https://docs.google.com/document/d/1IGCS5-f6-SJkpfHFChE4L_iaNRXVjrx0zW-NPDOyfkU/edit
    • [INF] Merlin commons - Shapes #813
    • [INF] Merlin commons - Operators #816
    • [INF] Merlin commons - Schemas #814
    • [INF] Merlin commons - General #815

[karlhigley]

These decisions affect WIP in the following roadmap issues:

• [RMP] Provide PyTorch serving support for T4R models in Torchscript #255
• [RMP] Tensorflow support for session based recommendations integration in Merlin #433
• [RMP] Support pre-trained vector embeddings as input features into a model via the dataloader #211
• [RMP] Support Offline Batch processing of Recs Generation Pipelines #419
• [RMP] GTC Session-based Web Recommendations #700

(in addition to future work)

[viswa-nvidia]

@karlhigley , please provide the PRs that close the Dtypes for tracking. Thanks

[viswa-nvidia]

Un checked items are not blocking