Skip to content

add inference logging reader client #365

Merged
azmat-meesho merged 22 commits intomainfrom
develop
Apr 6, 2026
Merged

add inference logging reader client #365
azmat-meesho merged 22 commits intomainfrom
develop

Conversation

@azmat-meesho
Copy link
Copy Markdown
Contributor

@azmat-meesho azmat-meesho commented Apr 6, 2026

🔁 Pull Request Template – BharatMLStack

Please fill out the following sections to help us review your changes efficiently.

Context:

all the details are here: https://meesho.atlassian.net/wiki/spaces/EW/pages/4993941516/Inference+Logging+Client+User+Guide+for+Decoding+MPLog+Data+into+Spark+DataFrames

Describe your changes:

added sdk

Testing:

manually on notebooks

Monitoring:

Explain how this change will be tracked after deployment. Indicate whether current dashboards, alerts, and logs are enough, or if additional instrumentation is required.

Rollback plan

Explain rollback plan in case of issues.

Checklist before requesting a review

  • I have reviewed my own changes?
  • Relevant or critical functionality is covered by tests?
  • Monitoring needs have been evaluated?
  • Any necessary documentation updates have been considered?

📂 Modules Affected

  • horizon (Real-time systems / networking)
  • online-feature-store (Feature serving infra)
  • trufflebox-ui (Admin panel / UI)
  • infra (Docker, CI/CD, GCP/AWS setup)
  • docs (Documentation updates)
  • Other: ___________

✅ Type of Change

  • Feature addition
  • Bug fix
  • Infra / build system change
  • Performance improvement
  • Refactor
  • Documentation
  • Other: ___________

📊 Benchmark / Metrics (if applicable)

Summary by CodeRabbit

Release Notes

  • New Features

    • Added SlateComponent configuration toggle for rankers and re-rankers in InferFlow
    • Inference logging client now supports selective column decoding via needed_columns parameter
  • Documentation

    • Expanded SDK documentation with comprehensive API reference, configuration guide, and examples
  • Chores

    • Updated inference logging client to version 0.3.1; replaced pandas dependency with Spark support

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Apr 6, 2026

Caution

Review failed

Pull request was closed or merged during review

Walkthrough

This PR adds a post-commit hook, introduces activity contract types for Go, propagates a SlateComponent boolean field across Horizon's inferflow models and handlers, refactors the Python inference logging client SDK from pandas-based to Spark-distributed processing, and updates the Trufflebox UI to support the new field with lowercase service name references.

Changes

Cohort / File(s) Summary
Pre-commit Configuration
.pre-commit-config.yaml
Added a new local post-commit hook (post-commit-scripts/runner.sh) configured to run with always_run: true.
Core Activity Contracts
core/activity-contracts/activity/activity.go, core/activity-contracts/go.mod
Introduced new Go package defining shared activity contract types: GlobalActivityOutputs, ActivityInput, ActivityOutput, and ActivityFunc function signature; added corresponding module definition.
Horizon InferFlow Models
horizon/internal/inferflow/etcd/models.go, horizon/internal/repositories/sql/inferflow/models.go, horizon/internal/inferflow/handler/models.go, horizon/pkg/configschemaclient/types.go
Added SlateComponent bool field (JSON: slate_component) to component and ranker structs (NumerixComponent, PredatorComponent, OnboardRanker, OnboardReRanker, Ranker, ReRanker) across all model layers.
Horizon InferFlow Handlers
horizon/internal/inferflow/handler/adaptor.go, horizon/internal/inferflow/handler/config_builder.go
Implemented SlateComponent field mapping in adaptors for DB↔domain and DB↔ETCD transformations; propagated field in component builders for Predator and Numerix components.
Python SDK Core Logic
py-sdk/inference_logging_client/inference_logging_client/__init__.py, py-sdk/inference_logging_client/inference_logging_client/cli.py, py-sdk/inference_logging_client/inference_logging_client/formats.py, py-sdk/inference_logging_client/inference_logging_client/utils.py
Refactored from pandas to Spark DataFrames: decode_mplog() and decode_mplog_dataframe() now accept SparkSession and return Spark DataFrames; added needed_columns parameter for selective feature decoding; implemented distributed decoding via mapInPandas() and Spark-based float formatting; updated CLI to initialize SparkSession and write outputs using Spark APIs.
Python SDK Utilities & Dependencies
py-sdk/inference_logging_client/inference_logging_client/decoder.py, py-sdk/inference_logging_client/inference_logging_client/io.py, py-sdk/inference_logging_client/inference_logging_client/types.py, py-sdk/inference_logging_client/inference_logging_client/exceptions.py, py-sdk/inference_logging_client/pyproject.toml, py-sdk/inference_logging_client/readme.md
Replaced pandas>=1.3.0 dependency with pyspark==3.3.0; bumped version from 0.1.0 to 0.3.1; applied code formatting standardizations; expanded README with comprehensive API reference, examples, and Spark-focused usage guidance.
Trufflebox UI InferFlow Config
trufflebox-ui/src/pages/InferFlow/DiscoveryRegistry/InferflowRegistry/CloneInferflowConfigModal.jsx, trufflebox-ui/src/pages/InferFlow/DiscoveryRegistry/InferflowRegistry/EditInferflowConfigModal.jsx, trufflebox-ui/src/pages/InferFlow/DiscoveryRegistry/InferflowRegistry/InferflowConfigForm.jsx, trufflebox-ui/src/pages/InferFlow/DiscoveryRegistry/InferflowRegistry/OnboardInferflowConfigModal.jsx
Added slate_component boolean toggle UI for rankers and re-rankers; changed service discovery API parameter from service_name=InferFlow to service_name=inferflow; updated validation/error messages to use lowercase naming.
🚥 Pre-merge checks | ✅ 1
✅ Passed checks (1 passed)
Check name Status Explanation
Dynamic Configuration Validation ✅ Passed The pull request contains no changes to files matching the application-dyn-*.yml pattern, and no such files exist in the codebase.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@azmat-meesho azmat-meesho merged commit 8ac9e31 into main Apr 6, 2026
57 of 58 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants