Skip to content

Feature/when block in redis lookup#398

Merged
spicy-sauce merged 19 commits into
datayoga-io:mainfrom
ZdravkoDonev-redis:feature/when-block-in-redis-lookup
Dec 28, 2025
Merged

Feature/when block in redis lookup#398
spicy-sauce merged 19 commits into
datayoga-io:mainfrom
ZdravkoDonev-redis:feature/when-block-in-redis-lookup

Conversation

@ZdravkoDonev-redis

Copy link
Copy Markdown
Contributor

In order to reduce latency, we can avoid doing lookups altogether by looking in some of the fields in the input data and deciding whether we should doo a lookup in advance. Thus with this commit, a 'when' field is introduced to support this functionality

In order to reduce latency, we can avoid doing lookups altogether by looking in some of the fields in the input data and deciding whether we should doo a lookup in advance. Thus with this commit, a 'when' field is introduced to support this functionality
@spicy-sauce spicy-sauce self-requested a review December 28, 2025 11:38
@spicy-sauce spicy-sauce merged commit e471e8f into datayoga-io:main Dec 28, 2025
5 checks passed
@spicy-sauce

Copy link
Copy Markdown
Member

Thanks a lot, @ZdravkoDonev-redis! This PR raised a few issues as this was the first one done on a forked branch.

The feature itself looks good - the when condition for redis.lookup is well implemented with:

  • Clean schema definition supporting both jmespath and sql expression languages
  • Proper error handling (lookup is skipped if condition evaluation fails)
  • Efficient pipeline usage - only records passing the condition trigger Redis calls
  • Comprehensive test coverage for both languages and edge cases

Infrastructure fixes included in this PR:

Since this was our first fork PR, we discovered several CI/workflow issues:

  1. Checkout for forks - Workflows couldn't fetch the branch from a forked repo. Fixed by adding the repository parameter to actions/checkout.
  2. Auto-push for fork PRs - GitHub doesn't expose secrets to fork PR workflows, so the auto-commit of generated schemas/docs fails. We now detect fork PRs and display a helpful error message with the exact command to run locally.
  3. Non-deterministic schema generation - The generated JSON schemas differed between local and CI runs due to Python's hash randomization and unordered file iteration. Fixed by sorting block types and using sort_keys=True in JSON output.
  4. Shared generation scripts - Created scripts/generate-jsonschemas.sh and scripts/generate-docs.sh to ensure local and CI generation produce identical results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

block enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants