feat: Add `now()` to troubleshoot pipeline latencies by abhishekrb19 · Pull Request #19386 · apache/druid

abhishekrb19 · 2026-04-28T16:22:56Z

Adds a now() expression macro to Druid's native expression. It returns the current system timestamp in milliseconds since epoch, evaluated per row at processing time. Intended primarily for ingestion-time troubleshooting - e.g.,
capturing now() - __time as an ingestion_lag_ms dimension to measure end-to-end pipeline delay. This expression is only allowed for non-__time columns.

Release note

Added now() expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., now() - __time). Note: now() is non-deterministic as it evaluates for every row, so it can break idempotency. This can only be added to columns that are not __time.

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
a release note entry in the PR description.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
been tested in a test Druid cluster.

aho135 · 2026-04-28T23:08:31Z

Fixes #18208

Co-authored-by: aho135 <andrewho135@gmail.com>

aho135 · 2026-04-28T23:18:23Z

@abhishekrb19 Curious if you have any thoughts on this old issue: #6513

Seems like the reservations were mainly for using now() for __time which you already mentioned should not be used. Maybe worth adding some validation/enforcement for this

aho135 · 2026-04-28T23:27:39Z

+`now()` is non-deterministic: replicated streaming tasks and task replays evaluate it to different
+values, producing inconsistent results across replicas. Do not use `now()` to overwrite `__time`.
+For Kafka, prefer [`kafka.timestamp`](../ingestion/data-formats.md#kafka) as the
+`__time` source.


I'm not sure if it makes sense to mention Kafka here. The timestamps aren't equivalent as one is the time the event lands on Kafka and the other is the time the event is ingested in Druid

aho135 · 2026-04-28T23:30:47Z

 |----|-----------|
+|now|now() returns the current system timestamp in milliseconds since epoch (1970-01-01 00:00:00 UTC). This function is evaluated for each row at processing time. It's recommended to use this only for troubleshooting issues - see [Using now() in ingestion](#using-now-in-ingestion).|
 |timestamp|timestamp(expr[,format-string]) parses string expr into date then returns milliseconds from java epoch. without 'format-string' it's regarded as ISO datetime format |
 |unix_timestamp|same with 'timestamp' function but returns seconds instead |


FrankChen021

Severity	Findings
P0	0
P1	0
P2	1
P3	0
Total	1

This is an automated review by Codex GPT-5

FrankChen021 · 2026-04-29T13:27:04Z

+      {
+        // Return analysis indicating this is NOT constant
+        // by pretending we have a free variable
+        return new BindingAnalysis();


[P2] Prevent now() from being planned as constant

Returning an empty BindingAnalysis makes ExpressionPlanner mark now() and expressions that only depend on now() as CONSTANT. Query-time expression selectors then build a ConstantExprEvalSelector and evaluate now() once when the selector is created, and expression filters/virtual columns also get stable cache keys, so native queries can reuse stale timestamps instead of evaluating at processing time. Add explicit non-deterministic handling, or otherwise keep now() out of constant selector/index/cache optimizations.

yes, good catch, this is fixed in the latest commit.

Thanks, this fixes the ExpressionPlan constant trait, but I think the selector path can still fold now() once. NowExpression.analyzeInputs() now reports only the synthetic __dummy__ binding; for normal segments that column has no capabilities and its selector is nil, so ExpressionSelectors.createBindings drops it and returns InputBindings.nilBindings(). makeExprEvalSelector then treats those nil bindings as a constant expression and returns new ConstantExprEvalSelector(plan.getExpression().eval(bindings)), so now() is still evaluated when the selector is created instead of at row/query processing time. The new tests cover the planner only; could you add selector-level coverage and avoid using a missing dummy input as the non-determinism marker?

Ack, updated the code with your non-determinism suggestion

Thanks, the selector constant-folding path I called out is fixed now: now() carries a non-deterministic analysis flag and ExpressionSelectors skips ConstantExprEvalSelector for it. I think two related constant/cache paths still need the same treatment, though. Expr.asBitmapColumnIndex still treats any expression with no required bindings as a constant bitmap index, so an expression filter around now() can still be evaluated once while building the bitmap. Also, ExpressionDimFilter and ExpressionVirtualColumn cache keys still serialize the stable expression cache key (now() stringifies the same way every time), so cached query results can be reused across wall-clock changes. Could you either make non-deterministic expressions non-cacheable for these query components, or include a non-stable cache key / disable the affected optimizations when analysis.isNonDeterministic() is true?

…ns. Fix constant handling.

abhishekrb19 · 2026-05-01T02:25:15Z

@abhishekrb19 Curious if you have any thoughts on this old issue: #6513
Seems like the reservations were mainly for using now() for __time which you already mentioned should not be used. Maybe worth adding some validation/enforcement for this

@aho135 @FrankChen021 - thanks for the reviews and for pointing me to the previous issues. I’ve added validation to restrict now() usage for non-__time columns and to throw an error when it’s used with the __time column.

FrankChen021

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.

Reviewed 17 of 17 changed files.

This is an automated review by Codex GPT-5.5

aho135 · 2026-05-12T20:38:02Z

+    // Verify the same instability propagates when now() is nested in a parent expression.
+    Expr nested = Parser.parse("now() > 0", ExprMacroTable.nil());
+    Assertions.assertFalse(
+        java.util.Arrays.equals(nested.getCacheKey(), nested.getCacheKey()),


Suggested change

java.util.Arrays.equals(nested.getCacheKey(), nested.getCacheKey()),

Arrays.equals(nested.getCacheKey(), nested.getCacheKey()),

Can be imported

aho135 · 2026-05-12T21:33:27Z

    )::get;
+
+    if (ColumnHolder.TIME_COLUMN_NAME.equals(name)
+        && BuiltInExprMacros.NowExprMacro.containsNow(parsedExpression.get())) {


Would it make sense to just check isNonDeterministic() instead of containsNow?

Yeah, good point, updated. I think we can make this be the case for all non-deterministic expressions (now() being the only one currently)

aho135 · 2026-05-12T21:38:09Z

+  /**
+   * now() depends on wall-clock time, so its cache key must not be stable across calls — otherwise queries containing
+   * now() can hit a stale cached result. NowExpression overrides {@link Expr#decorateCacheKeyBuilder} to mix in
+   * nanoTime, which makes any containing Expr produce a fresh key on every {@link Expr#getCacheKey()} call.


Stale comment, cacheKey is monotonically increasing long

Removed this test doc altogether

aho135 · 2026-05-12T21:41:32Z

  private Supplier<byte[]> makeCacheKeySupplier()
  {
-    return Suppliers.memoize(() -> {
+    return () -> {


Should memoize be conditional depending on whether the expression isNonDeterministic?

It shouldn't be needed bc non-deterministic expressions like NowExpr.decorateCacheKeyBuilder() appends an incrementing nonce, which makes the cache key unique every time it's built.

I see that this was reverted in the latest commit. Was that intentional?

That's right, it was reverted to the version in master to align with decorateCacheKeyBuilder()

aho135

LGTM, thanks @abhishekrb19!

abhishekrb19 added 2 commits April 28, 2026 08:58

now() expression in ingestion to help troublehsoot

3ab8079

Dcoument

7e691f3

github-actions Bot added the Area - Documentation label Apr 28, 2026

abhishekrb19 added 3 commits April 28, 2026 09:39

fix CI check & cleanup

69083f1

doc link fix

8557ca9

equals() and hashcode() for test

8fa6a6d

aho135 reviewed Apr 28, 2026

View reviewed changes

Comment thread docs/querying/math-expr.md Outdated

Update docs/querying/math-expr.md

4297baa

Co-authored-by: aho135 <andrewho135@gmail.com>

aho135 reviewed Apr 28, 2026

View reviewed changes

FrankChen021 reviewed Apr 29, 2026

View reviewed changes

Add validation to restrict __time; only allow this for non-time colum…

7a708dd

…ns. Fix constant handling.

github-actions Bot added the Area - Segment Format and Ser/De label May 1, 2026

abhishekrb19 added 4 commits May 4, 2026 10:47

Non-deterministic expr binding

e890cf2

review adjustments for query-time

b31fbcb

Merge branch 'master' into now_func

f167c7e

Simplify & adjust

7dfe4ff

abhishekrb19 closed this May 12, 2026

abhishekrb19 reopened this May 12, 2026

abhishekrb19 added 2 commits May 12, 2026 00:14

Adjust

d33e268

makeCacheKeySupplier() adjustment

2211384

abhishekrb19 force-pushed the now_func branch from 82f9a69 to 2211384 Compare May 12, 2026 07:27

FrankChen021 reviewed May 12, 2026

View reviewed changes

Adjust cache key

2d3fa33

aho135 reviewed May 12, 2026

View reviewed changes

review: fix import

5d836fa

aho135 reviewed May 12, 2026

View reviewed changes

repurpose nonDeterminstic()

db640d0

aho135 approved these changes May 13, 2026

View reviewed changes

abhishekrb19 merged commit 3ba9977 into apache:master May 13, 2026
38 checks passed

abhishekrb19 deleted the now_func branch May 13, 2026 07:31

github-actions Bot added this to the 38.0.0 milestone May 13, 2026

	\|unix_timestamp\|same with 'timestamp' function but returns seconds instead \|
	\|unix_timestamp\|unix_timestamp(expr[,format-string]) same with 'timestamp' function but returns seconds instead \|

	java.util.Arrays.equals(nested.getCacheKey(), nested.getCacheKey()),
	Arrays.equals(nested.getCacheKey(), nested.getCacheKey()),

Conversation

abhishekrb19 commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

aho135 commented Apr 28, 2026

Uh oh!

aho135 commented Apr 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhishekrb19 commented May 1, 2026

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aho135 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

abhishekrb19 commented Apr 28, 2026 •

edited

Loading