Proposal: Asynchronous Listener Registration

[AndreaOrru]

Currently, the WebExtensions platform requires event listeners to be registered synchronously during a service worker's initial script evaluation. This makes e.g. conditional listener registration based on asynchronous state (like browser.storage) brittle, forcing developers to rely on complex workarounds.

This proposal introduces an "async_initialization": true manifest opt-in and a new browser.runtime.markInitializationComplete() API to address this issue.

[Rob--W]

This proposal looks useful to extensions, and feasibly implementable by browsers.

Right now it is focused on adjusting event registration behavior at initialization.
Would you be willing to consider expanding to the post-initialization phase? With a method to control listener registration behavior, we can reduce the number of unnecessary wake-ups (#719) and writes to disk (frequency and size). E.g.:

browser.runtime.markBackgroundEventStage({
  // Commit listeners and flush (as proposed)
  listenersReady: true, // can only be true once

  // Whether to persist across browser/extension restarts
  persistAcrossSessions: true, // default true

  // Whether to persist past worker termination.
  persistAcrossUnload: true, // default true
});

try {
  browser.runtime.markBackgroundEventStage({
    persistAcrossUnload: false,
  });
  browser.webNavigation.onBeforeNavigate.addListener(...);
} finally {
  browser.runtime.markBackgroundEventStage({
    persistAcrossUnload: false,
  });
}

This is just an example, an enum could also apply here. The method would update a renderer-side flag specific to the extension worker, and whenever addListener is called, the flag would be passed along to help the browser with making the decision on whether to persist, and to what extent.

[Rob--W]

Potential future work: dispatch runtime.onPerformanceWarning event when events have been dropped. This would enable extensions to detect bugs. This event was already described in #456 and implemented in Firefox, documented at https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/runtime/onPerformanceWarning

[AndreaOrru]

What do you mean by "events have been dropped"? Because of queue overflow / initialization timeout?

Completing initialization without re-registering a listener can lead to "events being dropped" in some sense, but that would be working as intended.

[Rob--W]

What do you mean by "events have been dropped"? Because of queue overflow / initialization timeout?

Yes to both. If the extension is somehow in a deadlock (e.g. initialization depends on an event, but events are all suppressed), then knowing that (critical) events have been dropped would be useful.

If the browser is permitted to flush the queue and force the initialization when a timeout is reached, notifying extensions would enable them to detect bugs. Perhaps this should be one of the events that are not delayed once registered.

In theory a full queue could eventually trigger a crash and theoretically trigger crash reports that could integrate in the Reporting API. Only Chrome recognizes the "crash-reporting" endpoint though.

Completing initialization without re-registering a listener can lead to "events being dropped" in some sense, but that would be working as intended.

Indeed.

[AndreaOrru]

[...] Would you be willing to consider expanding to the post-initialization phase? With a method to control listener registration behavior, we can reduce the number of unnecessary wake-ups (#719) and writes to disk (frequency and size). E.g.:

browser.runtime.markBackgroundEventStage({
  // Commit listeners and flush (as proposed)
  listenersReady: true, // can only be true once

  // Whether to persist across browser/extension restarts
  persistAcrossSessions: true, // default true

  // Whether to persist past worker termination.
  persistAcrossUnload: true, // default true
});

...

I agree this is a real problem, and this API shouldn't preclude a future solution for "non-persistent" listeners. But I prefer not to fold that into this proposal.

I'd rather add this as "Future Work", and I think a per-listener API would be better, e.g. an explicit persistent listener option for addListener, which is similar to what you've already proposed here: #719 (comment)

[AndreaOrru]

Additionally, I realized markInitializationComplete is not very explicit ("initialization" can mean different things), and it would be better to call it markListenerRegistrationComplete.

[AndreaOrru]

Sorry for the delayed response -- please take another look.

[carlosjeurissen]

@AndreaOrru checked on the potential backwards compatibility issue. Tested the recent versions of Chrome, Firefox, Safari and Orion on desktop and all of them load fine with additional properties in the background manifest key. See https://jeurissen.co/webext-demos/background-unknown-key

[dotproto]

We should call out in this section that handling of and limits on the event queues up to individual user agents.

While the limits themselves may be undefined, we may want to define some reporting mechanisms for these cases similar to what @Rob--W suggested in this comment.

[tophf]

This solutions has an inherent problem of delaying the first dispatch, it's conceptually like the if (initStorage) await initStorage; in each listener that we currently use, so hopefully its implementation won't become a reason to delay implementation of an arguably cleaner solution of providing a synchronous small config storage available in the service worker, which was discussed in WECG some time ago.

[Rob--W]

This solutions has an inherent problem of delaying the first dispatch, it's conceptually like the if (initStorage) await initStorage; in each listener that we currently use, so hopefully its implementation won't become a reason to delay implementation of an arguably cleaner solution of providing a synchronous small config storage available in the service worker, which was discussed in WECG some time ago.

These features are developed independently, the PR for the synchronous storage is at #793 and actively going through review and revision cycles right now.

It is a subject of discussion whether synchronous storage should persist across browser restarts. If that is not the case, an implementation could try to synchronously access data where available, and otherwise fall back to the delayed registration proposed here.

[Rob--W]

@AndreaOrru The current proposal postpones all events, by design. While this penalty hopefully encourages extension developers to minimize the delay, the reality may be different.

I'm sketching a few scenarios below where the specified behavior may be too strict. They could be mitigated but that may result in additional complexity. I'd like to see your thoughts on these. To be clear, this not intended as a long lasting blocker, but to check if the API is in the desired shape before we proceed to the implementation.

In a comment above I mentioned runtime.onPerformanceWarning as an example of an event that could enable extension developers to discover issues. If events were postponed indefinitely, extensions could not be notified of issues and handle them accordingly.

Time-sensitive events: Network requests may be blocked on webRequest event handlers (unconditionally in Firefox, in Chrome conditional for policy-installed extensions except webRequest.onAuthRequired). A potential mitigation is to dispatch events as soon as a listener is registered, instead of queueing them all up until the end. There are pros and cons to this.

Documenting risks of API usage: When events are delayed, TOCTOU issues become more likely. This has always been an issue with asynchronous APIs and is not new, but I think that it is worth noting that the use of this API increases the risk of that. There may be some API-specific solutions in some cases (e.g. documentId).

Some events may have different requirements for handling them. E.g. maybe runtime.onMessage and webRequest listeners should be processed ASAP, but webNavigation events might be less urgent.

---

I see the appeal in the proposal here from the ease of implementation and their obvious semantics. Perhaps it is sufficient - I am not opposed to the approach. Alternative API design knobs I can think of:

• decoupling async registration from blocking event dispatch: fire at the next microtask (or later) after the first registration. Potential issues: This can result in out-of-order event dispatches if an extension registers events at different times. This can also result in missed events if an extension registers two listeners for the same event at different times.
• letting extensions declare (in the manifest? At runtime, per synchronous API call at the top level?) whether they would like to postpone the registration of specific events (independently of event filters).

[AndreaOrru]

Thanks @Rob--W, this is a good articulation of the main trade-offs in the proposal. Allow me to group a few paragraphs and answer inline. =)

I see the appeal in the proposal here from the ease of implementation and their obvious semantics [...]

Yes, the current design postpones extension API events until markListenerRegistrationComplete() because that gives the extension a single, atomic initialization boundary. Async state can be loaded, all intended listeners can be registered, routing state can be committed, and only then are queued events flushed.

Time-sensitive events: Network requests may be blocked on webRequest event handlers (unconditionally in Firefox, in Chrome conditional for policy-installed extensions except webRequest.onAuthRequired). A potential mitigation is to dispatch events as soon as a listener is registered, instead of queueing them all up until the end. There are pros and cons to this.
[...]
decoupling async registration from blocking event dispatch: fire at the next microtask (or later) after the first registration. Potential issues: This can result in out-of-order event dispatches if an extension registers events at different times. This can also result in missed events if an extension registers two listeners for the same event at different times.

I don't think we should switch the default behavior to "dispatch as soon as a listener for this event is registered". Although it improves latency in some cases, it introduces order-dependent behavior during initialization, as you correctly point out. That undermines the atomicity that this proposal is trying to provide, which seems like a greater loss to me.

Some events may have different requirements for handling them. E.g. maybe runtime.onMessage and webRequest listeners should be processed ASAP, but webNavigation events might be less urgent.
[...]
letting extensions declare (in the manifest? At runtime, per synchronous API call at the top level?) whether they would like to postpone the registration of specific events (independently of event filters).

Agreed that not all events have the same requirements. More granular controls, such as per-event manifest declarations or an API to opt individual events in/out of postponed dispatch, may be useful. My inclination is to treat those as future work unless the group feels the default semantics are too broad for the initial implementation.

In a comment above I mentioned runtime.onPerformanceWarning as an example of an event that could enable extension developers to discover issues. If events were postponed indefinitely, extensions could not be notified of issues and handle them accordingly.

This seems qualitatively different from dispatching e.g. webRequest early. Those events are part of the extension's normal application logic, and dispatching them during partial initialization can create order-dependent behavior as explained above. runtime.onPerformanceWarning, by contrast, is a diagnostic signal, IIUC. Delivering it early does not compromise the atomicity of normal listener initialization in the same way. So it seems worth treating as an exception.

Documenting risks of API usage: When events are delayed, TOCTOU issues become more likely. This has always been an issue with asynchronous APIs and is not new, but I think that it is worth noting that the use of this API increases the risk of that. There may be some API-specific solutions in some cases (e.g. documentId).

Acknowledged, but I don't see a way around it; it's the nature of async APIs, and this proposal can increase the frequency of such scenarios. The documentation should warn that delayed event delivery can increase stale-state / TOCTOU risks. Extensions should revalidate when handling delayed events and prefer stable identifiers where available.

---

@dotproto, once we reach consensus on the semantics of "late registration", I will update the proposal with your suggestions. @Rob--W, let me know if you're comfortable with the trade-offs, and I'll update the proposal to call out some of these points explicitly as well.

[AndreaOrru]

Addressed several of @dotproto and @Rob--W comments, also renamed the "Initialization Lifecycle" section to "Listener Registration Lifecycle" and rewritten for extra clarity.

[Rob--W]

LGTM!