Skip to content

Add Memq Metadata Client#134

Open
ArtemTetenkin wants to merge 4 commits into4.0from
memq_metadata_client
Open

Add Memq Metadata Client#134
ArtemTetenkin wants to merge 4 commits into4.0from
memq_metadata_client

Conversation

@ArtemTetenkin
Copy link
Contributor

@ArtemTetenkin ArtemTetenkin commented Mar 10, 2026

Add Memq Metadata Client
Here's a summary of all the changes:

Files created
psc/src/main/java/com/pinterest/psc/metadata/client/memq/PscMemqMetadataClient.java - The main Memq metadata client implementation extending PscBackendMetadataClient. Since Memq doesn't have a dedicated admin client (unlike Kafka's AdminClient), it uses short-lived MemqConsumer instances for all metadata queries. The implementation:

listTopicRns() - throws UnsupportedOperationException since Memq doesn't support listing all topics
describeTopicUris() - creates a consumer per topic, subscribes, and calls getPartition() to discover partitions
listOffsets() - groups partitions by topic, uses getEarliestOffsets() / getLatestOffsets() based on the requested option
listOffsetsForTimestamps() - groups by topic and uses offsetsOfTimestamps()
listOffsetsForConsumerGroup() - creates a consumer with the target consumer group ID and uses committed() per partition
psc/src/main/java/com/pinterest/psc/metadata/creation/PscMemqMetadataClientCreator.java - The plugin creator class annotated with @PscMetadataClientCreatorPlugin(backend = "memq", priority = 1), following the exact same pattern as PscKafkaMetadataClientCreator. It creates and initializes PscMemqMetadataClient instances and validates Memq topic URIs via MemqTopicUri.validate().

psc/src/main/java/com/pinterest/psc/config/PscMetadataClientToMemqConsumerConfigConverter.java - Config converter that extends PscMetadataClientToBackendMetatadataClientConfigCoverter, mapping PSC metadata client configuration to Memq consumer configs (e.g., PSC_METADATA_CLIENT_ID to ConsumerConfigs.CLIENT_ID).

File modified
psc/src/main/java/com/pinterest/psc/consumer/memq/MemqTopicUri.java - Changed the constructor from package-private to public so it can be used from the com.pinterest.psc.metadata.client.memq package (consistent with how KafkaTopicUri has a public constructor).

@ArtemTetenkin ArtemTetenkin requested a review from a team as a code owner March 10, 2026 18:52
public List<TopicRn> listTopicRns(Duration duration)
throws ExecutionException, InterruptedException, TimeoutException {
throw new UnsupportedOperationException(
"[Memq] Listing all topics is not supported by the Memq backend.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is needed - we'll update MemQ consumer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants