[feat][cli] Support for transaction consistence test with PerformanceProducer.

[thetumbled]

Fixes #19780

Motivation

enhance PerformanceProducer to support for transaction consistence test.

Modifications

• implement a message formatter to generate specific message format.
• implement client-side persistence scheme.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: thetumbled#18

[thetumbled]

BTW, this PR do not contain the code for reading content from topics and check for the data integrity, because i implement it out of pulsar project. If needed, i could implement it into the pulsar-perf tools.

the detailed code is as follows:

package org.example.utils;

public class PulsarCommonUtils {
    private static final Logger log = LoggerFactory.getLogger(PulsarCommonUtils.class);

    @Parameter(names = "-topic", description = "topic name")
    String topic = "persistent://test/test/testTxn10";

    @Parameter(names = "-sub", description = "subscription name")
    String sub = "sub";

    @Parameter(names = { "-u", "--service-url" }, description = "Pulsar Service URL")
    public String serviceURL;

    @Parameter(names = "-token", description ="token used to authenticate.")
    public String token = "";

    @Parameter(names = "--help", description = "help info", help = true)
    private boolean help = false;


    // check the content of topic whether constitute a continuous range
    public static boolean checkCorrectness(PulsarCommonUtils pulsarCommonUtils) throws PulsarClientException {
        boolean result = true;
        if (pulsarCommonUtils.serviceURL == null || pulsarCommonUtils.serviceURL.length() == 0) {
            log.error("please input Pulsar Service URL");
            return false;
        }
        PulsarClient client = PulsarClient.builder()
                .serviceUrl(pulsarCommonUtils.serviceURL)
                .authentication(AuthenticationFactory.token(pulsarCommonUtils.token))
                .build();
        Consumer<byte[]> consumer = client.newConsumer()
                .topic(pulsarCommonUtils.topic)
                .subscriptionName(pulsarCommonUtils.sub)
                .subscribe();
        RangeSet<Long> rangeSet = TreeRangeSet.create();
        Message msg;
        Long id;
        long msgCount = 0l;
        long duplicateCount = 0l;
        while (true) {
            msg = consumer.receive(10, TimeUnit.SECONDS);
            if (msg == null) {
                break;
            }
            msgCount++;
            id = CommonUtils.bytesToLong(msg.getData());
            if (rangeSet.contains(id)) {
                duplicateCount++;
            } else {
                // for n, we add range [n,n+1) into rangeSet.
                rangeSet.add(Range.closedOpen(id, id + 1));
            }
            consumer.acknowledge(msg);
        }
        // check for message loss
        Set<Range<Long>> set = rangeSet.asRanges();
        if (set.size() > 1) {
            long validCount = 0;
            for (Range<Long> range : set) {
                validCount += range.upperEndpoint() - range.lowerEndpoint();
            }
            log.info("Some messages are missing! more than one range exists." +
                    "Valid messages Count:{} RangeSet:{}", validCount, rangeSet);
            result = false;
        }else if (set.size() == 0) {
            log.info("there is no any message!");
            return true;
        }else{
            log.info("only one range exists. RangeSet:{}", rangeSet);
        }

        // check for message duplication
        if (duplicateCount != 0) {
            result = false;
        }
        log.info("duplicate message count:{}, total message count:{}, duplicate rate:{}%",
                duplicateCount, msgCount, 100.0 * duplicateCount / msgCount);


        client.close();
        return result;
    }

    public static void main(String[] args) {
        PulsarCommonUtils pulsarCommonUtils = new PulsarCommonUtils();
        JCommander jcommander = JCommander.newBuilder().addObject(pulsarCommonUtils).build();
        try {
            jcommander.parse(args);
            if (pulsarCommonUtils.help) {
                jcommander.usage();
                System.exit(1);
            } else {
                if (pulsarCommonUtils.checkCorrectness) {
                    checkCorrectness(pulsarCommonUtils);
                } 
            }
        } catch (Exception e) {
            System.err.println(e);
            System.exit(1);
        }
    }
}

[github-actions]

The pr had no activity for 30 days, mark with Stale label.