KAFKA-18418: Use CDL to block the thread termination to avoid flaky tests

[aoli-al]

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.

This PR fixes KAFKA-18418. KafkaStreamsTest uses Thread.sleep to prevent threads from terminating. This introduces flaky tests if the sleep duration is not long enough. This patch fixes the issue by replacing the Thread.sleep with a CountDownLatch. The CountDownLatch will be released after assertions are validated.

Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.

I tested the proposed fix against the patch: aoli-al@aced4f1 and verified that all tests have passed. I also tested the new code using Fray (the tool that found the bug), and Fray did not report any bug using the POS strategy after 10 minutes.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[mjsax]

Should we move this into a finally block? Same below.

[aoli-al]

No, because the try (final KafkaStreams streams = new KafkaStreams(getBuilderWithSource().build(), props, supplier, time)) statement will call streams.close() at the end of the try block but before the final block. So we need to call terminableThreadBlockingLatch.countDown(); inside the try block.

If we want to ensure terminableThreadBlockingLatch.countDown(); is always called. We may write:

        try (final KafkaStreams streams = new KafkaStreams(getBuilderWithSource().build(), props, supplier, time)) {
            streams.start();
            try {
                waitForCondition(
                    () -> streams.state() == KafkaStreams.State.RUNNING,
                    "Streams never started.");
                streams.close(Duration.ZERO);
                assertThat(streams.state() == State.PENDING_SHUTDOWN, equalTo(true));
                assertThrows(IllegalStateException.class, streams::cleanUp);
                assertThat(streams.state() == State.PENDING_SHUTDOWN, equalTo(true));
            } finally {
                terminableThreadBlockingLatch.countDown();
            }
        }

Please let me know if you prefer this.