SIGN IN SIGN UP
apache / spark UNCLAIMED

Apache Spark - A unified analytics engine for large-scale data processing

0 0 1122 Scala

[SPARK-56202][SS] Refactor streaming join tests: split Base/Suite hierarchy and simplify mode dispatch

### What changes were proposed in this pull request?

Refactor streaming join test suites to separate test mode dispatch from the test hierarchy.

**Before:** `AlsoTestWithVirtualColumnFamilyJoins` runs every test in both VCF and non-VCF
modes within a single suite class. V4 suites extend these classes and use a string-matching
skip list to exclude incompatible tests.

**After:** Each join type has a two-layer hierarchy:

```
StreamingInnerJoinBase          (universal tests — all state format versions)
├── StreamingInnerJoinSuite     (adds V3-specific tests: window joins, metrics, schema)
│   ├── StreamingInnerJoinWithVCFSuite      (testMode = WithVCF)
│   └── StreamingInnerJoinWithoutVCFSuite   (testMode = WithoutVCF)
└── StreamingInnerJoinV4Suite   (testMode = WithVCF, stateFormatVersion = 4)
```

Mode dispatch uses a `testMode: Mode` enum in `StreamingJoinSuite`, replacing the
`AlsoTestWithVirtualColumnFamilyJoins` trait. Each concrete suite overrides `testMode`
with a single `Mode` value. The enum is more future-proof than a boolean — new modes
can be added without changing the dispatch type. V4 suites extend `*Base` directly —
no skip list needed.

### Why are the changes needed?

1. **Failure isolation**: Suite name identifies the exact mode — no long disambiguating suffixes.
2. **Selective retrigger**: Rerun only the failing mode's suite without rerunning the passing mode.
3. **No skip lists**: V4 suites get only universal tests by inheriting from `*Base`, not `*Suite`. Adding V3-specific tests in the future won't accidentally break V4.
4. **Extensible dispatch**: A `Mode` enum replaces a trait with hard-coded dual-mode dispatch. Adding new modes (e.g., Avro encoding) is a new enum case + a one-liner suite.

### Does this PR introduce _any_ user-facing change?

No. Test infrastructure only. Same tests, same coverage.

### How was this patch tested?

Existing tests — the `*WithVCFSuite` and `*WithoutVCFSuite` classes run the same tests as before, split by mode.

### Was this patch authored or co-authored using generative AI tooling?

Yes

Closes #55005 from cloud-fan/SPARK-XXX-join-test-modes.

Authored-by: Wenchen Fan <wenchen@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
W
Wenchen Fan committed
6d4b8b04fa5f136b52d925d78df7de910ce07c7e
Parent: 842d4dd