SIGN IN SIGN UP
apache / spark UNCLAIMED

Apache Spark - A unified analytics engine for large-scale data processing

0 0 1093 Scala

[SPARK-50540][PYTHON][SS] Fix string schema for StatefulProcessorHandle

### What changes were proposed in this pull request?

Fix string schema for StatefulProcessorHandle, it was throwing an error before when passing the schema as String type because the utility method we used `_parse_datatype_string` requires a SparkContext which is not available on executors.

The way we support it is to create a new API `ParseStringSchema` from the client side (Python worker) to server side (JVM). Client passes a string schema to the server, we do the parsing on server side and then send the result back to the client.

### Why are the changes needed?

This is to fix an issue/bug with the existing code.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Updated existing test cases to include string schemas.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #49138 from bogao007/parse-schema.

Authored-by: bogao007 <bo.gao@databricks.com>
Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
B
bogao007 committed
5538d8536e9d7fd027c7724463ff856081702599
Parent: 3e7b614
Committed by Jungtaek Lim <kabhwan.opensource@gmail.com> on 12/14/2024, 1:10:58 AM