SIGN IN SIGN UP
apache / spark UNCLAIMED

Apache Spark - A unified analytics engine for large-scale data processing

0 0 1111 Scala

[SPARK-45600][PYTHON] Make Python data source registration session level

### What changes were proposed in this pull request?

This PR makes dynamic Python data source registration session-scoped. Previously, registered data sources were stored in the `sharedState` and can be referenced by other sessions, which won't work with Spark Connect.

### Why are the changes needed?

To make Python data source support Spark Connect in the future.

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

New unit test

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #43742 from allisonwang-db/spark-45600-session-level.

Authored-by: allisonwang-db <allison.wang@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
A
allisonwang-db committed
c8ad616b988efdd47d7091f51c1e4563564b4e10
Parent: 9c291e1
Committed by Hyukjin Kwon <gurwls223@apache.org> on 11/23/2023, 2:04:09 AM