Expose SparkSubmitHook fields used by submit backends - #71661

Problem

#68679 moved K8s/YARN submit tracking into strategy backends. Those backends have to read and mutate SparkSubmitHook state that is still private: _conf, _kubernetes_driver_pod, _yarn_application_id, _poll_k8s_driver_via_api, _kill_yarn_application. Closed draft #71059 over-promoted about 10 members. This is the narrower follow-up.

What I did

I promoted only the members the backends actually use:

• conf — live dict, not a copy, so in-place key writes still work
• kubernetes_driver_pod — get/set
• yarn_application_id — get only
• poll_k8s_driver_via_api
• kill_yarn_application (Yarn on_kill has the same leak)

Private names stay as the backing store, so existing hook tests that set _yarn_application_id / _kubernetes_driver_pod keep working. I switched the K8s and YARN operator backends to the public names.

Out of scope, on purpose: _connection, _start_yarn_application_status_tracking, _run_post_submit_commands, _driver_id / _driver_status.

How I did it

Properties and a rename on SparkSubmitHook, then the same names in _KubernetesSparkSubmitBackend and _YarnSparkSubmitBackend. Operator MagicMock fixtures now use the public names so .conf / .yarn_application_id are not accidental new MagicMocks.

Impact

Submit backends no longer reach through private attributes. External subclasses that need the YARN kill or K8s poll can call the public methods. Hook internals and tracking behavior are unchanged.

Testing

Hook tests:

• test_conf_mutation_writes_through
• test_kubernetes_driver_pod_writes_through
• test_yarn_application_id_is_get_only (assignment raises AttributeError)
• existing K8s poll / YARN kill tests retargeted to the public names

Operator tests updated so MagicMock fixtures hit the public surface, including on_kill → kill_yarn_application.

pytest providers/apache/spark/tests/unit/apache/spark/hooks/test_spark_submit.py providers/apache/spark/tests/unit/apache/spark/operators/test_spark_submit.py

closes: #71058

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: Cursor following the guidelines

Made with Cursor