[j1wonpark]

Why are the changes needed?

Closes #4377.

Decouple the Iceberg version used by AMS and optimizers from mixed-format connectors, allowing maintenance upgrades while preserving support for older connector engines.

Brief change log

• Use Iceberg 1.10.2 by default. Shared Iceberg format and mixed-format modules explicitly retain 1.8.1 through connector-iceberg.version.
• Align Parquet to 1.16.0 in the parent dependency management and optimizer Avro to 1.12.1.
• Adapt shared Parquet readers to APIs available in both Iceberg versions and add reader and serialization regression tests.
• Update an optimizing test fixture for Iceberg 1.10's REPLACE validation.
• Exclude AMS and dist from Spark 3.3 CI builds; retain connector and optimizer builds.
• Fix Flink optimizer version selection, Java 11 image tags, and cancellation timeout compatibility.
• Update distribution dependency metadata and licenses.

Hadoop 2 support removal and persisted-task recovery handling were merged separately in #4452 and #4442.

How was this patch tested?

• Added AMS tests that read Parquet through shared Hive and Spark readers.
• Added an optimizer DataFile serialization round-trip test and Flink cancellation timeout tests.
• During development, validated shared readers against both Iceberg versions, AMS optimizing and cleanup operations, reactor builds, and Flink optimizer execution.
• Connector tests passed after the final Parquet alignment. Additional Flink integration failures also reproduced with the previous dependency versions.

Validation was performed across multiple revisions, not as a complete rerun against the final merged revision. The earlier Flink integration run used an AMS distribution from a separate BOM experiment.

Notes

• AMS and optimizers must use the same Iceberg version because task descriptors contain Java-serialized Iceberg objects.
• Shared format modules still compile against 1.8.1; placement of code requiring newer APIs remains follow-up work.
• Spark 3.3 connectors remain supported, but AMS requires a newer Spark binding.
• BOM adoption and engine-specific Iceberg version selection remain follow-up work.

[zhoujinsong]

@j1wonpark Thanks for the work.

Regarding whether Hadoop 2 should be removed or only supported within the mixed format, I suggest initiating a discussion thread on the dev mailing list to raise awareness of this issue and gather more opinions.

[sivakumar-mahalingam]

PR notes that the Hadoop 2 distribution still builds, but AMS cannot read Parquet with the upgraded dependency. Could we remove or disable the Hadoop 2 distribution build before merging, so users aren't provided with an artifact that cannot perform this operation?

[j1wonpark]

@zhoujinsong Thanks, opened the thread: https://lists.apache.org/thread/v7zhhzzgn14wk1vwfqygy1mswwwgb9fy
Independent of the outcome, AMS and the optimizers cannot read Parquet on Hadoop 2 either way, so 6533fdd removes the Hadoop 2 distribution build; the connector side is untouched until the discussion settles.

[j1wonpark]

@sivakumar-mahalingam Agreed, done in 6533fdd: the Hadoop 2 binary release, docker image and dist dependency list are removed.
The optimizer fails the same way on Hadoop 2.10.2, so the README now describes the Hadoop 2 build as connectors only.

[sivakumar-mahalingam]

The change looks good 👍

[sivakumar-mahalingam]

Since the Hadoop 2 distribution is being removed, would it be useful to create a follow-up issue to clean up the remaining Hadoop 2 dependencies/configuration in AMS and optimizers, such as flink-shaded-hadoop-2-uber?

I can take this up if it is aligned with the direction of the project.

[sivakumar-mahalingam]

@j1wonpark I have already fixed the ci failure as part #4392
Once it is merged, this ci failure won't occur

[sivakumar-mahalingam]

@j1wonpark created a pr #4394

[xxubai]

Could we handle incompatible persisted task inputs/outputs during AMS recovery? After upgrading Iceberg, deserialization failures wrapped as PersistenceException bypass the current recovery handler and can leave tables stuck.

[j1wonpark]

@xxubai @zhoujinsong @czy006 Could you review this PR? Updated the default Iceberg version as suggested, addressed recovery in #4442, and removed changes already merged in #4452.

[xxubai]

LGTM.