Skip to content

[spark][flink] Reject row-level operations and rewrites under query auth rules - #10906

Draft
plusplusjiajia wants to merge 1 commit into
apache:masterfrom
plusplusjiajia:fix/reject-rewrites-under-query-auth-rules
Draft

plusplusjiajia wants to merge 1 commit into
apache:masterfrom
plusplusjiajia:fix/reject-rewrites-under-query-auth-rules

Conversation

@plusplusjiajia

@plusplusjiajia plusplusjiajia commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Purpose

On a table with query auth, operations that read rows and write data back did not respect the caller's row filter and column masks:

  • Spark UPDATE, DELETE and MERGE INTO on append and Data Evolution tables find and read the candidate files through raw readers (newSnapshotReader and KnownSplitsTable). So UPDATE t SET note = ssn copied a masked column into a visible one, and rows hidden by the row filter were updated or deleted, for example DELETE FROM t without a condition.
  • On primary-key tables, Spark and Flink UPDATE and MERGE INTO, and the Flink delete and merge-into actions, read the filtered, masked rows and wrote them back. Masked values were stored as real data, and masked keys could delete rows hidden by the row filter.
  • Flink sort compaction and rescale, and Flink and Spark chain table compaction, read through the caller's filtered, masked view and overwrote partitions with it. The hidden rows were lost and the masks were stored.

These operations now fail when query auth gives the caller a row filter or a column mask. The check is a shared helper next to the search rejection, TableQueryAuthResult.rejectRewriteUnderQueryAuth, which authorizes with all columns and refuses when the result has rules. Users with full access are not affected, which matches PyPaimon, where update, delete, merge and index builds require full table access.

The check is called from:

  • Spark: the UPDATE, DELETE and MERGE INTO commands (including Data Evolution tables and the Spark 4.0 merge commands), the V2 row-level operations, the optimized DELETE / truncate, and CompactChainTableProcedure;
  • Flink: the row-level UPDATE / DELETE sink and the Delete, MergeInto, DataEvolutionMergeInto, SortCompact, Rescale and CompactChainTable actions.

Rewrites that read the raw files, such as the Spark sort compaction, clustering and rescale procedures, keep the data intact and are unchanged.

This builds on #10903, which stops a dynamic query-auth.enabled=false from turning query auth off; without it the check here can be bypassed the same way.

Tests

  • SparkCatalogWithRestTest#testRowLevelOperationsRequireFullTableAccess: Spark UPDATE, DELETE and MERGE INTO are refused for a restricted user on append tables (with V1 and V2 writes, and with deletion vectors), primary-key tables and Data Evolution tables, and still work with full access.
  • RESTCatalogITCase#testRowLevelOperationsRequireFullTableAccess and #testMaintenanceRewritesRequireFullTableAccess: Flink row-level operations, sort compaction and rescale are refused for a restricted user and still work with full access.
  • RESTCatalogITCase#testRowLevelActionsRequireFullTableAccess: the Flink delete and merge-into actions are refused for a restricted user on primary-key and Data Evolution tables without changing the data or creating a snapshot, and still work with full access.
  • The chain table compaction checks are not covered by a REST test yet: the existing chain table tests do not run on the REST catalog.
  • The new Spark cases fail on master. The related tests pass locally, including the existing Spark and Flink query auth, row-level, merge-into and compaction suites. Spark 3, Spark 4 and Flink 1 compile without the fast-build profile.

@plusplusjiajia
plusplusjiajia force-pushed the fix/reject-rewrites-under-query-auth-rules branch from 976cfe7 to 22ae92e Compare October 31, 2026 05:16
…uth rules

On a table with query auth, operations that read rows and write data
back did not respect the caller's row filter and column masks:

- Spark UPDATE, DELETE and MERGE INTO on append and Data Evolution
  tables read the raw files, so UPDATE t SET note = ssn copied a masked
  column into a visible one, and rows hidden by the row filter were
  updated or deleted.
- On primary-key tables, Spark and Flink UPDATE and MERGE INTO, and the
  Flink delete and merge-into actions, read the filtered, masked rows and
  wrote them back, so masked values were stored as real data and masked
  keys could delete rows hidden by the row filter.
- Flink sort compaction and rescale, and Flink and Spark chain table
  compaction, overwrote partitions with the caller's filtered, masked
  rows, losing the hidden rows and storing the masks.

These operations now fail when query auth gives the caller a row filter
or a column mask, through a shared check next to the search rejection,
TableQueryAuthResult.rejectRewriteUnderQueryAuth. Users with full access
are not affected. Rewrites that read the raw files, such as the Spark
sort compaction, clustering and rescale procedures, are unchanged.
@plusplusjiajia
plusplusjiajia force-pushed the fix/reject-rewrites-under-query-auth-rules branch from 22ae92e to 4e4d863 Compare October 31, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant