Repository navigation
[spark][flink] Reject row-level operations and rewrites under query auth rules - #10906
Draft
plusplusjiajia wants to merge 1 commit into
Draft
plusplusjiajia wants to merge 1 commit into
plusplusjiajia wants to merge 1 commit into
Conversation
plusplusjiajia
force-pushed
the
fix/reject-rewrites-under-query-auth-rules
branch
from
October 31, 2026 05:16
976cfe7 to
22ae92e
Compare
…uth rules On a table with query auth, operations that read rows and write data back did not respect the caller's row filter and column masks: - Spark UPDATE, DELETE and MERGE INTO on append and Data Evolution tables read the raw files, so UPDATE t SET note = ssn copied a masked column into a visible one, and rows hidden by the row filter were updated or deleted. - On primary-key tables, Spark and Flink UPDATE and MERGE INTO, and the Flink delete and merge-into actions, read the filtered, masked rows and wrote them back, so masked values were stored as real data and masked keys could delete rows hidden by the row filter. - Flink sort compaction and rescale, and Flink and Spark chain table compaction, overwrote partitions with the caller's filtered, masked rows, losing the hidden rows and storing the masks. These operations now fail when query auth gives the caller a row filter or a column mask, through a shared check next to the search rejection, TableQueryAuthResult.rejectRewriteUnderQueryAuth. Users with full access are not affected. Rewrites that read the raw files, such as the Spark sort compaction, clustering and rescale procedures, are unchanged.
plusplusjiajia
force-pushed
the
fix/reject-rewrites-under-query-auth-rules
branch
from
October 31, 2026 06:03
22ae92e to
4e4d863
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
On a table with query auth, operations that read rows and write data back did not respect the caller's row filter and column masks:
UPDATE,DELETEandMERGE INTOon append and Data Evolution tables find and read the candidate files through raw readers (newSnapshotReaderandKnownSplitsTable). SoUPDATE t SET note = ssncopied a masked column into a visible one, and rows hidden by the row filter were updated or deleted, for exampleDELETE FROM twithout a condition.UPDATEandMERGE INTO, and the Flink delete and merge-into actions, read the filtered, masked rows and wrote them back. Masked values were stored as real data, and masked keys could delete rows hidden by the row filter.These operations now fail when query auth gives the caller a row filter or a column mask. The check is a shared helper next to the search rejection,
TableQueryAuthResult.rejectRewriteUnderQueryAuth, which authorizes with all columns and refuses when the result has rules. Users with full access are not affected, which matches PyPaimon, where update, delete, merge and index builds require full table access.The check is called from:
UPDATE,DELETEandMERGE INTOcommands (including Data Evolution tables and the Spark 4.0 merge commands), the V2 row-level operations, the optimizedDELETE/ truncate, andCompactChainTableProcedure;UPDATE/DELETEsink and theDelete,MergeInto,DataEvolutionMergeInto,SortCompact,RescaleandCompactChainTableactions.Rewrites that read the raw files, such as the Spark sort compaction, clustering and rescale procedures, keep the data intact and are unchanged.
This builds on #10903, which stops a dynamic
query-auth.enabled=falsefrom turning query auth off; without it the check here can be bypassed the same way.Tests
SparkCatalogWithRestTest#testRowLevelOperationsRequireFullTableAccess: SparkUPDATE,DELETEandMERGE INTOare refused for a restricted user on append tables (with V1 and V2 writes, and with deletion vectors), primary-key tables and Data Evolution tables, and still work with full access.RESTCatalogITCase#testRowLevelOperationsRequireFullTableAccessand#testMaintenanceRewritesRequireFullTableAccess: Flink row-level operations, sort compaction and rescale are refused for a restricted user and still work with full access.RESTCatalogITCase#testRowLevelActionsRequireFullTableAccess: the Flink delete and merge-into actions are refused for a restricted user on primary-key and Data Evolution tables without changing the data or creating a snapshot, and still work with full access.