Skip to content

fix(s3): drop the whole-object checksum from ranged GET responses - #3292

Open
jayhemnani9910 wants to merge 1 commit into
OpenListTeam:mainfrom
jayhemnani9910:fix/s3-range-get-checksum
Open

jayhemnani9910 wants to merge 1 commit into
OpenListTeam:mainfrom
jayhemnani9910:fix/s3-range-get-checksum

Conversation

@jayhemnani9910

Copy link
Copy Markdown
Contributor

Summary / 摘要

Reading a byte range of an object through the S3 server fails in boto3 when the object was uploaded with put_object:

put_object(b1, a.txt, b"hello world")              # boto3 sends x-amz-checksum-crc32
get_object(b1, a.txt, Range="bytes=0-4")
  main:         FlexibleChecksumError: Expected checksum DUoRhQ== did not match calculated checksum: NhCmhg==
  this branch:  b'hello'

PutObject keeps the client's x-amz-* headers in b.meta, and GetObject sends them back on every response, including ranged ones. The stored x-amz-checksum-crc32 covers the whole object, so an SDK that validates checksums checks it against the partial body and rejects it. GetObject now leaves the stored X-Amz-Checksum-* entries out when it serves a range; full GETs and HEAD still return them, and other metadata is unchanged.

  • This PR has breaking changes.
    / 此 PR 包含破坏性变更。
  • This PR changes public API, config, storage format, or migration behavior.
    / 此 PR 修改了公开 API、配置、存储格式或迁移行为。
  • This PR requires corresponding changes in related repositories.
    / 此 PR 需要关联仓库同步修改。

Related Issues / 关联 Issue

None.

Testing / 测试

  • go test ./... (only pkg/aria2/rpc fails: it needs an aria2 daemon on :6800, same on main)

  • Manual test / 手动测试:

  • Built main and this branch, S3 server on with a bucket mapped to a local storage, boto3 1.43: bytes=0-4, bytes=-5 and bytes=6- fail on main with FlexibleChecksumError and return hello / world / world here; a full GET still returns the body and ChecksumCRC32.

  • A 55-request boto3 run (listing, keys with spaces and +, metadata, copy, delete, multipart) gives the same results on main and this branch apart from the three ranged GETs.

  • New TestGetObjectRangeOmitsWholeObjectChecksum fails without the change; go vet ./server/s3/ passes.

Checklist / 检查清单

  • I have read CONTRIBUTING.
    / 我已阅读 CONTRIBUTING。
  • I confirm this contribution follows the repository license, contribution policy, and code of conduct.
    / 我确认此贡献符合仓库许可证、贡献规范和行为准则。
  • I have formatted the changed code with gofmt, go fmt, or prettier where applicable.
    / 我已按适用情况使用 gofmt、go fmt 或 prettier 格式化变更代码。
  • I have requested review from relevant maintainers or code owners where applicable.
    / 我已在适用情况下请求相关维护者或代码所有者审查。

AI Disclosure / AI 使用声明

  • This PR includes AI-assisted content.
    / 此 PR 包含 AI 辅助内容。

Tools used / 使用工具:

  • Claude

Usage scope / 使用范围:

  • Code generation / 代码生成

  • Tests / 测试

  • I have reviewed and validated all AI-assisted content included in this PR.
    / 我已审核并验证此 PR 中的所有 AI 辅助内容。

PutObject keeps the client's x-amz-checksum-* headers and GetObject sent
them back on every response, including ranged ones. boto3 validates the
checksum it gets, so reading any byte range of an object it uploaded
with put_object failed with FlexibleChecksumError. Leave the stored
checksums out when a range is served.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant