Chunked (multipart) cloud upload: lift the 5 GiB AWS limit and stop reading whole files into RAM #834

Problem

storage file-upload, storage upload-table and the SDK Client.upload_table all go through KeboolaClient._upload_to_cloud (src/keboola_agent_cli/client/storage_tables.py). It sends the whole file to cloud storage in one request on every provider. That sets a hard size ceiling and, on AWS, causes a memory problem:

There is no resume and no per-part retry either: one dropped connection 4 GB into an upload restarts from zero (the same gap #417 lists for sliced downloads).

Reference: the Go CLI (kbc, keboola/keboola-as-code)

kbc remote file upload (pkg/lib/operation/project/remote/file/upload/operation.go) streams io.Copy(reader -> blobWriter) into transfer.NewUploadWriter from keboola/keboola-sdk-go (v2/transfer/{s3,abs,gcs}/upload.go), which wraps gocloud.dev/blob:

• S3: AWS SDK upload manager (multipart above one part), PartSize = 5 MiB. Sets ACL and ServerSideEncryption from uploadParams. Ceiling: 10,000 parts x 5 MiB ≈ 48.8 GiB.
• Azure: staged blocks (Put Block + Put Block List) via azblob UploadStream. The SDK requests 512,000-byte blocks; azblob v1.6.x clamps them to 1 MiB, so the ceiling is ≈ 48.8 GiB (50,000 blocks).
• GCS: resumable upload, chunk rounded to 512 KiB, retry policy RetryIdempotent.
• Memory is bounded by buffer size x concurrency. It also accepts stdin (-), and a directory as a sliced file.

The old CLI was clearly better (chunked everywhere, bounded memory), but its fixed small part sizes still cap it at ~49 GiB. We should scale the part size with the file size.

For comparison, sapi-python-client (kbcstorage/files.py) does a single s3_object.put(Body=file) on AWS (same ~5 GiB ceiling), upload_blob on Azure (the SDK chunks) and upload_from_file on GCS (resumable).

Server-side constraints (keboola/connection)

From connection/legacy-app/core/services/FileStorage/Adapter/S3Adapter.php:

1. The write federation token's policy grants only s3:PutObject and s3:PutObjectAcl on the file key. CreateMultipartUpload, UploadPart and CompleteMultipartUpload are authorized as s3:PutObject, so multipart should be allowed (the Go CLI relies on it). s3:AbortMultipartUpload is not explicitly granted. Whether abort still works (AWS documents default abort rights for the initiator) and whether the files bucket has an AbortIncompleteMultipartUpload lifecycle rule must be verified. Until then, assume a failed upload leaves orphaned parts behind.
2. With isEncrypted: true, the policy adds Deny s3:PutObject with StringNotEquals s3:x-amz-server-side-encryption = AES256. SSE parameters belong on CreateMultipartUpload, not on each part. If the condition key is absent from UploadPart's authorization context, the Deny matches and multipart fails on encrypted files. Unverified; needs a live test.
3. isEncrypted defaults to false on the server (FilePrepareRequestInterface::DEFAULT_IS_ENCRYPTED). kbagent does not send it, so it never activates Keboola's SSE enforcement policy. (The objects are still SSE-S3 encrypted by the bucket's default encryption.) The Go SDK and sapi-python-client send isEncrypted: true. kbagent's S3 path also ignores the returned uploadParams.acl and x-amz-server-side-encryption. This issue should forward both, on the single PUT and on CreateMultipartUpload. Whether to start sending isEncrypted: true is a separate decision; do not change it silently.
4. Credential lifetimes: AWS GetFederationToken with no DurationSeconds lasts 12 h, and the Azure SAS is also 12 h. The GCS access_token lifetime is not a problem, because resumable chunk PUTs are authorized by the session URI (valid one week), not by the bearer. Treat the session URI as a secret: never log it.

Proposal

Replace the single-request upload with a chunked upload per provider. Implement it natively over httpx: no boto3 / azure-storage-blob / google-cloud-storage dependency. Each protocol needs only a few operation types, and the SDKs would grow the PyInstaller binaries. Keep the legacy presigned-POST and signed-URL fallbacks as they are; those credentials cannot authorize multipart.

• S3: CreateMultipartUpload (with x-amz-acl, plus SSE when present) -> UploadPart x N (each part buffered, SigV4 with that part's x-amz-content-sha256) -> CompleteMultipartUpload with the recorded ETags in order.
  • The existing _s3_signed_headers (client/_transfer.py) is not sufficient. It sorts raw query fragments and cannot sign extra headers. It needs canonical query encoding (uploads=, uploadId, partNumber) and signed x-amz-acl / SSE headers, with tests for reserved characters in keys.
  • CompleteMultipartUpload can return HTTP 200 with an <Error> body, preceded by whitespace keep-alives. Parse the full XML and require a CompleteMultipartUploadResult.
• Azure: Put Block x N (fixed-length base64 block IDs, Content-MD5 per block) -> Put Block List using <Latest> entries. Pin the operation version explicitly. Do not break the SAS signature when you add parameters.
• GCS: resumable session (POST .../upload/storage/v1/b/{bucket}/o?uploadType=resumable&name={key} with the bearer) -> chunked PUTs with Content-Range (fixed chunk, multiple of 256 KiB), handling 308. After a transport error or 5xx, query the session status and resume at Range.end + 1; a missing Range means 0 bytes persisted. Validate the whole object at the end (e.g. a crc32c or md5 comparison).
• Part size: S3 uses max(8 MiB, ceil(size / 10,000)), Azure uses max(8 MiB, ceil(size / 50,000)), GCS uses a fixed aligned chunk. Use integer arithmetic and refuse before initiation when the file exceeds the provider maximum. Note that peak RAM ≈ one part: 8 MiB for most files, but ~524 MiB for a 5 TiB S3 object. Files below a threshold (e.g. 64 MiB) keep one request.
• Sequential parts first. Parallelism can come later behind a constant.
• Retry contract (in-process only, no resume across runs): retry 429 / 5xx / transport errors per part with the existing backoff constants, honouring a bounded Retry-After. A retry reuses the same identity: the S3 (uploadId, partNumber) with the same bytes, the same Azure block ID, the GCS status query. That includes lost responses on initiate and finalize, not just explicit 503s. An exhausted part raises UPLOAD_FAILED naming the part. FILE_UPLOAD_TIMEOUT values are inactivity limits, so add an overall elapsed-time budget for the whole upload.
• Sizes, threshold, retries and the time budget live in constants.py.

upload-table caveat: the import must start exactly once and only after a confirmed cloud completion. Its 600 s poll budget (IMPORT_JOB_MAX_WAIT) can run out on a multi-GB import while the server keeps importing; blindly re-running an incremental upload would duplicate rows. At minimum, the timeout error must say the job continues and name its ID (as workspace load does since #687).

Out of scope (follow-ups): parallel parts, a progress bar, stdin input, sliced upload from a directory (Go CLI parity), defaulting isEncrypted to true, resume across process restarts.

Tests

Unit tests (pytest-httpx), per provider:

• boundary sizes: 0 B, threshold - 1, threshold, an exact multiple of the part size, a short last part, and over the provider maximum (refused before any request)
• part-size scaling when 8 MiB would exceed the part or block count
• a part fails once with 503 / a transport error and then succeeds: only that part is re-sent
• lost response on initiate and on Complete / Put Block List / the final GCS chunk
• exhausted retries -> UPLOAD_FAILED naming the part
• S3 Complete returns 200 with an <Error> body -> handled as failure / retry
• SigV4: canonical query for uploads=, uploadId, partNumber; ACL/SSE headers signed; keys with reserved characters
• Azure: equal-length block IDs, <Latest> order matches upload order, the version is pinned, the SAS query stays intact
• GCS: 308 with and without Range; status query after a 5xx; the session URI never appears in logs or errors
• seeks past 4 GiB (sparse temp file), and the source file shrinking during upload -> clear error
• Windows: file handles closed on failure (Windows CI)

Verification (live, before merge)

• Upload a 6 GB file with storage file-upload on an AWS, an Azure and a GCP stack, storage file-download it, and compare checksums.
• AWS: one run with isEncrypted: true (constraint 2), one deliberate mid-upload failure to see whether abort works and whether orphaned parts are cleaned up (constraint 1).
• Azure: record the sv of the issued SAS and the x-ms-version echoed in responses.
• Peak RSS for a 2 GB upload on AWS before and after (measured with a portable method such as psutil sampling, not just /usr/bin/time -l). Target: from ~2 GB down to tens of MB.
• storage upload-table with a > 5 GB CSV on AWS end to end (prepare -> multipart -> import): row count matches, no duplicate import.