Auto SDKAuto DriveS3 Layer (S3 compatibility)

Auto Drive S3 Layer Guide

Overview

Auto Drive provides an S3-compatible API layer that allows you to interact with decentralized storage (DSN) using standard Amazon Web Services Simple Storage Service (AWS S3) SDK commands. This bridges the gap between familiar cloud storage patterns and next-generation decentralized infrastructure, giving developers the best of both worlds: the reliability and developer experience of S3 APIs with the permanence and censorship-resistance of decentralized storage.

For those unfamiliar, Amazon Web Services Simple Storage Service (AWS S3) is an industry-standard object storage service that powers much of the modern web’s file storage needs. Auto Drive maintains compatibility with the most-used parts of S3’s APIs while storing your data on a decentralized network instead of centralized servers.

Permanent storage, versioned keys. Content stored on the Autonomys DSN is permanent: nothing is ever erased. Your S3 keys behave like a versioned S3 bucket on top of it. DeleteObject hides an object and returns 204, but its data stays on the DSN. Uploading to an existing key adds a new version, and earlier versions are kept. Only destroying a version is refused (403).

How It Works

Auto Drive maps S3 (bucket, key) pairs in your account to Content Identifiers (CIDs) and MD5 checksums. When you upload via the S3 API, the system:

  1. Stores the file content on the decentralized network (DSN)
  2. Computes the MD5 of the content and records the (bucket, key) → (cid, md5) mapping for your account
  3. Returns the MD5 as the ETag (standard S3 format) and exposes the CID in the x-amz-meta-cid response header
  4. Makes the file available in your Auto Drive account, in the web app and through the native Auto Drive API

Key Features

1. Standard S3 SDK and CLI Compatibility

  • Works with the official AWS S3 SDKs (@aws-sdk/client-s3, boto3), the AWS CLI, and S3-compatible tools like rclone
  • Supported operations:
    • Objects: PutObject, GetObject, HeadObject, DeleteObject, CopyObject
    • Listing: ListBuckets, ListObjectsV2, ListObjectVersions
    • Multipart uploads: CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload
    • Versioning and Object Lock (read-only): GetBucketVersioning, GetObjectLockConfiguration, GetObjectRetention, GetObjectLegalHold
  • No code changes required for existing S3 applications other than swapping the endpoint and credentials, and removing bucket-creation calls

2. Buckets

Buckets behave like standard S3 buckets. The first path segment of the object key is the bucket name, and the remainder is the key:

S3 pathBucketKey
my-archive/report.pdfmy-archivereport.pdf
my-archive/sub/file.txtmy-archivesub/file.txt
test.txt (no slash)defaulttest.txt
  • Buckets are created implicitly by the first upload into them, and disappear from ListBuckets once they have no visible objects.
  • Buckets and keys are private to your account. ListBuckets (GET /) returns only your own buckets, and two accounts can use the same bucket and key names without seeing each other’s objects.
  • There are no bucket-level endpoints. Auto Drive reads a request to a bare bucket path as a request for an object of that name in the default bucket. CreateBucket would store an object named after the bucket, and DeleteBucket would hide such an object if one exists, so don’t send CreateBucket, HeadBucket or DeleteBucket.
import { ListBucketsCommand } from "@aws-sdk/client-s3";
 
const result = await s3Client.send(new ListBucketsCommand({}));
console.log(result.Buckets); // [{ Name: "my-archive", CreationDate: ... }, ...]

Single-segment keys uploaded before bucket support was introduced remain accessible under the default bucket.

3. ListObjectsV2 with Prefix, Delimiter, and Pagination

Auto Drive implements ListObjectsV2 end-to-end, with the parameters most clients depend on:

  • Prefix filtering: ?prefix=subdir/ returns only keys starting with subdir/
  • Delimiter folding: ?delimiter=/ collapses keys into <CommonPrefixes> virtual directories
  • Pagination: ?max-keys=N plus ?continuation-token=… for cursor-based paging
  • URL-encoded keys: ?encoding-type=url returns keys URL-encoded. Keys containing control characters can’t be represented in XML, so a listing that includes one without encoding-type=url fails with 400 InvalidArgument
  • Object size: <Size> is populated from indexed metadata
  • MD5 ETag in listings: listing entries return the MD5 ETag, so checksum verification does not require an extra HeadObject per object

Auto Drive implements ListObjectsV2 only. The legacy ListObjects (V1) API is not implemented: a V1 request is read as a GetObject and fails with 404 NoSuchKey.

import { ListObjectsV2Command } from "@aws-sdk/client-s3";
 
const result = await s3Client.send(
  new ListObjectsV2Command({
    Bucket: "my-archive",
    Prefix: "logs/",
    Delimiter: "/",
    MaxKeys: 100,
  })
);
 
console.log(result.Contents);        // objects directly under "logs/"
console.log(result.CommonPrefixes);  // virtual subdirectories like "logs/2026/"
 
if (result.IsTruncated) {
  // Fetch the next page using result.NextContinuationToken
}

4. MD5 ETags + CID via x-amz-meta-cid

PutObject, GetObject, HeadObject, CopyObject, ListObjectsV2, and CompleteMultipartUpload return a standard quoted MD5 ETag (e.g. "d41d8cd98f00b204e9800998ecf8427e") for objects uploaded after the MD5 ETag feature shipped. For multipart uploads, the composite ETag follows the AWS format "<md5_of_part_md5s>-<N>". Legacy objects (those uploaded before the feature) have no stored MD5: HEAD/GET omit the ETag header entirely, and ListObjectsV2 falls back to wrapping the CID as the ETag value. See the migration note below.

The Autonomys CID is exposed on every object response as a custom header:

x-amz-meta-cid: bafkreig...

This makes rclone check, rclone md5sum, AWS CLI checksum verification, and any other MD5-based tooling work out of the box.

Migration note. Objects uploaded before the MD5 ETag feature shipped have no stored MD5. They will not return an ETag header on HEAD/GET until they are re-uploaded. The CID is always available via x-amz-meta-cid. For listings of legacy objects, the CID is returned as a fallback.

import { HeadObjectCommand } from "@aws-sdk/client-s3";
 
const head = await s3Client.send(
  new HeadObjectCommand({ Bucket: "my-archive", Key: "report.pdf" })
);
 
console.log(head.ETag);                    // '"d41d8cd98f00b204e9800998ecf8427e"' (MD5)
console.log(head.Metadata?.cid);           // 'bafkreig...' (CID, via x-amz-meta-cid)

5. Deleting Objects

DeleteObject hides the object and returns 204 No Content, whether or not the key existed. The data is never removed from the DSN:

  • The key disappears from listings and GetObject/HeadObject return 404 NoSuchKey.
  • The response carries x-amz-delete-marker: true, as on a versioned S3 bucket, and the key’s earlier versions are kept.
  • If no other key in your account points at the same content, the object also moves to the Trash in the web app at ai3.storage. Restoring it there makes the key visible again.

What is not supported:

  • Destroying a version. DeleteObject with a VersionId is refused with 403 AccessDenied, because versions are permanent.
  • Multi-object delete. DeleteObjects (POST /{bucket}?delete) returns 501 NotImplemented. Delete keys one at a time with DeleteObject.
import { DeleteObjectCommand } from "@aws-sdk/client-s3";
 
const result = await s3Client.send(
  new DeleteObjectCommand({ Bucket: "my-archive", Key: "draft.pdf" })
);
console.log(result.DeleteMarker); // true: the key is hidden, its data stays on the DSN

6. Server-side Copy

CopyObject copies an object to a new key in a single request. Content-addressed storage makes this cheap: the new key points at the existing content, so no data is transferred and no upload credits are used.

  • Metadata: with the default x-amz-metadata-directive: COPY, the destination keeps the source’s metadata. With REPLACE, it gets exactly the metadata sent with the copy request.
  • Source version: a copy always uses the source key’s current version. A versionId in x-amz-copy-source is ignored. To restore an earlier version, download it with GetObject and upload it again.
  • Multipart copy (UploadPartCopy) is not supported and returns 501 NotImplemented. Copy objects of any size with a single CopyObject request instead.
import { CopyObjectCommand } from "@aws-sdk/client-s3";
 
await s3Client.send(
  new CopyObjectCommand({
    Bucket: "my-archive",
    Key: "final/report.pdf",
    CopySource: "my-archive/drafts/report.pdf",
  })
);

7. Versioning

Every version of every key is kept, so versioning is always on:

  • GetBucketVersioning always reports Enabled. PutBucketVersioning returns 501 NotImplemented, because versioning cannot be turned off.
  • Each version’s VersionId is its content CID. PutObject, CopyObject, and CompleteMultipartUpload return it in x-amz-version-id.
  • GetObject and HeadObject accept a VersionId to read an earlier version. The version that was current when a key was deleted can’t be read this way; restore the object from the Trash instead.
  • ListObjectVersions (GET /{bucket}?versions) lists each key’s versions, newest first, with a delete marker for keys that have been deleted. MaxKeys limits the number of keys rather than versions, and every version of a key is returned on the same page.

Version history starts from when Auto Drive began recording versions, so objects last written before then show only their current version.

import { GetObjectCommand, ListObjectVersionsCommand } from "@aws-sdk/client-s3";
 
const { Versions } = await s3Client.send(
  new ListObjectVersionsCommand({ Bucket: "my-archive", Prefix: "report.pdf" })
);
 
const previous = Versions?.find((v) => !v.IsLatest);
const old = await s3Client.send(
  new GetObjectCommand({
    Bucket: "my-archive",
    Key: "report.pdf",
    VersionId: previous?.VersionId, // the CID of that version
  })
);

8. Object Lock

Auto Drive reports its permanence in S3’s own terms, as Object Lock in COMPLIANCE mode. Clients can read the lock but cannot configure it:

OperationResponse
GetObjectLockConfigurationObjectLockEnabled: Enabled, default retention COMPLIANCE for 100 years
GetObjectRetentionCOMPLIANCE, retained until 100 years after the object was written
GetObjectLegalHoldOFF (Auto Drive has no legal holds)
PutObjectLockConfiguration, PutObjectRetention, PutObjectLegalHold501 NotImplemented

As on any S3 bucket with COMPLIANCE-mode Object Lock, a plain DeleteObject still succeeds by hiding the key behind a delete marker, while destroying a locked version is refused.

9. Metadata

Auto Drive stores the metadata sent with an upload and returns it unchanged on GetObject and HeadObject:

  • Content-Type exactly as uploaded, with no charset added
  • Cache-Control, Content-Language, Content-Disposition and Content-Encoding
  • User metadata (x-amz-meta-*), up to the S3 limit of 2 KB in total. A larger write fails with 400 MetadataTooLarge
  • Modification time sent as x-amz-meta-mtime, as rclone does, echoed back exactly

Objects stored with Content-Encoding: gzip are returned as the stored bytes with the header, never decompressed. Auto Drive also recognizes compression and encryption user metadata for content stored with those transforms applied client-side.

const command = new PutObjectCommand({
  Bucket: "my-archive",
  Key: "page.html",
  Body: buffer,
  ContentType: "text/html",
  CacheControl: "max-age=3600",
  Metadata: {
    project: "website",
  },
});

10. Range Requests

Partial file downloads using standard HTTP Range headers are supported, including suffix ranges such as bytes=-500 (the last 500 bytes). A range that starts past the end of the object returns 416 InvalidRange. A request for several ranges at once returns the whole object, as on AWS S3.

const command = new GetObjectCommand({
  Bucket: "my-archive",
  Key: "large-file.bin",
  Range: "bytes=0-9", // first 10 bytes
});

11. Multipart Uploads

The full multipart workflow (CreateMultipartUpload → UploadPart × N → CompleteMultipartUpload) is supported, and AbortMultipartUpload cancels an upload. Part numbers run from 1 to 10,000, and each part can be up to 200 MiB. UploadPart returns the MD5 of the part body; CompleteMultipartUpload returns the standard AWS composite ETag "<md5_of_part_md5s>-<N>".

ListParts and ListMultipartUploads are not implemented and return 501 NotImplemented, so keep track of your own upload IDs.

const key = "large-file.bin";
const Bucket = "my-archive";
 
// Step 1: Create multipart upload
const { UploadId } = await s3Client.send(
  new CreateMultipartUploadCommand({ Bucket, Key: key })
);
 
// Step 2: Upload one or more parts
const part1 = await s3Client.send(
  new UploadPartCommand({
    Bucket,
    Key: key,
    UploadId: UploadId!,
    PartNumber: 1,
    Body: fileChunk1,
  })
);
 
// Step 3: Complete the upload
const result = await s3Client.send(
  new CompleteMultipartUploadCommand({
    Bucket,
    Key: key,
    UploadId: UploadId!,
    MultipartUpload: {
      Parts: [{ ETag: part1.ETag!, PartNumber: 1 }],
    },
  })
);
 
console.log(result.ETag); // '"<composite_md5>-1"'

Limits and Credits

LimitBehaviour
Request sizeA single request body (PutObject or one UploadPart) can be up to 200 MiB. A larger one fails with 413 EntityTooLarge. Use multipart upload for bigger files
UploadsEvery upload uses upload credits from your account. Copies within Auto Drive do not
ReadsReading an object of 100 MiB or less is free. Reading a larger object uses download credits from your free tier for the bytes requested; a range request uses only the length of the range. Purchased credits don’t add download credits. HeadObject and listings are free
Insufficient creditsThe request fails with 402 PaymentRequired before any data is sent
Rate limitsClients that send too many requests at once get 503. AWS SDKs and rclone retry 503 automatically with backoff
Listing page sizeMaxKeys is capped at 1,000

Two other cases also return a retryable 503: an object read moments after it was uploaded can be briefly unavailable (the response includes Retry-After), and a repeated CompleteMultipartUpload for an upload that is still completing returns 503 SlowDown.

Configuration

Endpoint

EnvironmentEndpoint
Mainnet (public)https://s3.auto-drive.autonomys.xyz

Use the s3. endpoint. The older https://public.auto-drive.autonomys.xyz/s3 path still works for existing clients, but it runs through a CDN that can alter responses: it may decompress objects stored with Content-Encoding: gzip, change HTML content, and limit upload size. https://s3.auto-drive.autonomys.xyz returns stored bytes and headers exactly.

Client Setup (AWS JS SDK)

The Auto Drive backend exposes objects under a single /:key(*) route and parses bucket and key from the path, so use path-style requests:

import { S3Client } from "@aws-sdk/client-s3";
 
const s3Client = new S3Client({
  region: "us-east-1",                                // required by the SDK; ignored by Auto Drive
  endpoint: "https://s3.auto-drive.autonomys.xyz",
  credentials: {
    accessKeyId: "your-auto-drive-api-key",          // your Auto Drive API key
    secretAccessKey: "placeholder",                   // any non-empty string; ignored
  },
  forcePathStyle: true,                               // bucket lives in the path
});

Then use bucket names as you would with AWS S3:

await s3Client.send(
  new PutObjectCommand({ Bucket: "my-archive", Key: "report.pdf", Body: buffer })
);

Send request bodies unencoded with a known length, such as a Buffer. Requests that use aws-chunked encoding (chunked SigV4 payload signing) are refused with 501 NotImplemented.

Authentication

  • Uses Auto Drive API key-based authentication (the same keys as the native Auto Drive API)
  • The API key goes in accessKeyId; secretAccessKey must be present but is ignored (use any non-empty placeholder)
  • Every request must carry an Authorization header. Presigned URLs are not supported and return 403 AccessDenied; to share a file, use its gateway link, https://gateway.autonomys.xyz/file/<CID>
  • An unknown or revoked API key returns 403 InvalidAccessKeyId

Get an API key from the Developers section at ai3.storage.

File Ownership & Cross-API Access

  • Per-account keys: your buckets and keys are visible only to your account
  • Cross-API access: files uploaded via the S3 API are owned by your account and appear in the Auto Drive web app and native API. Files uploaded through the web app or the native API have no S3 key, so they don’t appear in S3 listings
  • One Trash: deleting the last S3 key that points at some content moves it to the Trash in the web app, and removing a file in the web app hides the S3 keys that point at it
  • Content addressing across accounts: identical content produces the same CID, whichever account uploads it, but each account sees only its own keys

Storage Characteristics

Content Addressing

  • Files are stored using Content Identifiers (CIDs) on the Autonomys DSN
  • The CID is exposed via the x-amz-meta-cid response header on every object response (and as a fallback ETag for legacy objects)
  • Content is immutable: the same content always produces the same CID

Decentralized Backend

  • Files are stored on the Distributed Storage Network of the Autonomys Network
  • Automatic replication and redundancy
  • No single point of failure

rclone Integration

Auto Drive is a named rclone provider. From rclone v1.76.0, choose provider = Autonomys (Autonomys Auto Drive) and rclone sets everything Auto Drive needs, including path-style requests, ListObjectsV2 listings and single-request copies. rclone check, rclone md5sum, virtual directory listings, multipart uploads, deletes, moves and sync all work, so any rclone-driven workflow (archival, scheduled backups, mounts, cloud-to-cloud migration, CI/CD artifact storage) works against Auto Drive.

For the dedicated walkthrough (quickstart, full config reference, every supported command, workflows, and troubleshooting) see the Using rclone guide, or rclone’s own Auto Drive setup guide.

Migrating from AWS S3

For developers moving from traditional AWS S3:

  1. Update the endpoint to https://s3.auto-drive.autonomys.xyz
  2. Change credentials to use your Auto Drive API key as accessKeyId; set secretAccessKey to a placeholder
  3. Set forcePathStyle: true in the S3 client configuration
  4. Drop bucket-level calls (CreateBucket, HeadBucket, DeleteBucket): buckets are created implicitly on first write
  5. Review deletion logic: DeleteObject hides an object but never erases its data, and DeleteObjects is not supported, so delete keys one at a time
  6. Replace presigned URLs with gateway links (https://gateway.autonomys.xyz/file/<CID>)
  7. Read CIDs from x-amz-meta-cid, not from the ETag (the ETag is the standard MD5)
  8. Handle longer response times due to network latency vs. AWS edge
// Before (AWS S3)
const s3Client = new S3Client({
  region: "us-east-1",
  credentials: {
    accessKeyId: "AKIA...",
    secretAccessKey: "abc123...",
  },
});
 
// After (Auto Drive)
const s3Client = new S3Client({
  region: "us-east-1",
  endpoint: "https://s3.auto-drive.autonomys.xyz",
  credentials: {
    accessKeyId: "your-auto-drive-api-key",
    secretAccessKey: "placeholder",
  },
  forcePathStyle: true,
});

Building Your Own S3-Compatible Layer

If you are building your own S3-compatible service on top of Autonomys storage, the Auto SDK ships reusable, framework-agnostic S3 server-side helpers in @autonomys/file-server:

  • buildListResult(rows, prefix, delimiter, maxKeys): ListObjectsV2 delimiter folding into CommonPrefixes plus maxKeys pagination with correct continuation-token placement
  • computeListObjectsDbLimit(maxKeys, delimiter): how many rows to fetch from storage for a single page
  • finalizeListObjects(params, fetchedRows, dbLimit): wraps buildListResult and applies the full-batch truncation override that prevents duplicate CommonPrefixes across page boundaries
  • md5Hex, formatETag, multipartETag: S3 ETag computation, including the AWS composite multipart format
  • Types: S3ObjectListing, ListObjectsParams, ListObjectsResult
import {
  computeListObjectsDbLimit,
  finalizeListObjects,
  multipartETag,
} from "@autonomys/file-server";
 
const dbLimit = computeListObjectsDbLimit(maxKeys, delimiter);
const rows = await fetchSortedObjectsFromStorage(prefix, continuationToken, dbLimit);
 
const result = finalizeListObjects(
  { bucket, prefix, delimiter, maxKeys, continuationToken },
  rows,
  dbLimit,
);
// result is ready to render as ListObjectsV2 XML
 
const completedETag = multipartETag(parts.map((p) => p.etag));

Auto Drive itself uses these helpers, so you get the same delimiter folding, pagination, and composite-ETag behavior the public service is validated against.

Limitations & Considerations

Performance

  • The DSN has higher write latency than traditional S3. Files of 200 MiB or more must use multipart upload, and multipart is worth using for smaller large files too
  • Listings paginate at the server’s configured maxKeys (default 1,000); requested values larger than 1,000 are clamped to the 1,000 hard cap
  • Range requests may have different performance characteristics than AWS S3

Compatibility Notes

  • Implemented: ListBuckets, ListObjectsV2 (with prefix, delimiter, max-keys, continuation-token, encoding-type), ListObjectVersions, PutObject, GetObject (with Range and VersionId), HeadObject, CopyObject, DeleteObject (hides the object), multipart uploads including AbortMultipartUpload, GetBucketVersioning, GetObjectLockConfiguration, GetObjectRetention, GetObjectLegalHold
  • Not implemented (return 501): DeleteObjects (multi-object delete), UploadPartCopy, ListParts, ListMultipartUploads, PutBucketVersioning, PutObjectLockConfiguration, PutObjectRetention, PutObjectLegalHold, browser POST uploads, aws-chunked request bodies
  • Not supported: ListObjects (V1), which is read as a GetObject and returns 404; bucket-level endpoints (CreateBucket, HeadBucket, DeleteBucket), which are read as object requests; presigned URLs, which return 403; lifecycle policies, ACLs and bucket policies
  • Refused by design (403): DeleteObject with a VersionId, because versions are permanent

Best Practices

  1. Treat storage as permanent: deleting hides an object but never erases it, and every overwrite is stored as a new version that uses credits
  2. Use multipart uploads for large files; requests over 200 MiB are refused
  3. Read CIDs from x-amz-meta-cid, not from the ETag
  4. Use the MD5 ETag for integrity verification: it matches md5sum of the content for non-multipart objects
  5. Paginate listings with MaxKeys + NextContinuationToken rather than fetching everything in one call
  6. Rely on your SDK’s retries for 503 responses and occasional network-induced timeouts
  7. For rclone, use rclone v1.76.0 or later with provider = Autonomys; see the Using rclone guide

Error Handling

Errors use the standard S3 XML format, so SDKs surface them as typed errors.

StatusCodeWhen
400InvalidArgumentA listing contains keys that need encoding-type=url, or a request parameter is invalid
400MetadataTooLargeUser metadata exceeds 2 KB
402PaymentRequiredNot enough upload or download credits
403InvalidAccessKeyIdThe API key is unknown or revoked
403AccessDeniedNo Authorization header (for example a presigned URL), or an attempt to destroy a version
404NoSuchKey / NoSuchUploadThe key, version or multipart upload does not exist
413EntityTooLargeA request body is over 200 MiB
416InvalidRangeThe requested range starts past the end of the object
501NotImplementedThe operation is not supported
503ServiceUnavailable / SlowDownRate-limited or briefly unavailable; retry with backoff

Network timeouts may be longer than traditional S3; configure your SDK accordingly.

This S3 layer provides a familiar interface while leveraging the benefits of decentralized storage, making it easy to migrate existing S3-based applications, and S3-compatible tooling like rclone and the AWS CLI, to Auto Drive.