Auto Drive S3 Layer Guide
Overview
Auto Drive provides an S3-compatible API layer that allows you to interact with decentralized storage (DSN) using standard Amazon Web Services Simple Storage Service (AWS S3) SDK commands. This bridges the gap between familiar cloud storage patterns and next-generation decentralized infrastructure, giving developers the best of both worlds: the reliability and developer experience of S3 APIs with the permanence and censorship-resistance of decentralized storage.
For those unfamiliar, Amazon Web Services Simple Storage Service (AWS S3) is an industry-standard object storage service that powers much of the modern web’s file storage needs. Auto Drive maintains compatibility with the most-used parts of S3’s APIs while storing your data on a decentralized network instead of centralized servers.
Permanent storage, versioned keys. Content stored on the Autonomys DSN is permanent: nothing is ever erased. Your S3 keys behave like a versioned S3 bucket on top of it.
DeleteObjecthides an object and returns204, but its data stays on the DSN. Uploading to an existing key adds a new version, and earlier versions are kept. Only destroying a version is refused (403).
How It Works
Auto Drive maps S3 (bucket, key) pairs in your account to Content Identifiers (CIDs) and MD5 checksums. When you upload via the S3 API, the system:
- Stores the file content on the decentralized network (DSN)
- Computes the MD5 of the content and records the
(bucket, key) → (cid, md5)mapping for your account - Returns the MD5 as the
ETag(standard S3 format) and exposes the CID in thex-amz-meta-cidresponse header - Makes the file available in your Auto Drive account, in the web app and through the native Auto Drive API
Key Features
1. Standard S3 SDK and CLI Compatibility
- Works with the official AWS S3 SDKs (
@aws-sdk/client-s3,boto3), the AWS CLI, and S3-compatible tools like rclone - Supported operations:
- Objects:
PutObject,GetObject,HeadObject,DeleteObject,CopyObject - Listing:
ListBuckets,ListObjectsV2,ListObjectVersions - Multipart uploads:
CreateMultipartUpload,UploadPart,CompleteMultipartUpload,AbortMultipartUpload - Versioning and Object Lock (read-only):
GetBucketVersioning,GetObjectLockConfiguration,GetObjectRetention,GetObjectLegalHold
- Objects:
- No code changes required for existing S3 applications other than swapping the endpoint and credentials, and removing bucket-creation calls
2. Buckets
Buckets behave like standard S3 buckets. The first path segment of the object key is the bucket name, and the remainder is the key:
| S3 path | Bucket | Key |
|---|---|---|
my-archive/report.pdf | my-archive | report.pdf |
my-archive/sub/file.txt | my-archive | sub/file.txt |
test.txt (no slash) | default | test.txt |
- Buckets are created implicitly by the first upload into them, and disappear from
ListBucketsonce they have no visible objects. - Buckets and keys are private to your account.
ListBuckets(GET /) returns only your own buckets, and two accounts can use the same bucket and key names without seeing each other’s objects. - There are no bucket-level endpoints. Auto Drive reads a request to a bare bucket path as a request for an object of that name in the
defaultbucket.CreateBucketwould store an object named after the bucket, andDeleteBucketwould hide such an object if one exists, so don’t sendCreateBucket,HeadBucketorDeleteBucket.
import { ListBucketsCommand } from "@aws-sdk/client-s3";
const result = await s3Client.send(new ListBucketsCommand({}));
console.log(result.Buckets); // [{ Name: "my-archive", CreationDate: ... }, ...]Single-segment keys uploaded before bucket support was introduced remain accessible under the default bucket.
3. ListObjectsV2 with Prefix, Delimiter, and Pagination
Auto Drive implements ListObjectsV2 end-to-end, with the parameters most clients depend on:
- Prefix filtering:
?prefix=subdir/returns only keys starting withsubdir/ - Delimiter folding:
?delimiter=/collapses keys into<CommonPrefixes>virtual directories - Pagination:
?max-keys=Nplus?continuation-token=…for cursor-based paging - URL-encoded keys:
?encoding-type=urlreturns keys URL-encoded. Keys containing control characters can’t be represented in XML, so a listing that includes one withoutencoding-type=urlfails with400 InvalidArgument - Object size:
<Size>is populated from indexed metadata - MD5 ETag in listings: listing entries return the MD5 ETag, so checksum verification does not require an extra
HeadObjectper object
Auto Drive implements
ListObjectsV2only. The legacyListObjects(V1) API is not implemented: a V1 request is read as aGetObjectand fails with404 NoSuchKey.
import { ListObjectsV2Command } from "@aws-sdk/client-s3";
const result = await s3Client.send(
new ListObjectsV2Command({
Bucket: "my-archive",
Prefix: "logs/",
Delimiter: "/",
MaxKeys: 100,
})
);
console.log(result.Contents); // objects directly under "logs/"
console.log(result.CommonPrefixes); // virtual subdirectories like "logs/2026/"
if (result.IsTruncated) {
// Fetch the next page using result.NextContinuationToken
}4. MD5 ETags + CID via x-amz-meta-cid
PutObject, GetObject, HeadObject, CopyObject, ListObjectsV2, and CompleteMultipartUpload return a standard quoted MD5 ETag (e.g. "d41d8cd98f00b204e9800998ecf8427e") for objects uploaded after the MD5 ETag feature shipped. For multipart uploads, the composite ETag follows the AWS format "<md5_of_part_md5s>-<N>". Legacy objects (those uploaded before the feature) have no stored MD5: HEAD/GET omit the ETag header entirely, and ListObjectsV2 falls back to wrapping the CID as the ETag value. See the migration note below.
The Autonomys CID is exposed on every object response as a custom header:
x-amz-meta-cid: bafkreig...This makes rclone check, rclone md5sum, AWS CLI checksum verification, and any other MD5-based tooling work out of the box.
Migration note. Objects uploaded before the MD5 ETag feature shipped have no stored MD5. They will not return an
ETagheader onHEAD/GETuntil they are re-uploaded. The CID is always available viax-amz-meta-cid. For listings of legacy objects, the CID is returned as a fallback.
import { HeadObjectCommand } from "@aws-sdk/client-s3";
const head = await s3Client.send(
new HeadObjectCommand({ Bucket: "my-archive", Key: "report.pdf" })
);
console.log(head.ETag); // '"d41d8cd98f00b204e9800998ecf8427e"' (MD5)
console.log(head.Metadata?.cid); // 'bafkreig...' (CID, via x-amz-meta-cid)5. Deleting Objects
DeleteObject hides the object and returns 204 No Content, whether or not the key existed. The data is never removed from the DSN:
- The key disappears from listings and
GetObject/HeadObjectreturn404 NoSuchKey. - The response carries
x-amz-delete-marker: true, as on a versioned S3 bucket, and the key’s earlier versions are kept. - If no other key in your account points at the same content, the object also moves to the Trash in the web app at ai3.storage. Restoring it there makes the key visible again.
What is not supported:
- Destroying a version.
DeleteObjectwith aVersionIdis refused with403 AccessDenied, because versions are permanent. - Multi-object delete.
DeleteObjects(POST /{bucket}?delete) returns501 NotImplemented. Delete keys one at a time withDeleteObject.
import { DeleteObjectCommand } from "@aws-sdk/client-s3";
const result = await s3Client.send(
new DeleteObjectCommand({ Bucket: "my-archive", Key: "draft.pdf" })
);
console.log(result.DeleteMarker); // true: the key is hidden, its data stays on the DSN6. Server-side Copy
CopyObject copies an object to a new key in a single request. Content-addressed storage makes this cheap: the new key points at the existing content, so no data is transferred and no upload credits are used.
- Metadata: with the default
x-amz-metadata-directive: COPY, the destination keeps the source’s metadata. WithREPLACE, it gets exactly the metadata sent with the copy request. - Source version: a copy always uses the source key’s current version. A
versionIdinx-amz-copy-sourceis ignored. To restore an earlier version, download it withGetObjectand upload it again. - Multipart copy (
UploadPartCopy) is not supported and returns501 NotImplemented. Copy objects of any size with a singleCopyObjectrequest instead.
import { CopyObjectCommand } from "@aws-sdk/client-s3";
await s3Client.send(
new CopyObjectCommand({
Bucket: "my-archive",
Key: "final/report.pdf",
CopySource: "my-archive/drafts/report.pdf",
})
);7. Versioning
Every version of every key is kept, so versioning is always on:
GetBucketVersioningalways reportsEnabled.PutBucketVersioningreturns501 NotImplemented, because versioning cannot be turned off.- Each version’s
VersionIdis its content CID.PutObject,CopyObject, andCompleteMultipartUploadreturn it inx-amz-version-id. GetObjectandHeadObjectaccept aVersionIdto read an earlier version. The version that was current when a key was deleted can’t be read this way; restore the object from the Trash instead.ListObjectVersions(GET /{bucket}?versions) lists each key’s versions, newest first, with a delete marker for keys that have been deleted.MaxKeyslimits the number of keys rather than versions, and every version of a key is returned on the same page.
Version history starts from when Auto Drive began recording versions, so objects last written before then show only their current version.
import { GetObjectCommand, ListObjectVersionsCommand } from "@aws-sdk/client-s3";
const { Versions } = await s3Client.send(
new ListObjectVersionsCommand({ Bucket: "my-archive", Prefix: "report.pdf" })
);
const previous = Versions?.find((v) => !v.IsLatest);
const old = await s3Client.send(
new GetObjectCommand({
Bucket: "my-archive",
Key: "report.pdf",
VersionId: previous?.VersionId, // the CID of that version
})
);8. Object Lock
Auto Drive reports its permanence in S3’s own terms, as Object Lock in COMPLIANCE mode. Clients can read the lock but cannot configure it:
| Operation | Response |
|---|---|
GetObjectLockConfiguration | ObjectLockEnabled: Enabled, default retention COMPLIANCE for 100 years |
GetObjectRetention | COMPLIANCE, retained until 100 years after the object was written |
GetObjectLegalHold | OFF (Auto Drive has no legal holds) |
PutObjectLockConfiguration, PutObjectRetention, PutObjectLegalHold | 501 NotImplemented |
As on any S3 bucket with COMPLIANCE-mode Object Lock, a plain DeleteObject still succeeds by hiding the key behind a delete marker, while destroying a locked version is refused.
9. Metadata
Auto Drive stores the metadata sent with an upload and returns it unchanged on GetObject and HeadObject:
Content-Typeexactly as uploaded, with no charset addedCache-Control,Content-Language,Content-DispositionandContent-Encoding- User metadata (
x-amz-meta-*), up to the S3 limit of 2 KB in total. A larger write fails with400 MetadataTooLarge - Modification time sent as
x-amz-meta-mtime, as rclone does, echoed back exactly
Objects stored with Content-Encoding: gzip are returned as the stored bytes with the header, never decompressed. Auto Drive also recognizes compression and encryption user metadata for content stored with those transforms applied client-side.
const command = new PutObjectCommand({
Bucket: "my-archive",
Key: "page.html",
Body: buffer,
ContentType: "text/html",
CacheControl: "max-age=3600",
Metadata: {
project: "website",
},
});10. Range Requests
Partial file downloads using standard HTTP Range headers are supported, including suffix ranges such as bytes=-500 (the last 500 bytes). A range that starts past the end of the object returns 416 InvalidRange. A request for several ranges at once returns the whole object, as on AWS S3.
const command = new GetObjectCommand({
Bucket: "my-archive",
Key: "large-file.bin",
Range: "bytes=0-9", // first 10 bytes
});11. Multipart Uploads
The full multipart workflow (CreateMultipartUpload → UploadPart × N → CompleteMultipartUpload) is supported, and AbortMultipartUpload cancels an upload. Part numbers run from 1 to 10,000, and each part can be up to 200 MiB. UploadPart returns the MD5 of the part body; CompleteMultipartUpload returns the standard AWS composite ETag "<md5_of_part_md5s>-<N>".
ListParts and ListMultipartUploads are not implemented and return 501 NotImplemented, so keep track of your own upload IDs.
const key = "large-file.bin";
const Bucket = "my-archive";
// Step 1: Create multipart upload
const { UploadId } = await s3Client.send(
new CreateMultipartUploadCommand({ Bucket, Key: key })
);
// Step 2: Upload one or more parts
const part1 = await s3Client.send(
new UploadPartCommand({
Bucket,
Key: key,
UploadId: UploadId!,
PartNumber: 1,
Body: fileChunk1,
})
);
// Step 3: Complete the upload
const result = await s3Client.send(
new CompleteMultipartUploadCommand({
Bucket,
Key: key,
UploadId: UploadId!,
MultipartUpload: {
Parts: [{ ETag: part1.ETag!, PartNumber: 1 }],
},
})
);
console.log(result.ETag); // '"<composite_md5>-1"'Limits and Credits
| Limit | Behaviour |
|---|---|
| Request size | A single request body (PutObject or one UploadPart) can be up to 200 MiB. A larger one fails with 413 EntityTooLarge. Use multipart upload for bigger files |
| Uploads | Every upload uses upload credits from your account. Copies within Auto Drive do not |
| Reads | Reading an object of 100 MiB or less is free. Reading a larger object uses download credits from your free tier for the bytes requested; a range request uses only the length of the range. Purchased credits don’t add download credits. HeadObject and listings are free |
| Insufficient credits | The request fails with 402 PaymentRequired before any data is sent |
| Rate limits | Clients that send too many requests at once get 503. AWS SDKs and rclone retry 503 automatically with backoff |
| Listing page size | MaxKeys is capped at 1,000 |
Two other cases also return a retryable 503: an object read moments after it was uploaded can be briefly unavailable (the response includes Retry-After), and a repeated CompleteMultipartUpload for an upload that is still completing returns 503 SlowDown.
Configuration
Endpoint
| Environment | Endpoint |
|---|---|
| Mainnet (public) | https://s3.auto-drive.autonomys.xyz |
Use the
s3.endpoint. The olderhttps://public.auto-drive.autonomys.xyz/s3path still works for existing clients, but it runs through a CDN that can alter responses: it may decompress objects stored withContent-Encoding: gzip, change HTML content, and limit upload size.https://s3.auto-drive.autonomys.xyzreturns stored bytes and headers exactly.
Client Setup (AWS JS SDK)
The Auto Drive backend exposes objects under a single /:key(*) route and parses bucket and key from the path, so use path-style requests:
import { S3Client } from "@aws-sdk/client-s3";
const s3Client = new S3Client({
region: "us-east-1", // required by the SDK; ignored by Auto Drive
endpoint: "https://s3.auto-drive.autonomys.xyz",
credentials: {
accessKeyId: "your-auto-drive-api-key", // your Auto Drive API key
secretAccessKey: "placeholder", // any non-empty string; ignored
},
forcePathStyle: true, // bucket lives in the path
});Then use bucket names as you would with AWS S3:
await s3Client.send(
new PutObjectCommand({ Bucket: "my-archive", Key: "report.pdf", Body: buffer })
);Send request bodies unencoded with a known length, such as a Buffer. Requests that use aws-chunked encoding (chunked SigV4 payload signing) are refused with 501 NotImplemented.
Authentication
- Uses Auto Drive API key-based authentication (the same keys as the native Auto Drive API)
- The API key goes in
accessKeyId;secretAccessKeymust be present but is ignored (use any non-empty placeholder) - Every request must carry an
Authorizationheader. Presigned URLs are not supported and return403 AccessDenied; to share a file, use its gateway link,https://gateway.autonomys.xyz/file/<CID> - An unknown or revoked API key returns
403 InvalidAccessKeyId
Get an API key from the Developers section at ai3.storage.
File Ownership & Cross-API Access
- Per-account keys: your buckets and keys are visible only to your account
- Cross-API access: files uploaded via the S3 API are owned by your account and appear in the Auto Drive web app and native API. Files uploaded through the web app or the native API have no S3 key, so they don’t appear in S3 listings
- One Trash: deleting the last S3 key that points at some content moves it to the Trash in the web app, and removing a file in the web app hides the S3 keys that point at it
- Content addressing across accounts: identical content produces the same CID, whichever account uploads it, but each account sees only its own keys
Storage Characteristics
Content Addressing
- Files are stored using Content Identifiers (CIDs) on the Autonomys DSN
- The CID is exposed via the
x-amz-meta-cidresponse header on every object response (and as a fallback ETag for legacy objects) - Content is immutable: the same content always produces the same CID
Decentralized Backend
- Files are stored on the Distributed Storage Network of the Autonomys Network
- Automatic replication and redundancy
- No single point of failure
rclone Integration
Auto Drive is a named rclone provider. From rclone v1.76.0, choose provider = Autonomys (Autonomys Auto Drive) and rclone sets everything Auto Drive needs, including path-style requests, ListObjectsV2 listings and single-request copies. rclone check, rclone md5sum, virtual directory listings, multipart uploads, deletes, moves and sync all work, so any rclone-driven workflow (archival, scheduled backups, mounts, cloud-to-cloud migration, CI/CD artifact storage) works against Auto Drive.
For the dedicated walkthrough (quickstart, full config reference, every supported command, workflows, and troubleshooting) see the Using rclone guide, or rclone’s own Auto Drive setup guide.
Migrating from AWS S3
For developers moving from traditional AWS S3:
- Update the endpoint to
https://s3.auto-drive.autonomys.xyz - Change credentials to use your Auto Drive API key as
accessKeyId; setsecretAccessKeyto a placeholder - Set
forcePathStyle: truein the S3 client configuration - Drop bucket-level calls (
CreateBucket,HeadBucket,DeleteBucket): buckets are created implicitly on first write - Review deletion logic:
DeleteObjecthides an object but never erases its data, andDeleteObjectsis not supported, so delete keys one at a time - Replace presigned URLs with gateway links (
https://gateway.autonomys.xyz/file/<CID>) - Read CIDs from
x-amz-meta-cid, not from the ETag (the ETag is the standard MD5) - Handle longer response times due to network latency vs. AWS edge
// Before (AWS S3)
const s3Client = new S3Client({
region: "us-east-1",
credentials: {
accessKeyId: "AKIA...",
secretAccessKey: "abc123...",
},
});
// After (Auto Drive)
const s3Client = new S3Client({
region: "us-east-1",
endpoint: "https://s3.auto-drive.autonomys.xyz",
credentials: {
accessKeyId: "your-auto-drive-api-key",
secretAccessKey: "placeholder",
},
forcePathStyle: true,
});Building Your Own S3-Compatible Layer
If you are building your own S3-compatible service on top of Autonomys storage, the Auto SDK ships reusable, framework-agnostic S3 server-side helpers in @autonomys/file-server:
buildListResult(rows, prefix, delimiter, maxKeys):ListObjectsV2delimiter folding intoCommonPrefixesplusmaxKeyspagination with correct continuation-token placementcomputeListObjectsDbLimit(maxKeys, delimiter): how many rows to fetch from storage for a single pagefinalizeListObjects(params, fetchedRows, dbLimit): wrapsbuildListResultand applies the full-batch truncation override that prevents duplicateCommonPrefixesacross page boundariesmd5Hex,formatETag,multipartETag: S3 ETag computation, including the AWS composite multipart format- Types:
S3ObjectListing,ListObjectsParams,ListObjectsResult
import {
computeListObjectsDbLimit,
finalizeListObjects,
multipartETag,
} from "@autonomys/file-server";
const dbLimit = computeListObjectsDbLimit(maxKeys, delimiter);
const rows = await fetchSortedObjectsFromStorage(prefix, continuationToken, dbLimit);
const result = finalizeListObjects(
{ bucket, prefix, delimiter, maxKeys, continuationToken },
rows,
dbLimit,
);
// result is ready to render as ListObjectsV2 XML
const completedETag = multipartETag(parts.map((p) => p.etag));Auto Drive itself uses these helpers, so you get the same delimiter folding, pagination, and composite-ETag behavior the public service is validated against.
Limitations & Considerations
Performance
- The DSN has higher write latency than traditional S3. Files of 200 MiB or more must use multipart upload, and multipart is worth using for smaller large files too
- Listings paginate at the server’s configured
maxKeys(default 1,000); requested values larger than 1,000 are clamped to the 1,000 hard cap - Range requests may have different performance characteristics than AWS S3
Compatibility Notes
- Implemented:
ListBuckets,ListObjectsV2(withprefix,delimiter,max-keys,continuation-token,encoding-type),ListObjectVersions,PutObject,GetObject(withRangeandVersionId),HeadObject,CopyObject,DeleteObject(hides the object), multipart uploads includingAbortMultipartUpload,GetBucketVersioning,GetObjectLockConfiguration,GetObjectRetention,GetObjectLegalHold - Not implemented (return 501):
DeleteObjects(multi-object delete),UploadPartCopy,ListParts,ListMultipartUploads,PutBucketVersioning,PutObjectLockConfiguration,PutObjectRetention,PutObjectLegalHold, browser POST uploads,aws-chunkedrequest bodies - Not supported:
ListObjects(V1), which is read as aGetObjectand returns404; bucket-level endpoints (CreateBucket,HeadBucket,DeleteBucket), which are read as object requests; presigned URLs, which return403; lifecycle policies, ACLs and bucket policies - Refused by design (403):
DeleteObjectwith aVersionId, because versions are permanent
Best Practices
- Treat storage as permanent: deleting hides an object but never erases it, and every overwrite is stored as a new version that uses credits
- Use multipart uploads for large files; requests over 200 MiB are refused
- Read CIDs from
x-amz-meta-cid, not from the ETag - Use the MD5 ETag for integrity verification: it matches
md5sumof the content for non-multipart objects - Paginate listings with
MaxKeys+NextContinuationTokenrather than fetching everything in one call - Rely on your SDK’s retries for
503responses and occasional network-induced timeouts - For rclone, use rclone v1.76.0 or later with
provider = Autonomys; see the Using rclone guide
Error Handling
Errors use the standard S3 XML format, so SDKs surface them as typed errors.
| Status | Code | When |
|---|---|---|
400 | InvalidArgument | A listing contains keys that need encoding-type=url, or a request parameter is invalid |
400 | MetadataTooLarge | User metadata exceeds 2 KB |
402 | PaymentRequired | Not enough upload or download credits |
403 | InvalidAccessKeyId | The API key is unknown or revoked |
403 | AccessDenied | No Authorization header (for example a presigned URL), or an attempt to destroy a version |
404 | NoSuchKey / NoSuchUpload | The key, version or multipart upload does not exist |
413 | EntityTooLarge | A request body is over 200 MiB |
416 | InvalidRange | The requested range starts past the end of the object |
501 | NotImplemented | The operation is not supported |
503 | ServiceUnavailable / SlowDown | Rate-limited or briefly unavailable; retry with backoff |
Network timeouts may be longer than traditional S3; configure your SDK accordingly.
This S3 layer provides a familiar interface while leveraging the benefits of decentralized storage, making it easy to migrate existing S3-based applications, and S3-compatible tooling like rclone and the AWS CLI, to Auto Drive.