Struct opendal::services::S3

source ·

pub struct S3 { /* private fields */ }

Expand description

Aws S3 and compatible services (including minio, digitalocean space and so on) support

Capabilities

This service can be used to:

read
write
list
scan
presign
blocking

Configuration

root: Set the work dir for backend.
bucket: Set the container name for backend.
endpoint: Set the endpoint for backend.
region: Set the region for backend.
access_key_id: Set the access_key_id for backend.
secret_access_key: Set the secret_access_key for backend.
security_token: Set the security_token for backend.
server_side_encryption: Set the server_side_encryption for backend.
server_side_encryption_aws_kms_key_id: Set the server_side_encryption_aws_kms_key_id for backend.
server_side_encryption_customer_algorithm: Set the server_side_encryption_customer_algorithm for backend.
server_side_encryption_customer_key: Set the server_side_encryption_customer_key for backend.
server_side_encryption_customer_key_md5: Set the server_side_encryption_customer_key_md5 for backend.
disable_config_load: Disable aws config load from env
enable_virtual_host_style: Enable virtual host style.

Refer to S3Builder’s public API docs for more information.

Temporary security credentials

OpenDAL now provides support for S3 temporary security credentials in IAM.

The way to take advantage of this feature is to build your S3 backend with Builder::security_token.

But OpenDAL will not refresh the temporary security credentials, please keep in mind to refresh those credentials in time.

Server Side Encryption

OpenDAL provides full support of S3 Server Side Encryption(SSE) features.

The easiest way to configure them is to use helper functions like

SSE-KMS: server_side_encryption_with_aws_managed_kms_key
SSE-KMS: server_side_encryption_with_customer_managed_kms_key
SSE-S3: server_side_encryption_with_s3_key
SSE-C: server_side_encryption_with_customer_key

If those functions don’t fulfill need, low-level options are also provided:

Use service managed kms key
- server_side_encryption="aws:kms"
Use customer provided kms key
- server_side_encryption="aws:kms"
- server_side_encryption_aws_kms_key_id="your-kms-key"
Use S3 managed key
- server_side_encryption="AES256"
Use customer key
- server_side_encryption_customer_algorithm="AES256"
- server_side_encryption_customer_key="base64-of-your-aes256-key"
- server_side_encryption_customer_key_md5="base64-of-your-aes256-key-md5"

After SSE have been configured, all requests send by this backed will attach those headers.

Reference: Protecting data using server-side encryption

Example

Basic Setup

use std::sync::Arc;

use anyhow::Result;
use opendal::services::S3;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    // Create s3 backend builder.
    let mut builder = S3::default();
    // Set the root for s3, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root("/path/to/dir");
    // Set the bucket name, this is required.
    builder.bucket("test");
    // Set the endpoint.
    //
    // For examples:
    // - "https://s3.amazonaws.com"
    // - "http://127.0.0.1:9000"
    // - "https://oss-ap-northeast-1.aliyuncs.com"
    // - "https://cos.ap-seoul.myqcloud.com"
    //
    // Default to "https://s3.amazonaws.com"
    builder.endpoint("https://s3.amazonaws.com");
    // Set the access_key_id and secret_access_key.
    //
    // OpenDAL will try load credential from the env.
    // If credential not set and no valid credential in env, OpenDAL will
    // send request without signing like anonymous user.
    builder.access_key_id("access_key_id");
    builder.secret_access_key("secret_access_key");

    let op: Operator = Operator::new(builder)?.finish();

    Ok(())
}

S3 with SSE-C

use anyhow::Result;
use log::info;
use opendal::services::S3;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default();

    // Setup builders

    // Enable SSE-C
    builder.server_side_encryption_with_customer_key("AES256", "customer_key".as_bytes());

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

S3 with SSE-KMS and aws managed kms key

use anyhow::Result;
use log::info;
use opendal::services::S3;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default();

    // Setup builders

    // Enable SSE-KMS with aws managed kms key
    builder.server_side_encryption_with_aws_managed_kms_key();

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

S3 with SSE-KMS and customer managed kms key

use anyhow::Result;
use log::info;
use opendal::services::S3;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default();

    // Setup builders

    // Enable SSE-KMS with customer managed kms key
    builder.server_side_encryption_with_customer_managed_kms_key("aws_kms_key_id");

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

S3 with SSE-S3

use anyhow::Result;
use log::info;
use opendal::services::S3;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default();

    // Setup builders

    // Enable SSE-S3
    builder.server_side_encryption_with_s3_key();

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

Compatible Services

AWS S3

AWS S3 is the default implementations of s3 services. Only bucket is required.

builder.bucket("<bucket_name>");

Alibaba Object Storage Service (OSS)

OSS is a s3 compatible service provided by Alibaba Cloud.

To connect to OSS, we need to set:

endpoint: The endpoint of oss, for example: https://oss-cn-hangzhou.aliyuncs.com
bucket: The bucket name of oss.

OSS provide internal endpoint for used at alibabacloud internally, please visit OSS Regions and endpoints for more details.

OSS only supports the virtual host style, users could meet errors like:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
 <Code>SecondLevelDomainForbidden</Code>
 <Message>The bucket you are attempting to access must be addressed using OSS third level domain.</Message>
 <RequestId>62A1C265292C0632377F021F</RequestId>
 <HostId>oss-cn-hangzhou.aliyuncs.com</HostId>
</Error>
In that case, please enable virtual host style for requesting.

builder.endpoint("https://oss-cn-hangzhou.aliyuncs.com");
builder.region("<region>");
builder.bucket("<bucket_name>");
builder.enable_virtual_host_style();

Minio

minio is an open-source s3 compatible services.

To connect to minio, we need to set:

endpoint: The endpoint of minio, for example: http://127.0.0.1:9000
region: The region of minio. If not specified, it could be ignored.
bucket: The bucket name of minio.

builder.endpoint("http://127.0.0.1:9000");
builder.region("<region>");
builder.bucket("<bucket_name>");

QingStor Object Storage

QingStor Object Storage is a S3-compatible service provided by QingCloud.

To connect to QingStor Object Storage, we need to set:

endpoint: The endpoint of QingStor s3 compatible endpoint, for example: https://s3.pek3b.qingstor.com
bucket: The bucket name.

Scaleway Object Storage

Scaleway Object Storage is a S3-compatible and multi-AZ redundant object storage service.

To connect to Scaleway Object Storage, we need to set:

endpoint: The endpoint of scaleway, for example: https://s3.nl-ams.scw.cloud
region: The region of scaleway.
bucket: The bucket name of scaleway.

Tencent Cloud Object Storage (COS)

COS is a s3 compatible service provided by Tencent Cloud.

To connect to COS, we need to set:

endpoint: The endpoint of cos, for example: https://cos.ap-beijing.myqcloud.com
bucket: The bucket name of cos.

Wasabi Object Storage

Wasabi is a s3 compatible service.

Cloud storage pricing that is 80% less than Amazon S3.

To connect to wasabi, we need to set:

endpoint: The endpoint of wasabi, for example: https://s3.us-east-2.wasabisys.com
bucket: The bucket name of wasabi.

Refer to What are the service URLs for Wasabi’s different storage regions? for more details.

Implementations§

source §

impl S3Builder

source

pub fn root(&mut self, root: &str) -> &mut Self

Set root of this backend.

All operations will happen under this root.

source

pub fn bucket(&mut self, bucket: &str) -> &mut Self

Set bucket name of this backend.

source

pub fn endpoint(&mut self, endpoint: &str) -> &mut Self

Set endpoint of this backend.

Endpoint must be full uri, e.g.

AWS S3: https://s3.amazonaws.com or https://s3.{region}.amazonaws.com
Aliyun OSS: https://{region}.aliyuncs.com
Tencent COS: https://cos.{region}.myqcloud.com
Minio: http://127.0.0.1:9000

If user inputs endpoint without scheme like “s3.amazonaws.com”, we will prepend “https://” before it.

source

pub fn region(&mut self, region: &str) -> &mut Self

Region represent the signing region of this endpoint.

If region is set, we will take user’s input first.
If not, We will try to detect region via RFC-0057: Auto Region.

Most of time, region is not need to be set, especially for AWS S3 and minio.

source

pub fn access_key_id(&mut self, v: &str) -> &mut Self

Set access_key_id of this backend.

If access_key_id is set, we will take user’s input first.
If not, we will try to load it from environment.

source

pub fn secret_access_key(&mut self, v: &str) -> &mut Self

Set secret_access_key of this backend.

If secret_access_key is set, we will take user’s input first.
If not, we will try to load it from environment.

source

pub fn role_arn(&mut self, v: &str) -> &mut Self

Set role_arn for this backend.

source

pub fn external_id(&mut self, v: &str) -> &mut Self

Set external_id for this backend.

source

pub fn server_side_encryption(&mut self, v: &str) -> &mut Self

Set server_side_encryption for this backend.

Available values: AES256, aws:kms.

Note

This function is the low-level setting for SSE related features.

SSE related options should be set carefully to make them works. Please use server_side_encryption_with_* helpers if even possible.

source

pub fn server_side_encryption_aws_kms_key_id(&mut self, v: &str) -> &mut Self

Set server_side_encryption_aws_kms_key_id for this backend

If server_side_encryption set to aws:kms, and server_side_encryption_aws_kms_key_id is not set, S3 will use aws managed kms key to encrypt data.
If server_side_encryption set to aws:kms, and server_side_encryption_aws_kms_key_id is a valid kms key id, S3 will use the provided kms key to encrypt data.
If the server_side_encryption_aws_kms_key_id is invalid or not found, an error will be returned.
If server_side_encryption is not aws:kms, setting server_side_encryption_aws_kms_key_id is a noop.

Note

This function is the low-level setting for SSE related features.

SSE related options should be set carefully to make them works. Please use server_side_encryption_with_* helpers if even possible.

source

pub fn server_side_encryption_customer_algorithm(&mut self, v: &str) -> &mut Self

Set server_side_encryption_customer_algorithm for this backend.

Available values: AES256.

Note

This function is the low-level setting for SSE related features.

SSE related options should be set carefully to make them works. Please use server_side_encryption_with_* helpers if even possible.

source

pub fn server_side_encryption_customer_key(&mut self, v: &str) -> &mut Self

Set server_side_encryption_customer_key for this backend.

Args

v: base64 encoded key that matches algorithm specified in server_side_encryption_customer_algorithm.

Note

This function is the low-level setting for SSE related features.

SSE related options should be set carefully to make them works. Please use server_side_encryption_with_* helpers if even possible.

source

pub fn server_side_encryption_customer_key_md5(&mut self, v: &str) -> &mut Self

Set server_side_encryption_customer_key_md5 for this backend.

Args

v: MD5 digest of key specified in server_side_encryption_customer_key.

Note

This function is the low-level setting for SSE related features.

SSE related options should be set carefully to make them works. Please use server_side_encryption_with_* helpers if even possible.

source

pub fn server_side_encryption_with_aws_managed_kms_key(&mut self) -> &mut Self

Enable server side encryption with aws managed kms key

As known as: SSE-KMS

NOTE: This function should not be used along with other server_side_encryption_with_ functions.

source

pub fn server_side_encryption_with_customer_managed_kms_key(
&mut self,
aws_kms_key_id: &str
) -> &mut Self

Enable server side encryption with customer managed kms key

As known as: SSE-KMS

NOTE: This function should not be used along with other server_side_encryption_with_ functions.

source

pub fn server_side_encryption_with_s3_key(&mut self) -> &mut Self

Enable server side encryption with s3 managed key

As known as: SSE-S3

NOTE: This function should not be used along with other server_side_encryption_with_ functions.

source

pub fn server_side_encryption_with_customer_key(
    &mut self,
    algorithm: &str,
    key: &[u8]
) -> &mut Self

Enable server side encryption with customer key.

As known as: SSE-C

NOTE: This function should not be used along with other server_side_encryption_with_ functions.

source

pub fn security_token(&mut self, token: &str) -> &mut Self

Set temporary credential used in AWS S3 connections

Warning

security token’s lifetime is short and requires users to refresh in time.

source

pub fn disable_config_load(&mut self) -> &mut Self

Disable config load so that opendal will not load config from environment.

For examples:

envs like AWS_ACCESS_KEY_ID
files like ~/.aws/config

source

pub fn enable_virtual_host_style(&mut self) -> &mut Self

Enable virtual host style so that opendal will send API requests in virtual host style instead of path style.

By default, opendal will send API to https://s3.us-east-1.amazonaws.com/bucket_name
Enabled, opendal will send API to https://bucket_name.s3.us-east-1.amazonaws.com

source