Configuring Object Storage for AWS S3 Apps¶

The Jetstream2 object store hosts an S3-compatible API at the endpoint https://js2.jetstream-cloud.org:8001. This document will cover how to generate credentials and configure a storage bucket for use with S3-style apps.

There is also a supplemental tutorial from Andrea Zonca on Using the distributed file format Zarr on Jetstream 2 object storage, which leverages S3FS.

Prerequisites¶

EC2 Credentials¶

If the you want to use S3 compatibility, you’ll need to generate EC2 credentials via the OpenStack CLI.

openstack ec2 credentials create

Tip

If you do not have CLI clients installed and an application credential openrc, please see Installing Openstack Clients and Authenticating Against the OpenStack CLI (Logging In).

Make note of the access and secret fields–these will be needed for authentication. If forgotten, they can be retrieved again in the future with

openstack ec2 credentials list

AWS CLI¶

Unfortunately, configuring S3-style settings for buckets is not currently possible through graphical interfaces like Horizon. As such, an API wrapper like the AWS CLI is needed.

After installing the AWS CLI, create a new file to house your EC2 credentials created above:

mkdir -p ~/.aws
touch ~/.aws/credentials

Edit the file to include the following lines, where the values for aws_access_key_id and aws_secret_access_key correspond to the aforementioned OpenStack fields access and secret, respectively:

[default]
region=RegionOne
aws_access_key_id=your_access_here
aws_secret_access_key=your_secret_here

Test the authentication by querying:

aws s3api --endpoint 'https://js2.jetstream-cloud.org:8001' list-buckets

Full reference for aws s3api interface can be found here.

Tip

Remember that calls to aws s3api require specifying the Jetstream2 endpoint with the --endpoint flag, as shown above. One workaround to including this with every command is to set an alias s3api="aws s3api --endpoint 'https://js2.jetstream-cloud.org:8001'".

Bucket Configuration¶

Before they can be used for many common S3-compatible applications, buckets will need to be configured properly. On Jetstream2, bucket names need to be globally unique, meaning that two buckets with the same name cannot coexist, even on different projects.

Access Control List (ACL)¶

Use bucket policies if possible.

Amazon notes that the majority of modern use cases for S3 no longer require the use of ACLs, and instead strongly recommends the use of policies to control access.

Under Amazon S3, every bucket and object has an Access Control List (ACL) attached that defines both who can access that resource and the level of access they are granted.

There are a number of canned ACLs available for common scenarios. For example, to make a bucket private (only allowing the owner to read or write):

aws s3api --endpoint "https://js2.jetstream-cloud.org:8001" put-bucket-acl --acl private --bucket <bucket-name>

User-defined ACLs can also be set. See the output of aws s3api put-bucket-acl help for more information.

Bucket Policies¶

Bucket policies are the preferred method for controlling access to an object storage bucket; Jetstream2 supports a large subset of the Amazon S3 policy language. To apply a policy with the AWS CLI, first define the policy in a new file (for example, touch ./policy.json). Then, run the put-bucket-policy command against that file:

aws s3api --endpoint "https://js2.jetstream-cloud.org:8001" put-bucket-policy --policy file://policy.json --bucket <bucket-name>

Bucket policies are written in JSON. They can be authored by hand or with Amazon’s S3 Bucket Policy generator, and generally include the following components:

Action: What actions this policy will govern. For example, s3:ListBucket determines whether a user can list the contents of a bucket.
Effect: Whether to explicitly Allow or Deny the action.
Resource: What AWS resource is governed by this policy, specified by an ARN. In this case, the resource will always be a Jetstream2 bucket or object whose ARN follows the format arn:aws:s3:::bucket-name or arn:aws:s3:::bucket-name/object respectively.
Principal: To whom this policy applies, usually specified by an ARN. Examples can be found in the AWS docs.
- Note that on Jetstream2, the Amazon account number is replaced by a project ID; the active project’s ID can be found with openstack project show, and is not the same as an ACCESS grant number (e.g. TRA230001). For example, an ARN to reference all users on the project with ID 12345abcde would be arn:aws:iam::12345abcde:root

The following policy example allows anyone to list the objects in bucket-name and read those objects, and allows authenticated users on project 12345abcde to upload objects to the bucket:

{
  "Version": "2012-10-17",
  "Id": "S3Policy1",
  "Statement": [
    {
      "Sid": "BucketAllowPublicRead",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::bucket-name",
        "arn:aws:s3:::bucket-name/*"
      ],
      "Principal": "*"
    },
    {
      "Sid": "BucketAllowSpecificUpload",
      "Action": [
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::bucket-name",
        "arn:aws:s3:::bucket-name/*"
      ],
      "Principal": {"AWS":"arn:aws:iam::12345abcde:root"}
    }
  ]
}

More examples can be found here.

CORS Configuration¶

Cross-Origin Resource Sharing (CORS) allows a user’s browser to access resources hosted outside the domain of the current page/app. In the context of Jetstream2 object storage, it may be desirable for a web page/app to directly communicate from the client to the S3-compatible API, rather than proxying traffic through the original web server or handling S3 API requests on the backend.

Without CORS	With CORS

CORS is configured on a per-bucket basis and specified in JSON through the AWS CLI, similar to bucket policies. Start by creating a new file (e.g. touch ./cors.json)–the following example shows a CORS configuration to allow API requests to this bucket to be used by pages on mycoolsite.com:

{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://mycoolsite.com"],
      "AllowedHeaders": ["*"],
      "AllowedMethods": ["GET", "PUT", "POST", "HEAD"],
      "MaxAgeSeconds": 3000,
      "ExposeHeaders": ["x-amz-server-side-encryption"]
    }
  ]
}

The configuration can then be applied to the bucket with:

aws s3api --endpoint "https://js2.jetstream-cloud.org:8001" put-bucket-cors --policy file://cors.json --bucket <bucket-name>

For more information, see the output of aws s3api put-bucket-cors help.

Format requests properly.

In order to receive an Access-Control-Allow-Origin header from API responses, your app/browser must specify a proper Origin header in its request.