Using the Object Store on Jetstream2¶
Info
The object store is only available via Horizon and the CLI presently.
The Jetstream2 object store utilizes OpenStack Swift and is S3 compatible. You can utilize it via Horizon or the command line interface (CLI). From the CLI, you can use the OpenStack CLI’s container and object commands, the AWS CLI’s s3api, or other compatible tools.
Trying the object store from the CLI using OpenStack¶
To use the OpenStack CLI with the object store, install the base OpenStack client if you have not already done so:
pip install python-openstackclient
Once the OpenStack CLI is installed and your credentials are loaded, you can test object storage access by creating a container:
openstack container create my-unique-bucket-name
which will create a storage container called “my-unique-bucket-name”. You can then list your buckets by doing:
openstack container list
You can upload an object to that container with:
openstack object create my-unique-bucket-name example.txt
To list the objects in a container, run:
openstack object list my-unique-bucket-name
If you want to delete the test bucket, you can do:
openstack container delete --recursive my-unique-bucket-name
As with all Openstack clients, you can see the full list of commands with
openstack container --help
and
openstack object --help
You can also add, remove, and otherwise work with swift containers (buckets in the S3 vernacular) in Horizon on the Project → Object Store → Containers tab.
Sharing objects with temporary URLs¶
A temporary URL (or “temp URL”) is a signed, time-limited link that lets anyone download (GET) or upload (PUT) a single object without OpenStack credentials. The container stays private, and the link stops working when it expires. This is useful for sharing a dataset or result file with a collaborator who does not have access to your allocation.
Warning
Anyone who has the URL can use it until it expires. Treat it like a password and choose the shortest expiration that meets your needs.
Temp URLs are generated from the CLI; Horizon cannot create them. Expired URLs are rejected, as are URLs whose signature or method does not match.
One-time setup: set a temp URL key¶
Temp URLs are signed with a secret key stored on your project’s object store account. Generate a random key and set it:
openstack object store account set --property Temp-URL-Key=$(openssl rand -hex 32)
You can confirm it was set with:
openstack object store account show
Generate a temp URL¶
Install the Swift client if you have not already done so:
pip install python-swiftclient
Then create a URL. This example allows downloading example.txt from my-unique-bucket-name for 24 hours (86400 seconds):
swift tempurl GET 86400 /swift/v1/my-unique-bucket-name/example.txt <temp-url-key>
Replace <temp-url-key> with the key you set above. The command prints a signed path; prepend https://js2.jetstream-cloud.org:8001 to build the full URL:
https://js2.jetstream-cloud.org:8001/swift/v1/my-unique-bucket-name/example.txt?temp_url_sig=...&temp_url_expires=...
Note
The path you sign must match the path in the final URL exactly. On Jetstream2 it does not include your project ID (no AUTH_... segment).
To allow someone to upload an object instead, use PUT in place of GET. A URL is only valid for the method it was signed for.
Use the temp URL¶
Anyone with the full URL can download the object without credentials:
curl -O "<full-temp-url>"
Revoking temp URLs¶
Every temp URL signed with a key stops working as soon as that key is changed or removed. To revoke all outstanding URLs, set a new Temp-URL-Key. To rotate keys without breaking existing links, set a second key with Temp-URL-Key-2, generate new URLs with it, then replace the first key.