Storage configs

The storage config resource describes where the API can store the data belonging to results of polls. All locations specified in storage configs must be a cloud storage bucket in either Google Cloud Storage or Amazon S3 (AWS).

Attributes

Types

gs : the Google Cloud Storage storage config type. Requires Service Account credentials.

s3 : the Amazon S3 (AWS) storage config type. Requires AWS user credentials.

URLs

The url attribute follows the format:

<type>://<bucket name>/<path within bucket>

For example, given we have a bucket in Google Cloud Storage with the name 'my-bucket' and we want the API to publish to a folder within this bucket called 'results' the url would look like:

gs://my-bucket/results

Credentials

Google Cloud Storage credentials

The API expects the credentials attribute to be a dictionary with the format:

{
    "type": "service_account",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "client_id": "<Google Cloud Platform client ID>",
    "token_uri": "https://oauth2.googleapis.com/token",
    "project_id": "<Google Cloud Platform project ID>",
    "private_key": "<Service Account private key>",
    "client_email": "<Service Account ID>.iam.gserviceaccount.com",
    "private_key_id": "<Service Account private key ID>",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<Service Account ID>.iam.gserviceaccount.com",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs"
}

This matches exactly the output JSON file when creating a Service Account key through the Google Cloud Platform console.

To simplify setting up a storage config using the exported JSON file credentials, ricloud-py includes a simple cli command:

ricloud storage-config create --url "gs://<bucket name>" --credentials <path to JSON file>

Amazon S3 credentials

The API expected the credentials attribute to be a dictionary with the format:

{
    "user_name": "<aws user name>",
    "access_key_id": "<aws user access key ID>",
    "secret_access_key": "<aws user secret access key>",
}

In the credentials payload above, the user_name parameter corresponds to the username of the AWS IAM user the access key belongs to. This can be helpful to include for administrative and auditing purposes.

This can either be constructed manually from the details shown in the AWS console, or from the exportable CSV file through ricloud-py's cli command:

ricloud storage-config create --url "s3://<bucket name>" --credentials <path to CSV file>

Signed URLs

Signed URLs are time-limited, pre-authorised links to access an item in a cloud storage bucket. They let implementations give secure access to a specific item that the client would otherwise not have credentials to access, or no credentials at all.

When signed URLs are enabled on a storage config, any result published using that config will have the signed_url attribute populate. See result attributes for more details.

Signed URLs are supported by both Google Cloud Storage (link to GCS signed URLs docs) and AWS S3 (link to S3 docs).

One additional requirement for signed URLs is that the user associated with the storage configuration credentials must have read permissions on objects in the bucket. In GCS this is the "Storage Object Viewer" role. In S3 is the "s3:GetObject" policy action.

States

new : has just been created or recently updated but not yet tested.

valid : has passed validation and is ready to be used. An organisation must have at least one storage config in this state in order to be useable.

invalid : has failed the validation test. Must be updated or retested.

deactivated : has been turned off by the owning organisation.

Changelog

2020-02-27

• Adds support for signed URLs. This adds the signed_urls_enabled and signed_urls_expiry on the storage config object to toggle generation of signed URLs for results and how long the signed URLs should be active.

Create POST /configs/storage

When a new storage config is created the API will attempt to test it by creating a storage_config.test task. This task will simply publish a small test payload using the config, which can be safely deleted afterwards. If the operation succeeds, the config's state is set to valid and it becomes useable to use with polls. In case of failure, the config's state will be set to invalid, but this can be re-tested easily using the test action.

The first valid storage config within an organisation will automatically be set as the organisation's default.

Note that the creation of storage configs is deduplicated on the url attribute. Therefore, it is not possible to create two storage configs with the same url but different credentials.

Parameters

Using cURL

Here, the content of the credentials dictionary comes from copying the contents of the JSON credentials file generated during the Google Cloud Service Account setup.

curl https://ricloud-api.reincubate.com/configs/storage \
  -X POST \
  -H 'Authorization: Token <your key_token>' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "gs://my-storage-bucket",
  "credentials": {
      "type": "service_account",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "client_id": "<Google Cloud Platform client ID>",
      "token_uri": "https://oauth2.googleapis.com/token",
      "project_id": "<Google Cloud Platform project ID>",
      "private_key": "<Service Account private key>",
      "client_email": "<Service Account ID>.iam.gserviceaccount.com",
      "private_key_id": "<Service Account private key ID>",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<Service Account ID>.iam.gserviceaccount.com",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs"
  }
}'

Using ricloud-py

import json
import ricloud

with open("<path to JSON file>", 'r') as credentials_file:
    credentials = json.load(credentials_file)

storage_config = ricloud.StorageConfig.create(
    url='gs://my-storage-bucket',
    credentials=credentials
)

Retrieve GET /configs/storage/{storage_config ID}

Using cURL

curl https://ricloud-api.reincubate.com/configs/storage/<storage_config ID> \
  -H 'Authorization: Token <your key_token>'

Using ricloud-py

import ricloud

storage_config = ricloud.StorageConfig.retrieve(<storage_config ID>)

List GET /configs/storage

Storage configs in the deactivated state are hidden by default and can only be listed by explicit filtering.

Parameters

Using cURL

curl https://ricloud-api.reincubate.com/configs/storage \
  -H 'Authorization: Token <your key_token>'

Using ricloud-py

import ricloud

storage_configs = ricloud.StorageConfig.list()

Update POST /configs/storage/{storage_config ID}

This action creates a storage_config.test task in order to check the validity of any changes made to the configuration.

Parameters

Using cURL

curl https://ricloud-api.reincubate.com/configs/storage/<storage_config ID> \
  -X POST \
  -H 'Authorization: Token <your key_token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "gs://my-storage-bucket/subfolder"
}'

Using ricloud-py

import ricloud

storage_config = ricloud.StorageConfig.update_with_id(
  <storage_config ID>, url="gs://my-storage-bucket/subfolder"
)

# OR

storage_config = ricloud.StorageConfig.retrieve(<storage_config ID>)

storage_config.update(url='gs://my-storage-bucket/subfolder')

Delete DELETE /configs/storage/{storage_config ID}

Deletes the storage config from the API entirely. This removes any related credentials information stored in the API.

Using cURL

curl https://ricloud-api.reincubate.com/configs/storage/<storage_config ID> \
  -X DELETE \
  -H 'Authorization: Token <your key_token>'

Using ricloud-py

Since ricloud-py v3.0.0.

import ricloud

ricloud.StorageConfig.delete_with_id(<storage_config ID>)

# OR

storage_config = ricloud.StorageConfig.retrieve(<storage_config ID>)

storage_config.delete()

Test POST /configs/storage/{storage_config ID}/test

This action allows you to re-test the validity of a storage config. The state of the config will be set to valid or invalid depending on the outcome of the test.

Note that this action returns a task resource.

Using cURL

curl https://ricloud-api.reincubate.com/configs/storage/<storage_config ID>/test \
  -H 'Authorization: Token <your key_token>'

Using ricloud-py

import ricloud

test_task = ricloud.StorageConfig.test_with_id(<storage_config ID>)

# OR

storage_config = ricloud.StorageConfig.retrieve(<storage_config ID>)

test_task = storage_config.test()