Storage configs

Updated

The storage config resource describes where the API can store the data belonging to results of polls. All locations specified in storage configs must be a cloud storage bucket in either Google Cloud Storage or Amazon S3 (AWS).

Attributes

name type description
id storage config ID Resource identifier.
resource string, always storage_config Resource type specifier.
organisation organisation ID The organisation the resource belongs to.
type string One of: gs, s3. This is a read-only attribute generated from the url.
url string A URL representing the target to which results will be published.
credentials dictionary A dictionary containing the credential information used by the API to authenticate against the storage service.
state string One of: 'new', 'valid', 'invalid', 'deactivated'.
date_created datetime When the resource was created.

Types

gs
the Google Cloud Storage storage config type. Requires Service Account credentials.
s3
the Amazon S3 (AWS) storage config type. Requires AWS user credentials.

URLs

The url attribute follows the format:

<type>://<bucket name>/<path within bucket>

For example, given we have a bucket in Google Cloud Storage with the name 'my-bucket' and we want the API to publish to a folder within this bucket called 'results' the url would look like:

gs://my-bucket/results

Credentials

Google Cloud Storage credentials

The API expects the credentials attribute to be a dictionary with the format:

{
    "type": "service_account",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "client_id": "<Google Cloud Platform client ID>",
    "token_uri": "https://oauth2.googleapis.com/token",
    "project_id": "<Google Cloud Platform project ID>",
    "private_key": "<Service Account private key>",
    "client_email": "<Service Account ID>.iam.gserviceaccount.com",
    "private_key_id": "<Service Account private key ID>",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<Service Account ID>.iam.gserviceaccount.com",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs"
}

This matches exactly the output JSON file when creating a Service Account key through the Google Cloud Platform console.

To simplify setting up a storage config using the exported JSON file credentials, ricloud-py includes a simple cli command:

ricloud storage-config create --url "gs://<bucket name>" --credentials <path to JSON file>

Amazon S3 credentials

The API expected the credentials attribute to be a dictionary with the format:

{
    "user_name": "<aws user name>",
    "access_key_id": "<aws user access key ID>",
    "secret_access_key": "<aws user secret access key>",
}

This can either be constructed manually from the details shown in the AWS console, or from the exportable CSV file through ricloud-py's cli command:

ricloud storage-config create --url "s3://<bucket name>" --credentials <path to CSV file>

States

new
has just been created or recently updated but not yet tested.
valid
has passed validation and is ready to be used. An organisation must have at least one storage config in this state in order to be useable.
invalid
has failed the validation test. Must be updated or retested.
deactivated
has been turned off by the owning organisation.

Create POST /configs/storage

When a new storage config is created the API will attempt to test it by creating a storage_config.test task. This task will simply publish a small test payload using the config, which can be safely deleted afterwards. If the operation succeeds, the config's state is set to valid and it becomes useable to use with polls. In case of failure, the config's state will be set to invalid, but this can be re-tested easily using the test action.

The first valid storage config within an organisation will automatically be set as the organisation's default.

Note that the creation of storage configs is deduplicated on the url attribute. Therefore, it is not possible to create two storage configs with the same url but different credentials.

Parameters

name type description
url required The storage configuration URL for your bucket.
credentials required The storage configuration credentials for your bucket.

Using cURL

Here, the content of the credentials dictionary comes from copying the contents of the JSON credentials file generated during the Google Cloud Service Account setup.

curl https://ricloud-api.reincubate.com/configs/storage \
  -X POST \
  -H 'Authorization: Token <your key_token>' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "gs://my-storage-bucket",
  "credentials": {
      "type": "service_account",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "client_id": "<Google Cloud Platform client ID>",
      "token_uri": "https://oauth2.googleapis.com/token",
      "project_id": "<Google Cloud Platform project ID>",
      "private_key": "<Service Account private key>",
      "client_email": "<Service Account ID>.iam.gserviceaccount.com",
      "private_key_id": "<Service Account private key ID>",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<Service Account ID>.iam.gserviceaccount.com",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs"
  }
}'

Using ricloud-py

import json
import ricloud

with open("<path to JSON file>", 'r') as credentials_file:
    credentials = json.load(credentials_file)

storage_config = ricloud.StorageConfig.create(
    url='gs://my-storage-bucket',
    credentials=credentials
)

Retrieve GET /configs/storage/{storage_config ID}

Using cURL

curl https://ricloud-api.reincubate.com/configs/storage/<storage_config ID> \
  -H 'Authorization: Token <your key_token>'

Using ricloud-py

import ricloud

storage_config = ricloud.StorageConfig.retrieve(<storage_config ID>)

List GET /configs/storage

Storage configs in the deactivated state are hidden by default and can only be listed by explicit filtering.

Parameters

name type description
type string Filter by backing bucket's type.
state string Filter by the configuration's state.
date_created datetime filter Filter by resouce creation date.

Using cURL

curl https://ricloud-api.reincubate.com/configs/storage \
  -H 'Authorization: Token <your key_token>'

Using ricloud-py

import ricloud

storage_configs = ricloud.StorageConfig.list()

Update POST /configs/storage/{storage_config ID}

This action creates a storage_config.test task in order to check the validity of any changes made to the configuration.

Parameters

name type description
url string Update the URL to point to a different bucket or location within the bucket. Note that the type of a bucket cannot be changed through this operation.
credentials dictionary Update the credentials.
state string Only to new or deactivated.

Using cURL

curl https://ricloud-api.reincubate.com/configs/storage/<storage_config ID> \
  -X POST \
  -H 'Authorization: Token <your key_token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "gs://my-storage-bucket/subfolder"
}'

Using ricloud-py

import ricloud

storage_config = ricloud.StorageConfig.update_with_id(
  <storage_config ID>, url="gs://my-storage-bucket/subfolder"
)

# OR

storage_config = ricloud.StorageConfig.retrieve(<storage_config ID>)

storage_config.update(url='gs://my-storage-bucket/subfolder')

Delete DELETE /configs/storage/{storage_config ID}

Deletes the storage config from the API entirely. This removes any related credentials information stored in the API.

Using cURL

curl https://ricloud-api.reincubate.com/configs/storage/<storage_config ID> \
  -X DELETE \
  -H 'Authorization: Token <your key_token>'

Using ricloud-py

Since ricloud-py v3.0.0.

import ricloud

ricloud.StorageConfig.delete_with_id(<storage_config ID>)

# OR

storage_config = ricloud.StorageConfig.retrieve(<storage_config ID>)

storage_config.delete()

Test GET /configs/storage/{storage_config ID}/test

This action allows you to re-test the validity of a storage config. The state of the config will be set to valid or invalid depending on the outcome of the test.

Note that this action returns a task resource.

Using cURL

curl https://ricloud-api.reincubate.com/configs/storage/<storage_config ID>/test \
  -H 'Authorization: Token <your key_token>'

Using ricloud-py

import ricloud

test_task = ricloud.StorageConfig.test_with_id(<storage_config ID>)

# OR

storage_config = ricloud.StorageConfig.retrieve(<storage_config ID>)

test_task = storage_config.test()

How can we help?

Our support team are here to help!

Our office hours are Monday to Friday, 9am to 5pm GMT.

We aim to reply to all messages within one working day.

Get in touch › Our awesome support team

© 2008 - 2019 Reincubate Ltd. All rights reserved. Registered in England and Wales #5189175, VAT GB151788978. Reincubate® is a registered trademark. Privacy & terms. We recommend 2FA. Built with in London.