Configuring storage

Some parts of these instructions optionally use ricloud-py to simplify setup. To make use of this, check out the installation and configuration docs for ricloud-py.

The storage configuration tells the API where to return the results of your tasks. Currently, we support both Google Cloud Storage (GS) and Amazon S3 (S3) storage buckets.

Setting up storage with Google Cloud Storage

Creating a Google Cloud Platform project

If you do not have a Google Cloud Platform (GCP) account you will need to get signed up. Simply click 'Try free' (or 'Try it free') to go through the signup process.

Next, you will need a project to house your storage bucket. One may have been created automatically on signup or you might want a new, separate project just for this purpose. You can find additional instructions on how to manage projects in the Google Cloud docs.

Creating a Google Cloud Storage bucket

Now that the GCP project is ready, you can go ahead and create a new storage bucket. We recommend using at least the 'Regional' storage class and the 'us-east1' location (this is where most of our resources are located). For this guide we will name our bucket ricloud-storage, but you will likely have to pick a different name as bucket names are globally unique across their host service.

Creating a Service Account for the API

After the bucket has initialised, you will need to create a Service Account to give the API access to it. A Service Account is essentially a robot user of your project to which we can give permissions to only access our new bucket. Go ahead and follow the steps outlined in the Google Cloud IAM docs to setup a Service Account. The choice of name is up to you, but in this guide we will call it ricloud-sa. Ignore all optional fields when creating the Service Account, we will set permissions in a later step.

If you choose to create a key in the final step of the process, keep it safe as this is the file we will give to the API later. Otherwise, create a key now by following these steps.

Setting bucket-level permissions

Return to the Storage section of your GCP project and follow the steps outline under 'Adding a member to a bucket-level policy' in the Google Cloud Storage docs to add the ricloud-sa Service Account. Give it the 'Storage Object Creator' and 'Storage Legacy Bucket Reader' roles.

Creating the configuration

At this point, the Service Account should be fully setup and ready to be used with the API.

Using ricloud-py to create the configuration

Using ricloud-py, you can create a new storage configuration resource with the command:

> ricloud storage-config create --url "<storage bucket url>" --credentials-path <path to credentials file>

Where storage bucket url in our case would be gs://ricloud-storage and path to credentials file is just the path to the Service Account key file we created earlier.

Using cURL to create the configuration

curl -X POST \
  https://ricloud-api.reincubate.com/configs/storage \
  -H 'Authorization: Token <your key_token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "gs://ricloud-storage",
    "credentials": <all contents of the Service Account JSON key file>
}'

Setting up storage with Amazon S3

Creating an Amazon AWS account

If you do not have an Amazon AWS account you will need to get signed up. Click 'Get started with Amazon S3' to go through the setup process.

Creating an S3 storage bucket

Follow these instructions from the AWS docs to get a bucket setup in S3. Choose any name you like, for this guide we will pick ricloud-storage. We recommend locating it in 'US East (N. Virginia)' for proximity to the API's resources.

Creating a user for the API

Once the bucket is ready for usage, you will need to setup a new user with limited permissions to the new bucket for the API to use. This is done through the IAM console through these steps. Give the user a memorable name, like ricloud-user and make sure they are set up for 'Programmatic access'.

We need to create a custom policy, as when asked to give the user permissions there is no default policy that fits our needs. For standard result publishing, the API requires the s3:PutObject permission.

In the AWS policy editor, select 'Attach existing policies directly', click 'Create policy' (this should open a new tab), then select the 'JSON' tab (which will switch to the JSON policy editor) and paste the following in:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "s3:PutObject"
        ],
        "Resource": ["arn:aws:s3:::ricloud-storage/*"]
    }]
}

Make sure to replace the bucket name ricloud-storage with whatever you named the bucket earlier. Go through to save the new policy, name it whatever you would like.

Go back to the user creation flow and make sure the user has our new policy attached to them.

At the end of the user creation process the interface will display the 'Access key ID' and 'Secret access key', you should keep these safe. You can also download the credentials in CSV format.

Creating the configuration

At this point, the user should be fully setup and ready to be used with the API. Using ricloud-py, you can create a new storage configuration resource with the command:

> ricloud storage-config create --url "<storage bucket url>" --credentials-path <path to credentials file>

Where storage bucket url in this case would be s3://ricloud-storage and path to credentials file is just the path to the credentials CSV file exported at the end of the user creation process or a JSON file with the user credential information in the S3 credentials format.

The equivalent cURL call would look like:

curl -X POST \
  https://ricloud-api.reincubate.com/configs/storage \
  -H 'Authorization: Token <your key_token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "s3://ricloud-storage",
    "credentials": {
        "user_name": "ricloud-user",
        "access_key_id": "<user access key ID>",
        "secret_access_key": "<user secret access key>",
    }
}'