Skip to content

S3 compatible storage backend

Introduced in v13

InvenioRDM provides seamless integration with S3-compatible object storage systems. This allows institutions to leverage scalable, cost-effective, and highly durable cloud storage solutions for their research data, ensuring secure and efficient management of digital assets.

S3 via MinIO

MinIO is a high-performance, open-source object storage system that is API compatible with Amazon S3.

S3 selected

This section assumes you chose S3 when bootstrapping the instance.

License compatibility

The MinIO Object Store is licensed under Affero General Public License Version 3 (AGPLv3), see the compliance page. When using MinIO, make sure that your instance's license is compliant.

After having setup your MinIO application, you can review the settings below.

Bucket and data location

A default bucket (default) is created automatically in your MinIO container. This bucket will be used as the default files' location for your InvenioRDM instance if you choose s3 as file storage during project initialization.

If you want to change the bucket name, open a shell window and type pipenv run invenio files location s3-default s3://new-bucket --default. You will also need to manually create the new bucket in MinIO or your S3 storage system. The new bucket will only be used for new records or drafts (existing drafts and records will use default).

Credentials

The last step is to set your credentials so your instance can authenticate against the S3 container.

By default, the ROOT_USER and the ROOT_USER_PASSWORD of MinIO are "CHANGE_ME". For security reasons, you should change them. This will require you to:

  • Stop your services by using the CLI: invenio-cli services stop (Or stop only your MinIO container).
  • In the docker-services.yaml file change the MINIO_ROOT_USER and MINIO_ROOT_PASSWORD values.
  • Update the credentials:
    • You should be using environment variables to inject secrets. Define INVENIO_S3_ACCESS_KEY_ID for the user and INVENIO_S3_SECRET_ACCESS_KEY for the password.
    • If you are developing locally, you can change your invenio.cfg and set the value of S3_ACCESS_KEY_ID to the desired username and the value of S3_SECRET_ACCESS_KEY to the password.
  • Start your services: invenio-cli services start.

It's done! Ready to roll!

Amazon S3

If you're using Amazon S3, you'll want to provide the endpoint URL and set the region (if you're not using the default of us-east-1):

S3_ENDPOINT_URL='https://s3.us-west-2.amazonaws.com/'
S3_REGION_NAME='us-west-2'

To enable file previews in InvenioRDM when using Amazon S3, you must configure CORS (Cross-Origin Resource Sharing) for your S3 bucket. In the AWS console, navigate to your bucket's permissions tab and add: 

[
    {
        "AllowedHeaders": [],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": []
    }
]