Extension Multi-provider cloud storage with libcloud


Extension Basics

Title
Multi-provider cloud storage with libcloud
Name
ckanext-cloudstorage
Type
Public extension
Description
Store resources on S3, Azure, Rackspace or 15+ cloud providers via libcloud
CKAN versions
Download-Url (zip)
Download-Url commit date
2020-01-01
Url to repo
Category
Cloud Infrastructure & Storage


Background Infos

Description (long)
Show details

This extension implements comprehensive cloud storage support for CKAN using Apache libcloud, supporting S3, Azure, Rackspace and 15+ different storage providers. Features include: multi-provider cloud storage support (Azure Blobs, AWS S3, Rackspace, and 15+ libcloud providers), secure URLs for private resources with temporary one-time access links, multipart upload support for large files, automatic migration command from CKAN FileStorage to cloud storage, CORS auto-configuration with fix-cors command, configurable secure URL TTL (default 1 hour), cleanup of unfinished multipart uploads older than 7 days via cloudstorage_clean_multipart action (sysadmin only), flexible driver configuration via Python dict for provider-specific options, tested support for Azure (with azure-storage), AWS S3 (with boto and host key), and Rackspace (no secure URLs). Secure URLs enable CKAN access restrictions to apply to cloud-stored resources by generating temporary signed download URLs. Only resources are supported (group/organization images still use local storage). Recommend disabling public listing on cloud provider if supported.

Version
0.4.0
Version release date
2020-01-01
Contact name
Tyler Kennedy
Contakt email
Contact Url
(not set)


Installation Guide

Configuration hints

Setup CKAN file storage first if not already configured.

Install and enable the plugin:

ckan.plugins = stats cloudstorage

Required configuration (all case-sensitive):

Cloud storage driver name (see libcloud Provider Constant column)

ckanext.cloudstorage.driver = AZURE_BLOBS

Container/bucket name

ckanext.cloudstorage.container_name = demo

Driver-specific options (Python dict with provider credentials):

For most providers (key/secret authentication):

ckanext.cloudstorage.driver_options = {“key”: “”, “secret”: “”}

Optional secure URLs configuration:

Enable secure URLs for private resources (default: disabled)

ckanext.cloudstorage.use_secure_urls = 1

Run database migration when enabling secure URLs:

ckan db upgrade -p cloudstorage

Secure URL TTL in seconds (default: 3600 = 1 hour)

ckanext.cloudstorage.secure_ttl = 60

Max multipart upload lifetime in days (default: 7)

ckanext.cloudstorage.max_multipart_lifetime = 7

Migrate existing FileStorage resources to cloud:

paster cloudstorage migrate -c ../ckan/development.ini

Path should be ckan.storage_path + /resources

Fix CORS rules for your domains (required for DataViews):

paster cloudstorage fix-cors -c=

Clean old multipart uploads (sysadmin action):

Call cloudstorage_clean_multipart action to abort uploads >7 days old

Provider-specific notes: - Azure: Requires azure-storage installed for secure URLs - AWS S3: Requires boto installed and ‘host’ key in driver_options for secure URLs - Rackspace: Secure URLs not available

Find driver names: https://libcloud.readthedocs.io/en/latest/storage/supported_providers.html

Plugins to configure (ckan.ini)
cloudstorage
CKAN Settings (ckan.ini)
# ckanext.cloudstorage.driver = AZURE_BLOBS # ckanext.cloudstorage.container_name = demo # ckanext.cloudstorage.driver_options = {"key": "<your public key>", "secret": "<your secret key>"} # ckanext.cloudstorage.use_secure_urls = 1 # ckanext.cloudstorage.max_multipart_lifetime = 7
DB migration to be executed
cloudstorage
<< back to Extensions