Extension Archiver

These CKAN Versions are exactely matched:

• 2.1
• 2.1.1
• 2.1.2
• 2.1.3
• 2.1.4
• 2.1.5
• 2.1.6
• 2.10.0
• 2.10.1
• 2.10.2
• 2.10.3
• 2.10.4
• 2.10.5
• 2.10.6
• 2.10.7
• 2.10.8
• 2.11.0
• 2.11.1
• 2.11.2
• 2.11.3
• 2.12.0
• 2.2
• 2.2.1
• 2.2.2
• 2.2.3
• 2.2.4
• 2.3
• 2.3.1
• 2.3.2
• 2.3.3
• 2.3.4
• 2.3.5
• 2.4.0
• 2.4.1
• 2.4.2
• 2.4.3
• 2.4.4
• 2.4.5
• 2.4.6
• 2.4.7
• 2.4.8
• 2.4.9
• 2.5.0
• 2.5.1
• 2.5.2
• 2.5.3
• 2.5.4
• 2.5.5
• 2.5.6
• 2.5.7
• 2.5.8
• 2.5.9
• 2.6.0
• 2.6.1
• 2.6.2
• 2.6.3
• 2.6.4
• 2.6.5
• 2.6.6
• 2.6.7
• 2.6.8
• 2.6.9
• 2.7.0
• 2.7.1
• 2.7.10
• 2.7.11
• 2.7.12
• 2.7.2
• 2.7.3
• 2.7.4
• 2.7.5
• 2.7.6
• 2.7.7
• 2.7.8
• 2.7.9
• 2.8.0
• 2.8.1
• 2.8.10
• 2.8.11
• 2.8.12
• 2.8.2
• 2.8.3
• 2.8.4
• 2.8.5
• 2.8.6
• 2.8.7
• 2.8.8
• 2.8.9
• 2.9.0
• 2.9.1
• 2.9.10
• 2.9.11
• 2.9.2
• 2.9.3
• 2.9.4
• 2.9.5
• 2.9.6
• 2.9.7
• 2.9.8
• 2.9.9

The CKAN Archiver Extension will download all of a CKAN’s resources, for three purposes:

1. Offer the user it as a ‘cached’ copy, in case the link becomes broken
2. Tell the user (and publishers) if the link is broken, on both the dataset/resource and in a ‘Broken Links’ report
3. The downloaded file can be analysed by other extensions, such as ckanext-qa or ckanext-packagezip

When a resource is archived, the information about the archival - if it failed, the filename on disk, file size etc - is stored in the Archival table and added to dataset during the package_show call, so the information is also available over the API.

Other extensions can subscribe to the archiver’s IPipe interface to hear about datasets being archived. e.g. ckanext-qa will detect its file type and give it an openness score, or ckanext-packagezip will create a zip of the files in a dataset.

Archiver works on Celery queues (CKAN 2.6 and earlier) or background jobs (CKAN 2.7+), so when Archiver is notified of a dataset/resource being created or updated, it puts an ‘update request’ on a queue. You can start Celery/jobs with multiple processes, to archive in parallel.

By default, two queues are used: 1. ‘bulk’ for a regular archival of all the resources 2. ‘priority’ for when a user edits one-off resource

This means that the ‘bulk’ queue can happily run slowly, archiving large quantities slowly, such as re-archiving every single resource once a week. And meanwhile, if a new resource is put into CKAN then it can be downloaded straight away via the ‘priority’ queue.

Features: - Automatic resource caching and archiving - Broken link detection and reporting - IPipe interface for integration with other extensions - Configurable file size limits and formats - Queue-based processing with priority levels - Comprehensive broken links report - Web server integration for serving cached files - SNI support for HTTPS resources - Support for both Celery (pre-2.7) and CKAN jobs (2.7+)