Extension Custom Data.Qld extensions bundle


Extension Basics

Title
Custom Data.Qld extensions bundle
Name
ckanext-data-qld
Type
Public extension
Description
Custom CKAN extensions bundle for Data.Qld with Google Analytics integration, reporting, and data management features
CKAN versions

~2.9, ~2.10

Show details
Download-Url (zip)
https://github.com/DataShades/ckanext-data-qld.git#egg=ckanext-data-qld
Download-Url commit date
2023-01-01
Url to repo
https://github.com/DataShades/ckanext-data-qld
Category
Specialized Tools


Background Infos

Description (long)
Show details

This extension provides a comprehensive suite of custom features for Data.Qld. Features include: Google Analytics API action tracking with configurable event capture (capture_api_actions.json dictionary), engagement reporting with configurable thresholds (open data requests max days, comment no-reply max days), admin reporting with JSON config files (engagement_report_csv.json, admin_report_csv.json), legacy extra fields migration (Security classification, Used in data-driven application), publisher role demotion command (publisher-* users from admin to editor), dataset freshness notifications (due-to-publish, overdue emails), user fullname update command, missing values update utility, Docker-based local development with Pygmy/Ahoy, BDD testing with Behave/Behaving/Splinter, GitHub Actions CI/CD, Python 2.7 compatible (production stable). Multiple plugins: data_qld (main), data_qld_integration, data_qld_resources (placeholder), data_qld_google_analytics, data_qld_reporting (placeholder). Paster commands: migrate_extras, demote_publishers, update_fullname, send_email_dataset_due_to_publishing_notification, send_email_dataset_overdue_notification, update_missing_values. For Data.Qld deployment.

Version
7.0.1
Version release date
2023-01-01
Contact name
Queensland Government and Salsa Digital
Contakt email
(not set)
Contact Url
(not set)


Installation Guide

Configuration hints

Install via pip:

pip install -e git+https://github.com/qld-gov-au/ckanext-data-qld.git#egg=ckanext-data-qld

Enable plugins:

ckan.plugins = data_qld data_qld_google_analytics data_qld_reporting

Google Analytics configuration:

ckanext-data_qld_googleanalytics

ckan.data_qld_googleanalytics.id = UA-1010101-1

ckan.data_qld_googleanalytics.collection_url = http://www.google-analytics.com/collect

Change reporting start date (default: 2022-11-01)

ckanext.data_qld.reporting.de_identified_no_schema.count_from = 2045-01-01

Configure API action capture (capture_api_actions.json): - Dictionary key: api_action name from CKAN API reference - Dictionary value: event_label sent to Google Analytics with {0} replaced by query parameter (package_id, resource_id, query, sql)

Reporting configuration:

ckanext-data_qld_reporting

ckan.reporting.datarequest_open_max_days = 60 # Default: 60

ckan.reporting.comment_no_reply_max_days = 10 # Default: 10

ckan.reporting.engagement_json_config = PATH_TO_FILE # Default: ../engagement_report_csv.json

ckan.reporting.admin_json_config = PATH_TO_FILE # Default: ../admin_report_csv.json

Migration commands:

  1. Migrate legacy extra fields (run once during initial migration):

    . /usr/lib/ckan/default/bin/activate

    paster –plugin=ckanext-data-qld migrate_extras -c /path/to/ini_file.ini

    paster –plugin=ckan search-index rebuild -c /path/to/ini_file.ini

    Migrates ‘Security classification’ and ‘Used in data-driven application’ from free extras to schema fields.

  2. Demote publishers to editor role (run once):

    paster –plugin=ckanext-data-qld demote_publishers -c /etc/ckan/default/development.ini

    Sets users with names starting with ‘publisher-’ from admin to editor role.

  3. Update user fullname:

    paster –plugin=ckanext-data-qld update_fullname -c /path/to/ini_file.ini

  4. Dataset freshness notifications:

    paster –plugin=ckanext-data-qld send_email_dataset_due_to_publishing_notification -c /path/to/ini_file.ini

    paster –plugin=ckanext-data-qld send_email_dataset_overdue_notification -c /path/to/ini_file.ini

  5. Update missing values:

    paster –plugin=ckanext-data-qld update_missing_values -c /path/to/ini_file.ini

Local development (Docker):

pygmy up

ahoy build

ahoy up

Login: admin/password

Available ahoy commands: - build, clean, cli, doctor, down, flush-redis - info, install-site, lint, logs, pull, reset - restart, start, stop, test-bdd, test-unit, up

Testing:

ahoy test-unit # Run unit tests

ahoy test-bdd # Run BDD tests

Environment variables: - ALLOW_LINT_FAIL=1 to allow lint failures - ALLOW_UNIT_FAIL=1 to allow unit test failures - ALLOW_BDD_FAIL=1 to allow BDD test failures

Requires ckanext-scheming for dataset schema.

Restart web server after configuration:

sudo service apache reload

sudo service nginx reload

Plugins to configure (ckan.ini)
data_qld data_qld_integration data_qld_resources data_qld_google_analytics data_qld_reporting data_qld_test
CKAN Settings (ckan.ini)
# ckan.data_qld_googleanalytics.id = UA-1010101-1 # Relevant Google analytics ID # ckan.reporting.datarequest_open_max_days = 60 # Defaults to 60 # ckan.reporting.comment_no_reply_max_days = 10 # Defaults to 10 # ckan.reporting.engagement_json_config = PATH_TO_FILE # Defaults to os.path.dirname(os.path.realpath(__file__)) + '/../engagement_report_csv.json' # ckan.reporting.admin_json_config = PATH_TO_FILE # Defaults to os.path.dirname(os.path.realpath(__file__)) + '/../admin_report_csv.json'
DB migration to be executed
data-qld
<< back to Extensions