Extension Remote Data Harvesting

These CKAN Versions are exactely matched:

• 2.8.0
• 2.8.1
• 2.8.10
• 2.8.11
• 2.8.12
• 2.8.2
• 2.8.3
• 2.8.4
• 2.8.5
• 2.8.6
• 2.8.7
• 2.8.8
• 2.8.9
• 2.9.0
• 2.9.1
• 2.9.10
• 2.9.11
• 2.9.2
• 2.9.3
• 2.9.4
• 2.9.5
• 2.9.6
• 2.9.7
• 2.9.8
• 2.9.9
• 2.10.0
• 2.10.1
• 2.10.2
• 2.10.3
• 2.10.4
• 2.10.5
• 2.10.6
• 2.10.7
• 2.10.8

The Remote Data Harvesting extension provides a comprehensive framework for automatically importing datasets from external data sources, remote catalogs, and third-party APIs, enabling federated data management and seamless integration with distributed data ecosystems. This powerful extension supports multiple harvesting protocols including CKAN API, CSW (Catalog Service for Web), WAF (Web Accessible Folder), DCAT-RDF, and custom harvester implementations for specialized data sources. The system provides scheduled harvesting with configurable intervals, incremental updates, and intelligent change detection to minimize processing overhead while ensuring data freshness. Advanced features include data transformation pipelines, metadata mapping and enrichment, validation workflows, and conflict resolution mechanisms for handling duplicate datasets. The extension supports hierarchical harvesting configurations, multi-source aggregation, and distributed harvesting across multiple CKAN instances for large-scale data federation. Administrative tools provide harvesting status monitoring, job scheduling management, error handling and retry mechanisms, and comprehensive logging for troubleshooting. Integration capabilities include webhook notifications, API endpoints for external triggering, and integration with data quality assessment tools. Performance optimizations enable handling of large-scale harvesting operations with batch processing, parallel job execution, and resource management controls. Essential for data portals aggregating content from multiple sources, government platforms implementing open data federation, research networks sharing datasets across institutions, and organizations requiring automated data synchronization from diverse external systems where centralized data discovery and distributed data management are critical for comprehensive data accessibility.