Data pipeline at scale

The visual pipeline explorer is being prepared. The production system collects from multiple sources, stores more than 1.85 million queryable items, and runs unattended on deployed infrastructure.