DataNature platform
Jobs and Pipelines: Orchestration you can see into
Jobs and Pipelines is the DataNature orchestration module. It schedules and runs DAG workflows across ingestion, transformation and machine learning, and records run history, failures, latency and freshness for every pipeline.
Updated
- Visual DAGs
- Drag-and-drop or code-defined workflows.
- Run telemetry
- Runs, failures, latency and freshness per pipeline.
- Retries and alerts
- Automatic retries. Alerts routed through Atlas.
- Any task
- Connect loads, dbt models, notebooks, SQL, Python.
On this page
What does it orchestrate?
Everything that moves data: ingestion, dbt runs, notebooks, SQL and model training. Tasks depend on each other in a DAG.
What does monitoring look like?
Every run records duration, status, rows processed and freshness. Failing pipelines raise alerts in Atlas with a link to the failed task.
Can I bring existing Airflow or Lakeflow jobs?
Job logic in SQL, Python, Spark and dbt moves over. Scheduling definitions are re-created, usually as part of a migration assessment.
Platform components in this module
Learn more about each DataNature product and module mentioned here.
Platform modules
Learn more about Jobs and Pipelines
Articles and guides
- Lakehouse architectureOracle, SQL Server, MySQL and Postgres to Apache Iceberg: a practical ingestion guide
- Lakehouse architecturedbt on Spark and Trino with Apache Iceberg: building medallion pipelines that stay fast and correct
- Lakehouse architectureCan Databricks run on-premise? What is possible, what is not, and the sovereign alternative
- AI and agentsHow to build predictive models on DataNature: from lakehouse data to production predictions
- Lakehouse architectureWebsite and app event tracking for a sovereign CDP: SDKs, identity stitching and real-time profiles
- Industry guidesImplementing a sovereign data and AI platform in 12 months: a foundation-first playbook
See it on your own data.
Book a demoDemos and quotes are handled by Dlytica, the company behind DataNature.