You probably don’t need a DAG

This post was first published on Prefect’s blog. For most data engineers, “workflow” is virtually synonymous with “DAG.” Nearly every orchestrator, most notably Airflow, requires workflows to be DAGs. There’s no shortage of attempts to explain DAGs, but everything you …


A brief history of workflow orchestration

This post was first published on Prefect’s blog and later discussed on Prefect Live. For as long as we’ve been recording data with computers, we’ve been automating processes to handle that data. Basic orchestration concepts like “scheduler,” “job,” and …


How we talk about data

The language of technology is notoriously ambiguous, ridden with buzzwords and jargon. Specific technologies and general technical concepts rise and fall in relevance and perceived value so frequently that the phenomenon has its own moniker, the hype cycle. Over the past ten …


The value of services in building products

“Product” and “service” mean different things in different contexts. My frame of reference is business-to-business (B2B) software, where the distinction between products and services can be blurry. Software that is packaged and billed via a license or subscription (SaaS) model is often …