Skip to main content

Job commands

A dbt Cloud production job allows you to set up a system to run a dbt job and job commands on a schedule, rather than running dbt commands manually from the command line or IDE. A job consists of commands that are "chained" together and executed as run steps. Each run step can succeed or fail, which may determine the job's run status (Success, Cancel, or Error).

Each job allows you to:

  • Configure job commands
  • View job run details, including timing, artifacts, and detailed run steps
  • Access logs to view or help debug issues and historical invocations of dbt
  • Set up notifications, and more

Job command types​

Job commands are specific tasks executed by the job, and you can configure them seamlessly by either adding dbt commands or using the checkbox option in the Commands section.

During a job run, the commands are "chained" together and executed as run steps. When you add a dbt command in the Commands section, you can expect different outcomes compared to the checkbox option.

Configuring checkbox and commands listConfiguring checkbox and commands list

Built-in commands​

Every job invocation automatically includes the dbt deps command, meaning you don't need to add it to the Commands list in your job settings. You will also notice every job will include a run step to reclone your repository and connect to your data platform, which can affect your job status if these run steps aren't successful.

Job outcome — During a job run, the built-in commands are "chained" together. This means if one of the run steps in the chain fails, then the next commands aren't executed, and the entire job fails with an "Error" job status.

A failed job that had an error during the dbt deps run step.A failed job that had an error during the dbt deps run step.

Checkbox commands​

For every job, you have the option to select the Generate docs on run or Run source freshness checkboxes, enabling you to run the commands automatically.

Job outcome Generate docs on run checkbox — dbt Cloud executes the dbt docs generate command, after the listed commands. If that particular run step in your job fails, the job can still succeed if all subsequent run steps are successful. Read Set up documentation job for more info.

Job outcome Source freshness checkbox — dbt Cloud executes the dbt source freshness command as the first run step in your job. If that particular run step in your job fails, the job can still succeed if all subsequent run steps are successful. Read Source freshness for more info.

Command list​

You can add or remove as many dbt commands as necessary for every job. However, you need to have at least one dbt command. There are few commands listed as "dbt Cloud CLI" or "dbt Core" in the dbt Command reference page page. This means they are meant for use in dbt Core or dbt Cloud CLI, and not in dbt Cloud IDE.

Using selectors

Use selectors as a powerful way to select and execute portions of your project in a job run. For example, to run tests for one specific model, use the selector: dbt test --select one_specific_model. The job will still run if a selector doesn't match any models.

Advanced CI compare changes​

For users that have Advanced CI's compare changes feature enabled, you can use a custom dbt syntax to exclude specific large models (or groups of models with tags) to optimize running the comparison. Running comparisons on large models can significantly increase the time it takes for CI jobs to complete.

For example, to exclude the large fct_orders model from the comparison, use the --select state:modified --exclude fct_orders syntax.

Job outcome​

During a job run, the commands are "chained" together and executed as run steps. If one of the run steps in the chain fails, then the subsequent steps aren't executed, and the job will fail.

In the following example image, the first four run steps are successful. However, if the fifth run step (dbt run --select state:modified+ --full-refresh --fail-fast) fails, then the next run steps aren't executed, and the entire job fails. The failed job returns a non-zero exit code and "Error" job status:

A failed job run that had an error during a run stepA failed job run that had an error during a run step

Job command failures​

Job command failures can mean different things for different commands. Some common reasons why a job command may fail:

  • Failure atdbt run — dbt run executes compiled SQL model files against the current target database. It will fail if there is an error in any of the built models. Tests on upstream resources prevent downstream resources from running and a failed test will skip them.

  • Failure at dbt test — dbt test runs tests defined on models, sources, snapshots, and seeds. A test can pass, fail, or warn depending on its severity. Unless you set warnings as errors, only an error stops the next step.

  • Failure at dbt build — dbt build runs models, tests, snapshots, and seeds. This command executes resources in the DAG-specified order. If any upstream resource fails, all downstream resources are skipped, and the command exits with an error code of 1.

  • Selector failures

    • If a select matches multiple nodes and one of the nodes fails, then the job will have an exit code 1 and the subsequent command will fail. If you specified the —fail-fast flag, then the first failure will stop the entire connection for any models that are in progress.

    • If a selector doesn't match any nodes, it's not considered a failure.

0
Edit this page