Black Friday50% off your first month on any monthly plan with EARLY50 See plans
All 33 lessons
  1. Start here: cloud and data basics

    1. 1. Start here: the exam, the plan and what you already know
    2. 2. Cloud in plain words: SaaS, PaaS, IaaS and where Fabric fits
    3. 3. Data basics: files, tables, formats and the lakehouse idea
    4. 4. A tour of Microsoft Fabric: OneLake, workspaces and items
    5. 5. The three languages: SQL, PySpark and KQL side by side
    6. 6. Getting data in: the map of Fabric tools
  2. Store, ingest and transform batch data (exam area 2, 30-35%)

    1. 7. Choosing a data store and a transformation tool
    2. 8. Shortcuts and mirroring: use data without copying it
    3. 9. Ingest data with pipelines and Copy job
    4. 10. Transform with PySpark and SQL: join, denormalize, aggregate
    5. 11. Messy data: duplicates, missing values and late arrivals
    6. 12. Full and incremental loads, and loading a star schema
  3. Orchestrate and stream (exam areas 1 and 2)

    1. 13. Pipeline or notebook? Choosing how to orchestrate
    2. 14. Schedules and event-based triggers
    3. 15. Parameters and dynamic expressions
    4. 16. Streaming in Fabric: Eventstreams, Spark streaming and KQL
    5. 17. KQL and windowing functions for real-time data
  4. Set up and secure a Fabric solution (exam area 1, 30-35%)

    1. 18. Workspace settings: Spark, domains, OneLake and data workflows
    2. 19. Version control with Git and database projects
    3. 20. Deployment pipelines: development, test and production
    4. 21. Who can see what: workspace roles and item permissions
    5. 22. Fine-grained security: rows, columns, objects, files and masking
    6. 23. Governance: sensitivity labels and endorsement
  5. Monitor, fix and optimize (exam area 3, 30-35%)

    1. 24. Monitoring hub, alerts and Activator
    2. 25. Fixing errors in pipelines, dataflows and notebooks
    3. 26. Fixing errors in Eventhouse, Eventstream and T-SQL
    4. 27. Optimize lakehouse tables: V-Order, OPTIMIZE, VACUUM and partitions
    5. 28. Optimize Spark, pipelines and warehouses
    6. 29. Optimize eventstreams, eventhouses and queries
  6. Exam practice and final review

    1. 30. Exam day: question types, timing and how to think
    2. 31. Mock exam 1
    3. 32. Mock exam 2
    4. 33. Final review: every decision on one page
See the leaderboard

16 min

Pipeline or notebook? Choosing how to orchestrate

Pick between a pipeline and a notebook for a job and combine them when needed.

By the end of this lesson you can

  • Choose a pipeline, a notebook, or both for an orchestration scenario
  • Pick the right control flow activity: ForEach, If Condition, Until or Switch
  • Run a notebook from a pipeline and pass it parameters
  • Run other notebooks from a notebook with notebookutils run and runMultiple

From earlier lessons · 1 of 2

In an incremental load, when should you save the new watermark value?

Pick one answer.

From earlier lessons · 2 of 2

A customer's home city changes. Old trips must stay under the old city in reports. Which SCD type do you use for the city column?

Pick one answer.

Northwind Rides' nightly job has grown. It copies payments from SQL Server, runs three notebooks, calls a stored procedure in the Warehouse, and must post to Teams if anything fails. Priya wants one pipeline. Sam wants one notebook that calls the others. They're both a little right.

[4]

Pipelines: low-code steps in order

Exam depth: know it cold.

A pipeline is a group of steps you build on a visual canvas, then run, schedule and monitor as one unit. Each step is an activity. There are three kinds.

  • Data movement: Copy data and Copy job.
  • Data transformation: run other items, such as a Notebook, a Dataflow Gen2 or a stored procedure (saved SQL code in a Warehouse).
  • Control flow: decide what runs next.

Control flow means the logic that picks which step runs, how often, and in what order. The ones to know:

ActivityWhat it does.
ForEachRuns inner activities once per item in a list.
If ConditionRuns one branch if true, another if false.
UntilRepeats until a condition is true, with a timeout.
SwitchPicks one branch per value of an expression.

You link activities on success, failure or completion. Each activity has a timeout (default 12 hours) and optional retries. A pipeline holds up to 120 activities.

[106][107][108]

A ForEach activity on the pipeline canvas with its Settings tab open, showing a Sequential checkbox, a Batch count box and an Items box containing @createArray('Orange', 'Apple', 'Kiwi').
Microsoft Learn, CC BY 4.0

[108]

Match

Match each Northwind Rides need to the pipeline activity for it

Pick an item on the left, then its match on the right.

[107]

Notebooks: code for complex transforms

Exam depth: know it cold.

A Notebook runs code on Spark, in PySpark, Scala, Spark SQL or R. Each run starts a Spark session, the compute the notebook runs on, which takes time to start.

A notebook is the tool for complex transformations: joins across large tables, window functions, lesson 11's deduplication, or machine learning.

A pipeline itself does no transformation. It orchestrates items that do. A notebook does the transforming, and with notebookutils it can also run other notebooks.

What a notebook can't do alone is mix item types. It can't drop a Dataflow Gen2, a stored procedure and a Teams message onto one canvas with built-in retries.

[70][109]

Pipeline or notebook: the decision table

Exam depth: know it cold.

The job needsPipelineNotebook.
Built by a low-code teamYes: drag and drop.No: code only.
Copy data at scale from many sourcesCopy data, 50+ connectors.You write the read code.
Complex transformsCalls a notebook or dataflow.Yes: Spark, any library.
Steps of different item typesYes: one canvas.Notebooks only.
Loops and branchesForEach, If, Until, Switch.Python code, runMultiple.
Many notebooks in one Spark sessionNo.Yes: runMultiple.

Decision rule: many kinds of steps, copying, or a low-code team: pipeline. Heavy code, or many notebooks sharing one Spark session: notebook.

The common real answer is both: a pipeline runs the steps, and some steps are notebooks.

[70][107][112]

You are a data engineer at Northwind Rides. Each night you must copy payments from SQL Server, run a Warehouse stored procedure, run a notebook, and post to a Teams channel if any step fails. The ops team who will maintain it doesn't write code. What should you use to orchestrate this?

Pick one answer.

[70][107]

The pipeline is the conductor. The notebook plays the solo. Picture: Retro pixel art, 16-bit style: a small pixel orchestra on a stage.

Run a notebook from a pipeline

Exam depth: know it cold.

The Notebook activity runs a Fabric notebook as one pipeline step. On its Settings tab you pick the workspace and the notebook, and set Base parameters.

Parameters reach the notebook through its parameters cell: one cell you mark with Toggle parameter cell. It holds default values. At run time, Fabric adds a new cell right below it that sets the values the pipeline passed, so the pipeline's values win. With no marked cell, the activity's Base parameters don't fill in for you. Always mark one.

Parameters support simple types: int, float, bool and string. Pass a list as a JSON string and parse it inside.

To send a result back, the notebook calls notebookutils.notebook.exit("value"). The activity returns that exit value to the pipeline.

A session tag under Advanced settings lets several Notebook activities reuse one Spark session. It needs high concurrency mode for pipelines turned on in the workspace settings.

[54][110][112]

notebookutils: notebooks that run notebooks

Exam depth: know it cold.

notebookutils is a built-in package in Fabric notebooks. Its old name was MSSparkUtils, and old code still works. Its notebook module runs other notebooks.

  • notebookutils.notebook.run("name", 90, {"city": "Oslo"}) runs one notebook and waits; 90 is the timeout in seconds. It returns the child's exit value, always as a string. The child runs on the caller's Spark pool.
  • notebookutils.notebook.runMultiple(dag) runs many notebooks at once, in one Spark session. A DAG (directed acyclic graph) lists each notebook and which ones it must wait for, in its dependencies.

Defaults to know for runMultiple: whole DAG timeout 12 hours, per-cell timeout 90 seconds, and up to 3 times the CPU core count running at once. Call validateDAG(dag) first to catch mistakes.

Decision rule: one child, run. Many children in parallel or with an order, runMultiple.

[111][112]

python
dag = {
"activities": [
{"name": "clean_oslo", "path": "clean_trips",
"args": {"city": "Oslo"}},
{"name": "clean_lisbon", "path": "clean_trips",
"args": {"city": "Lisbon"}},
{"name": "build_gold", "path": "load_fact_trip",
"dependencies": ["clean_oslo", "clean_lisbon"],
"retry": 1},
],
"timeoutInSeconds": 3600,
}
notebookutils.notebook.validateDAG(dag)
results = notebookutils.notebook.runMultiple(dag)
print(results["build_gold"]["exitVal"])
Clean two cities in parallel, then build gold once both finish. All three share one Spark session.

[112]

[112]

You are a data engineer at Northwind Rides. A parent notebook must run 40 small notebooks, one per city, as fast as possible, then one final notebook after all 40 finish. They should share one Spark session to avoid startup time. What should you use?

Pick one answer.

[110][112]

Apache Airflow job

Exam depth: recognize it.

An Apache Airflow job is a Fabric item that runs Apache Airflow for you. Apache Airflow is an open-source tool where you write workflows as Python DAGs. It's code-first orchestration.

Pick it when the team already writes Airflow DAGs, or wants to orchestrate in Python. Pick a pipeline when the team wants no-code or low-code orchestration.

It's the next generation of Azure Data Factory's Workflow Orchestration Manager. Its workspace settings come back in lesson 18.

[70][113]

[107]

Try it in Fabric

A free Fabric trial is enough for this.

  1. In your workspace, create a Notebook. In the first cell, type city = "Oslo".
  2. On that cell's toolbar, select ... (More commands), then Toggle parameter cell.
  3. In a second cell, type print(city). Save the notebook.
  4. In the workspace, select + New item, then search for and select Pipeline. Name it and select Create.
  5. On the Activities tab, select Notebook to add it to the canvas.
  6. Select the activity. On Settings, pick your workspace and notebook.
  7. Expand Base parameters. Set city to Lisbon.
  8. On the Home tab, select Save, then Run.
  9. Watch the run in the Output tab below the canvas. When it succeeds, the notebook used Lisbon, the pipeline's value, not its default Oslo.

[54][110]

A Notebook activity on the pipeline canvas with its Settings tab open, showing the Connection, Workspace and Notebook dropdowns, and collapsed Base parameters and Advanced settings sections.
Microsoft Learn, CC BY 4.0

[110]

Exam-style practice

You are a data engineer at Northwind Rides. Northwind buys a delivery startup whose engineers run 60 workflows as Apache Airflow DAGs written in Python. They must move the workflows into Fabric with the least rework. What should they use?

Pick one answer.

[70][113]

You are a data engineer at Northwind Rides. A pipeline's Notebook activity passes base parameter run_date. The notebook's first cell sets run_date = "2026-01-01", and every run uses that date instead of the pipeline's value. You need the pipeline's value to apply. What should you do?

Pick one answer.

[54]

FlashcardsIndex cardsFlashcards ⚡

Card 1 of 14

[70][107][108][110][112][113]

Worth keeping

  1. Pipeline: low-code steps of many item types, copy at scale, built-in retries.
  2. Notebook: complex Spark code; runMultiple runs many notebooks in one session.
  3. Often both: a pipeline's Notebook activity runs notebooks with parameters.
  4. Parameters reach a notebook only through its marked parameters cell.
  5. Apache Airflow job: recognize it as code-first Python orchestration.

On the job

Priya and Sam settle on both.

A pipeline called nightly_load is the one thing anyone schedules or reruns. Its first activity copies new payments from SQL Server with the watermark from lesson 12. Next, a Notebook activity runs clean_all_cities.

That notebook is Sam's part. It builds a DAG of 40 city notebooks and runs them with runMultiple in one Spark session. That takes minutes, not the hour 40 separate sessions would.

Back in the pipeline, a Stored procedure activity runs the SCD Type 2 load in the Warehouse. Every activity's failure path leads to one Teams activity that posts the error to the ops channel.

The ops team can read the canvas and rerun a failed step. The engineers keep the heavy logic in code.

[107][112]

nightly_load is built, but someone still has to press Run at 2 a.m. And the partner's CSV doesn't always land on time. Next lesson: schedules, and triggers that start the pipeline the moment the file arrives.

Leaderboard

Loading…

50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.

Sources for this lesson (11)

Each fact in the lesson is marked with the number of the page it comes from.

  1. [4]Microsoft Learn: Study Guide for Exam DP-700
  2. [54]Microsoft Learn: Develop, execute, and manage Microsoft Fabric notebooks
  3. [70]Microsoft Learn: Microsoft Fabric decision guide: Choose a data integration strategy
  4. [106]Microsoft Learn: Pipeline overview
  5. [107]Microsoft Learn: Activity overview
  6. [108]Microsoft Learn: Transform data with a ForEach activity
  7. [109]Microsoft Learn: Prepare and transform data in Microsoft Fabric
  8. [110]Microsoft Learn: Transform data by running a notebook
  9. [111]Microsoft Learn: NotebookUtils (former MSSparkUtils) for Fabric
  10. [112]Microsoft Learn: NotebookUtils notebook run and orchestration
  11. [113]Microsoft Learn: What is an Apache Airflow job?