All 33 lessons
Start here: cloud and data basics
- 1. Start here: the exam, the plan and what you already know
- 2. Cloud in plain words: SaaS, PaaS, IaaS and where Fabric fits
- 3. Data basics: files, tables, formats and the lakehouse idea
- 4. A tour of Microsoft Fabric: OneLake, workspaces and items
- 5. The three languages: SQL, PySpark and KQL side by side
- 6. Getting data in: the map of Fabric tools
Store, ingest and transform batch data (exam area 2, 30-35%)
- 7. Choosing a data store and a transformation tool
- 8. Shortcuts and mirroring: use data without copying it
- 9. Ingest data with pipelines and Copy job
- 10. Transform with PySpark and SQL: join, denormalize, aggregate
- 11. Messy data: duplicates, missing values and late arrivals
- 12. Full and incremental loads, and loading a star schema
Orchestrate and stream (exam areas 1 and 2)
Set up and secure a Fabric solution (exam area 1, 30-35%)
- 18. Workspace settings: Spark, domains, OneLake and data workflows
- 19. Version control with Git and database projects
- 20. Deployment pipelines: development, test and production
- 21. Who can see what: workspace roles and item permissions
- 22. Fine-grained security: rows, columns, objects, files and masking
- 23. Governance: sensitivity labels and endorsement
Monitor, fix and optimize (exam area 3, 30-35%)
- 24. Monitoring hub, alerts and Activator
- 25. Fixing errors in pipelines, dataflows and notebooks
- 26. Fixing errors in Eventhouse, Eventstream and T-SQL
- 27. Optimize lakehouse tables: V-Order, OPTIMIZE, VACUUM and partitions
- 28. Optimize Spark, pipelines and warehouses
- 29. Optimize eventstreams, eventhouses and queries
Exam practice and final review
16 min
Pipeline or notebook? Choosing how to orchestrate
Pick between a pipeline and a notebook for a job and combine them when needed.
By the end of this lesson you can
- Choose a pipeline, a notebook, or both for an orchestration scenario
- Pick the right control flow activity: ForEach, If Condition, Until or Switch
- Run a notebook from a pipeline and pass it parameters
- Run other notebooks from a notebook with notebookutils run and runMultiple
From earlier lessons · 1 of 2
In an incremental load, when should you save the new watermark value?
Pick one answer.
From earlier lessons · 2 of 2
A customer's home city changes. Old trips must stay under the old city in reports. Which SCD type do you use for the city column?
Pick one answer.
Northwind Rides' nightly job has grown. It copies payments from SQL Server, runs three notebooks, calls a stored procedure in the Warehouse, and must post to Teams if anything fails. Priya wants one pipeline. Sam wants one notebook that calls the others. They're both a little right.
Pipelines: low-code steps in order
Exam depth: know it cold.
A pipeline is a group of steps you build on a visual canvas, then run, schedule and monitor as one unit. Each step is an activity. There are three kinds.
- Data movement: Copy data and Copy job.
- Data transformation: run other items, such as a Notebook, a Dataflow Gen2 or a stored procedure (saved SQL code in a Warehouse).
- Control flow: decide what runs next.
Control flow means the logic that picks which step runs, how often, and in what order. The ones to know:
| Activity | What it does. |
|---|---|
| ForEach | Runs inner activities once per item in a list. |
| If Condition | Runs one branch if true, another if false. |
| Until | Repeats until a condition is true, with a timeout. |
| Switch | Picks one branch per value of an expression. |
You link activities on success, failure or completion. Each activity has a timeout (default 12 hours) and optional retries. A pipeline holds up to 120 activities.
Match
Match each Northwind Rides need to the pipeline activity for it
Pick an item on the left, then its match on the right.
Notebooks: code for complex transforms
Exam depth: know it cold.
A Notebook runs code on Spark, in PySpark, Scala, Spark SQL or R. Each run starts a Spark session, the compute the notebook runs on, which takes time to start.
A notebook is the tool for complex transformations: joins across large tables, window functions, lesson 11's deduplication, or machine learning.
A pipeline itself does no transformation. It orchestrates items that do. A notebook does the transforming, and with notebookutils it can also run other notebooks.
What a notebook can't do alone is mix item types. It can't drop a Dataflow Gen2, a stored procedure and a Teams message onto one canvas with built-in retries.
Pipeline or notebook: the decision table
Exam depth: know it cold.
| The job needs | Pipeline | Notebook. |
|---|---|---|
| Built by a low-code team | Yes: drag and drop. | No: code only. |
| Copy data at scale from many sources | Copy data, 50+ connectors. | You write the read code. |
| Complex transforms | Calls a notebook or dataflow. | Yes: Spark, any library. |
| Steps of different item types | Yes: one canvas. | Notebooks only. |
| Loops and branches | ForEach, If, Until, Switch. | Python code, runMultiple. |
| Many notebooks in one Spark session | No. | Yes: runMultiple. |
Decision rule: many kinds of steps, copying, or a low-code team: pipeline. Heavy code, or many notebooks sharing one Spark session: notebook.
The common real answer is both: a pipeline runs the steps, and some steps are notebooks.
You are a data engineer at Northwind Rides. Each night you must copy payments from SQL Server, run a Warehouse stored procedure, run a notebook, and post to a Teams channel if any step fails. The ops team who will maintain it doesn't write code. What should you use to orchestrate this?
Pick one answer.
Run a notebook from a pipeline
Exam depth: know it cold.
The Notebook activity runs a Fabric notebook as one pipeline step. On its Settings tab you pick the workspace and the notebook, and set Base parameters.
Parameters reach the notebook through its parameters cell: one cell you mark with Toggle parameter cell. It holds default values. At run time, Fabric adds a new cell right below it that sets the values the pipeline passed, so the pipeline's values win. With no marked cell, the activity's Base parameters don't fill in for you. Always mark one.
Parameters support simple types: int, float, bool and string. Pass a list as a JSON string and parse it inside.
To send a result back, the notebook calls notebookutils.notebook.exit("value"). The activity returns that exit value to the pipeline.
A session tag under Advanced settings lets several Notebook activities reuse one Spark session. It needs high concurrency mode for pipelines turned on in the workspace settings.
notebookutils: notebooks that run notebooks
Exam depth: know it cold.
notebookutils is a built-in package in Fabric notebooks. Its old name was MSSparkUtils, and old code still works. Its notebook module runs other notebooks.
notebookutils.notebook.run("name", 90, {"city": "Oslo"})runs one notebook and waits;90is the timeout in seconds. It returns the child's exit value, always as a string. The child runs on the caller's Spark pool.notebookutils.notebook.runMultiple(dag)runs many notebooks at once, in one Spark session. A DAG (directed acyclic graph) lists each notebook and which ones it must wait for, in itsdependencies.
Defaults to know for runMultiple: whole DAG timeout 12 hours, per-cell timeout 90 seconds, and up to 3 times the CPU core count running at once. Call validateDAG(dag) first to catch mistakes.
Decision rule: one child, run. Many children in parallel or with an order, runMultiple.
dag = {"activities": [{"name": "clean_oslo", "path": "clean_trips","args": {"city": "Oslo"}},{"name": "clean_lisbon", "path": "clean_trips","args": {"city": "Lisbon"}},{"name": "build_gold", "path": "load_fact_trip","dependencies": ["clean_oslo", "clean_lisbon"],"retry": 1},],"timeoutInSeconds": 3600,}notebookutils.notebook.validateDAG(dag)results = notebookutils.notebook.runMultiple(dag)print(results["build_gold"]["exitVal"])
You are a data engineer at Northwind Rides. A parent notebook must run 40 small notebooks, one per city, as fast as possible, then one final notebook after all 40 finish. They should share one Spark session to avoid startup time. What should you use?
Pick one answer.
Apache Airflow job
Exam depth: recognize it.
An Apache Airflow job is a Fabric item that runs Apache Airflow for you. Apache Airflow is an open-source tool where you write workflows as Python DAGs. It's code-first orchestration.
Pick it when the team already writes Airflow DAGs, or wants to orchestrate in Python. Pick a pipeline when the team wants no-code or low-code orchestration.
It's the next generation of Azure Data Factory's Workflow Orchestration Manager. Its workspace settings come back in lesson 18.
Try it in Fabric
A free Fabric trial is enough for this.
- In your workspace, create a Notebook. In the first cell, type
city = "Oslo". - On that cell's toolbar, select ... (More commands), then Toggle parameter cell.
- In a second cell, type
print(city). Save the notebook. - In the workspace, select + New item, then search for and select Pipeline. Name it and select Create.
- On the Activities tab, select Notebook to add it to the canvas.
- Select the activity. On Settings, pick your workspace and notebook.
- Expand Base parameters. Set
cityto Lisbon. - On the Home tab, select Save, then Run.
- Watch the run in the Output tab below the canvas. When it succeeds, the notebook used Lisbon, the pipeline's value, not its default Oslo.
Exam-style practice
You are a data engineer at Northwind Rides. Northwind buys a delivery startup whose engineers run 60 workflows as Apache Airflow DAGs written in Python. They must move the workflows into Fabric with the least rework. What should they use?
Pick one answer.
You are a data engineer at Northwind Rides. A pipeline's Notebook activity passes base parameter run_date. The notebook's first cell sets run_date = "2026-01-01", and every run uses that date instead of the pipeline's value. You need the pipeline's value to apply. What should you do?
Pick one answer.
Worth keeping
- Pipeline: low-code steps of many item types, copy at scale, built-in retries.
- Notebook: complex Spark code; runMultiple runs many notebooks in one session.
- Often both: a pipeline's Notebook activity runs notebooks with parameters.
- Parameters reach a notebook only through its marked parameters cell.
- Apache Airflow job: recognize it as code-first Python orchestration.
On the job
Priya and Sam settle on both.
A pipeline called nightly_load is the one thing anyone schedules or reruns. Its first activity copies new payments from SQL Server with the watermark from lesson 12. Next, a Notebook activity runs clean_all_cities.
That notebook is Sam's part. It builds a DAG of 40 city notebooks and runs them with runMultiple in one Spark session. That takes minutes, not the hour 40 separate sessions would.
Back in the pipeline, a Stored procedure activity runs the SCD Type 2 load in the Warehouse. Every activity's failure path leads to one Teams activity that posts the error to the ops channel.
The ops team can read the canvas and rerun a failed step. The engineers keep the heavy logic in code.
nightly_load is built, but someone still has to press Run at 2 a.m. And the partner's CSV doesn't always land on time. Next lesson: schedules, and triggers that start the pipeline the moment the file arrives.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.
Sources for this lesson (11)
Each fact in the lesson is marked with the number of the page it comes from.
- [4]Microsoft Learn: Study Guide for Exam DP-700
- [54]Microsoft Learn: Develop, execute, and manage Microsoft Fabric notebooks
- [70]Microsoft Learn: Microsoft Fabric decision guide: Choose a data integration strategy
- [106]Microsoft Learn: Pipeline overview
- [107]Microsoft Learn: Activity overview
- [108]Microsoft Learn: Transform data with a ForEach activity
- [109]Microsoft Learn: Prepare and transform data in Microsoft Fabric
- [110]Microsoft Learn: Transform data by running a notebook
- [111]Microsoft Learn: NotebookUtils (former MSSparkUtils) for Fabric
- [112]Microsoft Learn: NotebookUtils notebook run and orchestration
- [113]Microsoft Learn: What is an Apache Airflow job?