Black Friday50% off your first month on any monthly plan with EARLY50 See plans
All 33 lessons
  1. Start here: cloud and data basics

    1. 1. Start here: the exam, the plan and what you already know
    2. 2. Cloud in plain words: SaaS, PaaS, IaaS and where Fabric fits
    3. 3. Data basics: files, tables, formats and the lakehouse idea
    4. 4. A tour of Microsoft Fabric: OneLake, workspaces and items
    5. 5. The three languages: SQL, PySpark and KQL side by side
    6. 6. Getting data in: the map of Fabric tools
  2. Store, ingest and transform batch data (exam area 2, 30-35%)

    1. 7. Choosing a data store and a transformation tool
    2. 8. Shortcuts and mirroring: use data without copying it
    3. 9. Ingest data with pipelines and Copy job
    4. 10. Transform with PySpark and SQL: join, denormalize, aggregate
    5. 11. Messy data: duplicates, missing values and late arrivals
    6. 12. Full and incremental loads, and loading a star schema
  3. Orchestrate and stream (exam areas 1 and 2)

    1. 13. Pipeline or notebook? Choosing how to orchestrate
    2. 14. Schedules and event-based triggers
    3. 15. Parameters and dynamic expressions
    4. 16. Streaming in Fabric: Eventstreams, Spark streaming and KQL
    5. 17. KQL and windowing functions for real-time data
  4. Set up and secure a Fabric solution (exam area 1, 30-35%)

    1. 18. Workspace settings: Spark, domains, OneLake and data workflows
    2. 19. Version control with Git and database projects
    3. 20. Deployment pipelines: development, test and production
    4. 21. Who can see what: workspace roles and item permissions
    5. 22. Fine-grained security: rows, columns, objects, files and masking
    6. 23. Governance: sensitivity labels and endorsement
  5. Monitor, fix and optimize (exam area 3, 30-35%)

    1. 24. Monitoring hub, alerts and Activator
    2. 25. Fixing errors in pipelines, dataflows and notebooks
    3. 26. Fixing errors in Eventhouse, Eventstream and T-SQL
    4. 27. Optimize lakehouse tables: V-Order, OPTIMIZE, VACUUM and partitions
    5. 28. Optimize Spark, pipelines and warehouses
    6. 29. Optimize eventstreams, eventhouses and queries
  6. Exam practice and final review

    1. 30. Exam day: question types, timing and how to think
    2. 31. Mock exam 1
    3. 32. Mock exam 2
    4. 33. Final review: every decision on one page
See the leaderboard

21 min

Fixing errors in pipelines, dataflows and notebooks

Read the error, find the cause and fix failing pipelines, dataflows and notebooks.

By the end of this lesson you can

  • Find the failed activity and its error in a pipeline run
  • Diagnose Dataflow Gen2 staging, gateway and credential errors
  • Debug a failing notebook from its cell error, driver logs and Spark UI
  • Work through a troubleshooting checklist from symptom to fix

From earlier lessons · 1 of 2

Where do you see the latest status of every pipeline, notebook and dataflow you can access, in one list?

Pick one answer.

From earlier lessons · 2 of 2

You run a pipeline by hand and it fails. It has schedule failure emails set up. Do you get an email?

Pick one answer.

Your new alerts work. This week at Northwind Rides they fired three times. The trips pipeline failed. The payments dataflow failed. A cleaning notebook failed. Each alert says only Failed. Your job now is to turn Failed into a cause and a fix, fast.

[4]

The method: symptom, where to look, fix

Exam depth: know it cold.

Every fix in this lesson follows three steps.

  1. Symptom. What failed, and what does the error say? Copy the exact message and code.
  2. Where to look. Go one level deeper than the run: the failed activity, table or cell.
  3. Fix. Ask if the error is transient, meaning it goes away on its own, like a brief network drop. If so, a retry helps. If it's permanent, like a wrong password or a missing column, fix the cause first.

The first stop is always the Monitor hub. Filter Job runs by failed status and open the run.

[221]

Pipelines: find the failed activity

Exam depth: know it cold.

A pipeline fails because one of its activities failed. So find that activity.

Select the run's name in the Monitor hub. The run detail view shows the pipeline diagram, the run ID, errors and a list of Activity runs. Filter that list by status to see only failures.

Each activity run has input and output links. The activity output is what the activity returned when it ran. For a failed activity, it holds the error. In the pipeline editor, the same details appear on the Output tab below the canvas.

After the fix, select Rerun. You can rerun the whole pipeline, start from the failed activity, or start from an activity you pick.

[224][233]

A pipeline run detail view. The diagram shows one Copy data activity. Below it, the Activity runs list shows activity name, status, type, start, duration and run ID. The toolbar has Rerun, Cancel, Refresh and Update pipeline, and a side pane shows run details.
Microsoft Learn, CC BY 4.0

[224]

Three real pipeline errors, read as symptom, cause and fix:

  • Symptom: The provided grant has expired due to it being revoked, a fresh auth token is needed. The user might have changed or reset their password. Cause: the pipeline's stored sign-in token stopped working. Fix: update and save the pipeline, for example its description, to refresh its sign-in.
  • Symptom: the result of the evaluation of 'foreach' expression '@pipeline().parameters.<parameterName>' is of type 'String'. The result must be a valid array. Cause: a parent pipeline passed an array to a child pipeline, and it arrived as text. Fix: wrap the value in the create array function.
  • Symptom: Please provide value for the required property '%propertyName;'. Cause: a required setting is empty. Fix: fill in the named property.

[235]

Retry settings: for errors that fix themselves

Exam depth: know it cold.

A retry reruns a failed activity automatically before the run is marked as failed. Set it on the activity's General tab:

SettingWhat it doesDefault
Enable retriesTurns retries on.Off.
RetryHow many retries, 1 to 1000.1.
Retry interval (sec)Wait between attempts.30 seconds.
TimeoutLongest an activity may run.12 hours.

Retry interval type is Fixed or Increasing Delay, which waits longer each time. Retry conditions (preview) retry only on errors you name. Error code 429 means the source is rate limiting you.

Decision rule: retry transient errors like timeouts. Never retry your way out of a wrong password.

[107][234]

You are a data engineer at Northwind Rides. A Copy data activity calls a partner's REST API. About once a week it fails with error code 429, then works when rerun by hand. Sign-in failures must still fail at once. What should you do?

Pick one answer.

[234]

Dataflow Gen2: refresh history first

Exam depth: know it cold.

Open the dataflow's Recent runs and select a run's Start time. The details have two lists:

  • Tables: each query loaded into staging. Staging is an internal Lakehouse where Dataflow Gen2 parks query results to process them faster.
  • Activities: each action, such as writing to your destination. Each shows its status and rows or bytes written.

Open the failed table or activity to read its error. For deeper detail, select Download detailed logs. They stay available for 28 days.

Two more words matter here. A gateway is software in your own network that lets Fabric reach data behind your firewall. Credentials are the sign-in a connection uses.

[228][237][241]

SymptomCauseFix
An on-premises data gateway is required to connectThe source is on-premises and no gateway is selected.Select a gateway, or install one.
Invalid connection credentials, with Login failed for user 'username'.Wrong username or password.Update the connection's credentials.
The dataflow refresh failed due to insufficient permissions for accessing staging artifactsThe user who made the workspace's first dataflow hasn't signed in for 90 days, or left.That user signs in, or you open a support ticket.
A warning when you pick a Warehouse destinationStaging is off for that query.Enable staging. A Warehouse destination needs it.
the gateway version you're using isn't supportedThe gateway is out of date.Update the gateway. Only the last six versions are supported.

[236][237][239][240][242]

[236][238]

You are a data engineer at Northwind Rides. You copy a Power Query query that reads the on-premises SQL Server payments database into a Dataflow Gen2. It fails with "An on-premises data gateway is required to connect". A gateway is already installed. What should you do?

Pick one answer.

[239]

Retrying a wrong password 1000 times is not a retry policy. It's a lockout. Picture: Three-panel comic strip, clean cartoon style.

Notebooks: cell error, driver logs, Spark UI

Exam depth: know it cold.

A notebook runs on a Spark session: compute Fabric starts for you. Inside it, the driver is the process that runs your code and hands work to executors, the workers.

Look in this order:

  1. The failed cell. Run it interactively and the error shows right under the cell.
  2. Driver logs. In the Monitor hub, open the notebook's run and select the Logs tab, then Driver. Its stderr (error output) holds the Python error, such as ModuleNotFoundError.
  3. Spark UI. Spark's own monitoring pages. Open the run, select the Jobs tab, then a job's description. Stages shows slow or uneven tasks. Executors shows each worker's status and exit codes.

[243][247]

SymptomCauseFix
Your session timed out after inactivity.The session idled past its timeout, 20 minutes by default.Rerun. Raise the timeout on the notebook or in workspace Spark settings.
SparkContextInitializationTimedOutThe session couldn't start, often a busy capacity or heavy libraries.Wait for capacity, or trim custom libraries.
ModuleNotFoundError in driver stderrA library isn't installed.Install it in an environment attached to the notebook.
[UNRESOLVED_COLUMN.WITH_SUGGESTION]A column name is wrong. The error lists real ones.Fix the name.
A schema mismatch detected when writing to the Delta tableYour DataFrame's columns or types differ from the table's.Align columns and types, or add mergeSchema.
The capacity with ID <ID> is paused.An admin paused the capacity.Ask an admin to resume it.

[159][244][246][248]

[159]

python
from pyspark.sql.functions import col
# 1. Compare what the table expects with what you're writing
spark.sql("DESCRIBE trips").show()
df.printSchema()
# 2. Fix a type that drifted, e.g. duration arrived as a string
df = df.withColumn("duration_min", col("duration_min").cast("int"))
# 3. If the new column is intended, let Delta add it on append
(df.write.format("delta")
.option("mergeSchema", "true")
.mode("append")
.saveAsTable("trips"))

[244][301]

A troubleshooting checklist

Put in order

Put the steps for fixing any failed run in order

Drag, or focus an item and press Space, move with the arrow keys, then Space to drop.

  1. Filter Job runs in the Monitor hub to failed runs
  2. Fix the cause, or add a retry for a transient error
  3. Decide whether the error is transient or permanent
  4. Copy the exact error message and code
  5. Open the run and find the failed activity, table or cell
  6. Rerun from the failed step and confirm it succeeds

[221][224]

Exam-style practice

You are a data engineer at Northwind Rides. The nightly trips pipeline has eight activities and shows Failed in the Monitor hub. You need the exact error from the step that failed. What should you do?

Pick one answer.

[224]

You are a data engineer at Northwind Rides. A notebook starts with %pip install geopy and works when you run it. When the nightly pipeline runs it, the driver log shows ModuleNotFoundError: No module named 'geopy'. You need a stable fix for scheduled runs. What should you do?

Pick one answer.

[159]

You are a data engineer at Northwind Rides. The partner added a battery_level column to the scooter feed. Your notebook appends the data to the Delta table scooters and fails with "A schema mismatch detected when writing to the Delta table". The team wants to keep the new column. What should you add to the write?

Pick one answer.

[244][301]

You are a data engineer at Northwind Rides. You set a Dataflow Gen2 query to load stations into a Fabric Warehouse. Fabric shows a warning and won't save the destination. Staging is turned off for that query. What should you do?

Pick one answer.

[237]

FlashcardsIndex cardsFlashcards ⚡

Card 1 of 15

[107][159][224][228][234][236][237][243]

Worth keeping

  1. Go from the failed run to the failed activity, table or cell, then read the exact error.
  2. Retry transient errors; fix permanent ones like credentials, gateways and schemas.
  3. Dataflow Gen2 errors live in refresh history, under Tables and Activities.
  4. For notebooks, check the cell, then driver logs, then the Spark UI.
  5. Pipeline runs need libraries from an environment, not inline %pip.

On the job

Here's how the three Northwind Rides failures from the start of the lesson end.

The trips pipeline failed with an expired sign-in token. The engineer who built it had reset their password. You update and save the pipeline, then rerun it from the failed activity.

The payments dataflow showed every table as Succeeded, but the write activity failed. It reads the SQL Server database through a gateway, and the firewall blocked port 1433. The network team opens the port.

The cleaning notebook hit a schema mismatch. The partner had added a column. You check with ops, then append with mergeSchema.

All three were fixed before lunch.

[235][238][244]

Batch jobs are fixed. But the live scooter events stopped reaching the Eventhouse at noon, and the Warehouse is throwing a T-SQL error. Next lesson: errors in Eventstream, Eventhouse and T-SQL.

Leaderboard

Loading…

50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.

Sources for this lesson (22)

Each fact in the lesson is marked with the number of the page it comes from.

  1. [4]Microsoft Learn: Study Guide for Exam DP-700
  2. [107]Microsoft Learn: Activity overview
  3. [159]Microsoft Learn: Manage Apache Spark libraries in Microsoft Fabric
  4. [221]Microsoft Learn: Monitor job runs in the Monitor hub
  5. [224]Microsoft Learn: Monitor pipeline runs in the legacy Monitoring hub
  6. [228]Microsoft Learn: View refresh history and monitor your dataflows
  7. [233]Microsoft Learn: Use the Fail activity to cause pipeline execution to fail with a customized error message and error code
  8. [234]Microsoft Learn: Activity retries in Microsoft Fabric Data Factory
  9. [235]Microsoft Learn: Troubleshoot pipelines for Data Factory in Microsoft Fabric
  10. [236]Microsoft Learn: Dataflow refresh
  11. [237]Microsoft Learn: Dataflow Gen2 data destinations and managed settings
  12. [238]Microsoft Learn: On-premises and virtual network data gateway considerations for data destinations in Dataflow Gen2
  13. [239]Microsoft Learn: Troubleshooting dataflow issues: Connection to the data source
  14. [240]Microsoft Learn: Troubleshoot on-premises data gateways
  15. [241]Microsoft Learn: On-premises data gateway FAQ
  16. [242]Microsoft Learn: Troubleshoot refresh scenarios
  17. [243]Microsoft Learn: Spark errors overview in Microsoft Fabric
  18. [244]Microsoft Learn: Troubleshoot Spark SQL and schema errors
  19. [246]Microsoft Learn: Troubleshoot Spark session, configuration, and platform errors
  20. [247]Microsoft Learn: Troubleshoot Spark application code, data, and library errors in Microsoft Fabric
  21. [248]Microsoft Learn: Fabric notebooks troubleshooting guide
  22. [301]Microsoft Learn: Delta Lake schema evolution