Black Friday50% off your first month on any monthly plan with EARLY50 See plans
All 33 lessons
  1. Start here: cloud and data basics

    1. 1. Start here: the exam, the plan and what you already know
    2. 2. Cloud in plain words: SaaS, PaaS, IaaS and where Fabric fits
    3. 3. Data basics: files, tables, formats and the lakehouse idea
    4. 4. A tour of Microsoft Fabric: OneLake, workspaces and items
    5. 5. The three languages: SQL, PySpark and KQL side by side
    6. 6. Getting data in: the map of Fabric tools
  2. Store, ingest and transform batch data (exam area 2, 30-35%)

    1. 7. Choosing a data store and a transformation tool
    2. 8. Shortcuts and mirroring: use data without copying it
    3. 9. Ingest data with pipelines and Copy job
    4. 10. Transform with PySpark and SQL: join, denormalize, aggregate
    5. 11. Messy data: duplicates, missing values and late arrivals
    6. 12. Full and incremental loads, and loading a star schema
  3. Orchestrate and stream (exam areas 1 and 2)

    1. 13. Pipeline or notebook? Choosing how to orchestrate
    2. 14. Schedules and event-based triggers
    3. 15. Parameters and dynamic expressions
    4. 16. Streaming in Fabric: Eventstreams, Spark streaming and KQL
    5. 17. KQL and windowing functions for real-time data
  4. Set up and secure a Fabric solution (exam area 1, 30-35%)

    1. 18. Workspace settings: Spark, domains, OneLake and data workflows
    2. 19. Version control with Git and database projects
    3. 20. Deployment pipelines: development, test and production
    4. 21. Who can see what: workspace roles and item permissions
    5. 22. Fine-grained security: rows, columns, objects, files and masking
    6. 23. Governance: sensitivity labels and endorsement
  5. Monitor, fix and optimize (exam area 3, 30-35%)

    1. 24. Monitoring hub, alerts and Activator
    2. 25. Fixing errors in pipelines, dataflows and notebooks
    3. 26. Fixing errors in Eventhouse, Eventstream and T-SQL
    4. 27. Optimize lakehouse tables: V-Order, OPTIMIZE, VACUUM and partitions
    5. 28. Optimize Spark, pipelines and warehouses
    6. 29. Optimize eventstreams, eventhouses and queries
  6. Exam practice and final review

    1. 30. Exam day: question types, timing and how to think
    2. 31. Mock exam 1
    3. 32. Mock exam 2
    4. 33. Final review: every decision on one page
See the leaderboard

20 min

Streaming in Fabric: Eventstreams, Spark streaming and KQL

Pick the right streaming engine and load streaming data into Fabric.

By the end of this lesson you can

  • Name each Real-Time Intelligence item and what it does
  • Configure an Eventstream's sources, transformations and destinations
  • Read a Spark structured streaming job and spot a missing checkpoint
  • Choose Eventstream, Spark structured streaming or KQL for a scenario

From earlier lessons · 1 of 2

Inside a ForEach that loops over a Lookup's rows, which expression reads the current row's source_table column?

Pick one answer.

From earlier lessons · 2 of 2

A pipeline must start every time a CSV file lands in a Lakehouse folder. What runs it?

Pick one answer.

Every Northwind Rides scooter sends its position and battery level every few seconds. Today those events pile up in a file, and a nightly pipeline loads them. The ops team wants a live map of low-battery scooters, updated within seconds. The analysts still want every trip in a Delta table. One stream, two very different needs.

[4]

Real-Time Intelligence: the map

Exam depth: know it cold.

Streaming data is data that keeps arriving as a flow of small events, instead of sitting in a finished file. Real-Time Intelligence is the part of Fabric built for it. It reacts to events as they happen, rather than on a schedule.

It has five pieces you must know:

  • Real-Time hub: the tenant-wide catalog of every stream and event you can use.
  • Eventstream: brings events in, shapes them, and routes them to destinations, with no code.
  • Eventhouse: a database built for time-stamped events. It holds one or more KQL databases.
  • KQL database: the tables inside an Eventhouse, queried with KQL (Kusto Query Language).
  • Activator: watches data and acts when a condition is met, such as sending an alert.

Recognize two more: a KQL queryset saves KQL queries, and a Real-Time dashboard shows live KQL results as charts.

[41][48][116][118][124]

Rendering diagram…
Northwind's scooter stream: one Eventstream feeds a live store, a Delta table and an alert engine at once.

[48][124]

Eventstream: bring events in without code

Exam depth: know it cold.

An Eventstream is a Fabric item with three parts on one canvas: sources, optional transformations, and destinations. You build it by dragging boxes, not by writing code.

Source familyExamples
Azure messaging.Azure Event Hubs, Azure IoT Hub, Azure Service Bus, Azure Event Grid.
Custom endpoint.Your own app, or a Kafka client, sends events using a connection string.
Database CDC.Azure SQL Database, Azure SQL Managed Instance, SQL Server on a VM, PostgreSQL, MySQL, Azure Cosmos DB.
Other clouds and Kafka.Amazon Kinesis, Google Cloud Pub/Sub, Apache Kafka, Confluent Cloud.
Fabric and Azure events.OneLake, job and workspace item events, and Azure Blob Storage events.
Testing.Sample data such as Bicycles or Yellow Taxi.

Azure Event Hubs and Azure IoT Hub are Azure services that collect events from apps and devices. Apache Kafka is a widely used open-source streaming platform.

CDC (change data capture) is a feed of every row inserted, updated or deleted in a database. A CDC source first copies a snapshot of the data, then streams each change.

Each Eventstream also offers a Kafka endpoint, so apps that already speak the Kafka protocol can send to it or read from it.

[48]

Match

Match each Eventstream transformation to what it does

Pick an item on the left, then its match on the right.

[48]

Exam depth: know it cold.

An Eventstream doesn't store data for you to query. It passes events through and forwards them to destinations:

DestinationUse it to
Eventhouse.Land events in a KQL table for queries in seconds.
Lakehouse.Write events as Delta tables, for Spark and SQL.
Activator.Act on conditions, such as an alert on low battery.
Custom endpoint.Send events to an app outside Fabric.
Derived stream.Publish a transformed stream that others can reuse from Real-Time hub.
Spark Notebook (preview).Process events with a structured streaming job.

The Eventhouse destination has two ingestion modes. Direct ingestion loads events as they are, with no transformations. Event processing before ingestion runs your transformations first.

You can attach several destinations to one Eventstream. Each gets the same events without interfering with the others. A SQL operator (preview) also lets you write transformations in SQL instead of boxes.

[48][67][132]

An Eventstream canvas. A bike sample source and a custom endpoint source feed one stream, which branches through Manage fields, Filter and Group by (tumbling window) nodes to a derived stream, a KQL destination, a lakehouse, an Activator and a custom endpoint.
Microsoft Learn, CC BY 4.0

[48]

[48][132][133]

You are a data engineer at Northwind Rides. Scooter events arrive from Azure IoT Hub. The ops team wants only events with battery under 15% in an Eventhouse table. The team doesn't write code. What should you configure?

Pick one answer.

[48][132]

Spark structured streaming

Exam depth: know it cold.

Structured streaming is Spark's engine for streams. It treats a live stream as a table that keeps getting new rows. You write the same DataFrame code as for batch, but start with readStream and end with writeStream.

It reads from Azure Event Hubs, Kafka, an Eventstream (through its Kafka endpoint), new files in a folder, or a Delta table. Spark processes events in micro-batches: small groups handled one after another.

PieceWhat it does
format("delta") and toTable("x").Write into a Delta table in the Lakehouse.
checkpointLocation.A folder where Spark saves its progress. After a restart, it resumes from there. Use one per query, never shared.
outputMode("append").Write only new rows.
trigger(processingTime="1 minute").Gather a minute of events per write, which means fewer, larger files.
trigger(availableNow=True).Process everything waiting, then stop. Put it on a schedule for cheap incremental loads.

[136][137][138]

python
from pyspark.sql import functions as f
# Read: a live stream of trip events
raw = (
spark.readStream
.format("eventhubs")
.options(**eh_conf)
.load()
)
trips = raw.select(f.col("body").cast("string").alias("event_json"))
# Write: append each micro-batch to a Delta table
query = (
trips.writeStream
.format("delta")
.option("checkpointLocation", "Files/checkpoints/trip_events")
.outputMode("append")
.trigger(processingTime="1 minute")
.toTable("trip_events")
)
query.awaitTermination()
Read scooter events from Event Hubs and append them to a Delta table every minute. eh_conf holds the connection settings.

[136][137]

[136][138][139]

A Northwind Rides streaming job writes scooter events to a Delta table. After a maintenance restart, it can't pick up where it stopped. Its writeStream sets the Delta output, outputMode("append") and toTable("trip_events"). Which missing line fixes it?

Pick one answer.

[137][138]

You are a data engineer at Northwind Rides. Station sensor files arrive a few times an hour. Data an hour old is fine, and cost matters. You want to keep your structured streaming code and its progress tracking. What should you do?

Pick one answer.

[137]

Eventstream routes it. Spark rewrites it. KQL answers it. Picture: Movie-poster parody, bold illustrated style.

KQL in an Eventhouse: store, query, transform

Exam depth: know it cold.

An Eventhouse is where streaming data lands for real-time queries. Data is organized by arrival time, so KQL queries over billions of events return in seconds. It takes events from an Eventstream, or straight from Event Hubs, OneLake or files through Get data.

KQL processes data in two places:

  • At query time: filter, aggregate and join as you read. Lesson 17 teaches these queries.
  • On arrival: an update policy (recognize it) runs a KQL query whenever new rows land in a source table. It writes the result to a target table in the same database. This is how you clean raw events inside the Eventhouse, with no pipeline.

Turn on OneLake availability to let other Fabric engines read Eventhouse tables. It's opt-in and adds some delay.

[41][43][67][124][140]

Choosing the streaming engine

Exam depth: know it cold.

EventstreamSpark structured streamingKQL in an Eventhouse
Code?None: drag and drop.PySpark, Scala or Spark SQL.KQL.
Best atIngest, simple shaping, routing to many places.Complex logic, libraries, machine learning, existing Spark code.Fast queries on recent events; complex transforms on arrival.
Lands data inEventhouse, Lakehouse, Activator and more.Delta tables in a Lakehouse.KQL tables in the Eventhouse.

Decision rule: no code, or many destinations, means Eventstream. Spark code, or a Delta table as the goal, means structured streaming. Querying events within seconds means KQL in an Eventhouse.

They often work together. In a common loading pattern, the Eventstream does simple filtering on the way in. It sends events to an Eventhouse for live queries and to a Lakehouse for history. Heavier cleanup then runs as an update policy inside the Eventhouse.

[41][48][68][136][141]

You are a data engineer at Northwind Rides. The data science team has PySpark code that scores each trip for fraud with a Python library. It must run on the live trip stream from Azure Event Hubs and write results to a Lakehouse Delta table. What should you use?

Pick one answer.

[136][137]

Try it in Fabric

A free Fabric trial is enough. First create an Eventhouse: in your workspace, select + New item, search for Eventhouse, enter a name and select Create. Fabric also creates a KQL database with the same name.

  1. Select + New item, search for Eventstream, and select it. Enter a name and select Create.
  2. On the get-started page, select Use sample data. Name the source, pick Bicycles, and select Add.
  3. Select Add destination on the ribbon, then Eventhouse. Choose Direct ingestion.
  4. Enter a Destination name, pick your workspace and Eventhouse, and select Save.
  5. Connect the destination to the stream if it isn't already, then select Publish.
  6. In Live view, select Configure on the Eventhouse node. Choose New table, name it, and follow the screens to Finish.
  7. Open the KQL database and watch the new table fill with bike events.

[132][134][135][303]

Exam-style practice

You are a data engineer at Northwind Rides. An Eventstream receives scooter events. Ops staff must query the last 15 minutes of events and get answers in seconds, all day long. Where should the Eventstream send the events?

Pick one answer.

[41][67]

You are a data engineer at Northwind Rides. Whenever a scooter's battery drops below 10%, the field team must get a Teams message within seconds. Events already flow through an Eventstream. You want the least effort and no code. What should you add?

Pick one answer.

[48][116]

You are a data engineer at Northwind Rides. Analysts notice that a few trip events appear twice in the Eventhouse table fed by an Eventstream. Nothing is misconfigured. What explains it?

Pick one answer.

[48]

FlashcardsIndex cardsFlashcards ⚡

Card 1 of 15

[41][48][124][132][136][137][140]

Worth keeping

  1. Real-Time Intelligence is Real-Time hub, Eventstream, Eventhouse, KQL database and Activator.
  2. Eventstream ingests, shapes and routes events with no code, to many destinations at once.
  3. Structured streaming runs Spark code on streams; a checkpoint lets it resume after a restart.
  4. Streaming data lands in an Eventhouse for queries in seconds, and in a Lakehouse for Delta history.
  5. No code: Eventstream. Spark code or Delta: structured streaming. Fast queries: KQL.

On the job

Here's Northwind Rides' scooter stream after the rebuild.

Scooters send events to Azure IoT Hub. One Eventstream reads them and has three destinations. The first is an Eventhouse, through a Filter that drops test scooters, with event processing before ingestion. The second is a Lakehouse table that keeps every raw event as Delta. The third is an Activator rule that messages the field team when a battery drops below 10%.

Inside the Eventhouse, an update policy parses each raw event into clean columns. The ops team's live map reads that clean table with KQL every few seconds.

The data science team runs its fraud model as a Spark job definition with structured streaming. It reads the same trips and writes scores to a Delta table, with a retry policy so it survives restarts.

One stream, three engines, each doing the part it's best at.

[48][136][140]

The trips now land in an Eventhouse, one row per event. Then ops asks: how many trips started at each station in every 5-minute window? Next lesson you answer it with KQL's summarize and bin(), and meet the window types the exam loves to test.

Leaderboard

Loading…

50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.

Sources for this lesson (20)

Each fact in the lesson is marked with the number of the page it comes from.

  1. [4]Microsoft Learn: Study Guide for Exam DP-700
  2. [41]Microsoft Learn: Eventhouse overview
  3. [43]Microsoft Learn: Choose an analytical data store in Microsoft Fabric
  4. [48]Microsoft Learn: Overview of Microsoft Fabric eventstreams
  5. [67]Microsoft Learn: Get data into Microsoft Fabric
  6. [68]Microsoft Learn: Microsoft Fabric decision guide: copy activity, Copy job, dataflow, Eventstream, or Spark
  7. [116]Microsoft Learn: What is Fabric Activator?
  8. [118]Microsoft Learn: Introduction to Fabric Real-Time hub
  9. [124]Microsoft Learn: What is Real-Time Intelligence?
  10. [132]Microsoft Learn: Add an Eventhouse destination to an eventstream
  11. [133]Microsoft Learn: Process event data by using the event processing editor
  12. [134]Microsoft Learn: Create an eventstream in Microsoft Fabric
  13. [135]Microsoft Learn: Add a sample data source to an eventstream
  14. [136]Microsoft Learn: Data streaming into a lakehouse with Spark
  15. [137]Microsoft Learn: Overview of Structured Streaming
  16. [138]Microsoft Learn: Best practices for Structured Streaming
  17. [139]Microsoft Learn: Get streaming data into lakehouse and access with SQL analytics endpoint
  18. [140]Microsoft Learn: Update policy overview
  19. [141]Microsoft Learn: Tutorial: Route data using table update policies
  20. [303]Microsoft Learn: Create an eventhouse