This project demonstrates a complete, end-to-end, real-time data pipeline built using core Microsoft Azure services. The objective was to simulate a live stream of transportation data, ingest it, process it in real-time, store it in a data lake, and visualize the final output.
The pipeline follows a classic real-time data processing architecture:
- Data Producer: A Python script reads from a local CSV file and streams each row as a JSON event.
- Ingestion: Azure Event Hub acts as the high-throughput, real-time message ingress point.
- Processing: An Azure Stream Analytics job subscribes to the Event Hub, running a simple query to process the data in-flight.
- Storage: The processed data is continuously written to an Azure Data Lake Storage (ADLS) Gen2 container as JSON files.
- Visualization: Power BI connects directly to the ADLS Gen2 container to model and visualize the processed streaming data.
- Cloud Platform: Microsoft Azure
- Ingestion: Azure Event Hub
- Stream Processing: Azure Stream Analytics
- Storage: Azure Data Lake Storage Gen2
- Data Producer: Python (
azure-eventhubSDK) - BI & Visualization: Power BI
The final result is a Power BI dashboard that visualizes the cleaned and processed data from the Azure Data Lake.
- Hands-on experience with provisioning and configuring core Azure streaming services.
- Practical application of building a data producer in Python to stream data to an Event Hub.
- In-depth troubleshooting of a real-world cloud pipeline, including debugging authentication errors (InvalidSignature), connection issues, and data format mismatches between services.