Introduction The project is a robust streaming pipeline built with FastAPI, Kafka, Pyspark, Cassandra, Spacy, Redshift, and Grafana. It provides a scalable and efficient architecture for processing...
Introduction In the world of modern software development and data engineering, Continuous Integration and Continuous Deployment (CI/CD) have become indispensable practices. These methodologies not...
Introduction In the vast landscape of enterprise data management, it’s surprising to observe that a significant number of organizations operate without a robust data catalog system to effectively ...
Introduction In this personal project, I am using Scala to leverage Spark Structured Streaming. Read stream data from socket, using Netcat to create a local socket and input data Read stream ...
Introduction This personal project, I use PySpark on Databrick on Azure to build data warehouse with Medallion architecture Git repo: General concept Hive metastore Reference: Hive Tabl...
Introduction Do you feel sadness when users cannot visit your BI platforms? Do you experience a sense of inadequacy when your Analytics Application is unattractive and lacking in analytical capabi...
Introduction Project Architecture Data Architecture Design (by @Kato) Core Warehouse Design 1. Data Ingestion layer Ingestion framework:Polars with Python to process data on a single machine...
Great purposes: Developed a robust data collector using Streamlit application and a Cloud Database. Designed and managed a non-relational staging database to ingest data from Jira (Track recru...
Introduction This mini project, titled “Django - Real Estate Web Application,” is driven by my goal as a Data Engineer to gain a profound understanding of a robust backend framework, with a specifi...
Introduction Tech usages: Python, Flask, Polars In the world of Data Engineering and Business Intelligence, we encounter numerous instances of data issues on a daily basis. Unfortunately, we o...