Why I Started This Project

I wanted to build something that went beyond analyzing a CSV file.

I wanted to understand what happens when data moves through a complete system:

API → Extraction → Transformation → Database → Visualization

That was the original idea.

The actual project became much more complicated.

What I Thought I Was Building

Initially, I thought the project was simply:

Get data from an API. Clean it. Put it into PostgreSQL. Create a dashboard.

That sounds straightforward.

But I quickly realized that the interesting problems were not necessarily the individual technologies.

The difficult questions were:

What data should I store? How should I structure the database? What happens when the same player changes teams? How do I handle historical data? What happens if the API request fails? How should the pipeline run automatically? What I Got Wrong

My first instinct was to focus on adding more technologies.

I wanted to use Python, PostgreSQL, Airflow, Docker, Power BI, and everything else immediately.

But adding technologies does not automatically create a better system.

The more important question is:

What problem does each component solve?

That changed how I approached the architecture.

What I Would Do Differently

If I rebuilt the project today, I would spend more time designing the data model before writing the extraction code.

I would also separate the pipeline into clearer stages:

Extract
↓
Validate
↓
Transform
↓
Load
↓
Analyze

The Biggest Lesson

The biggest lesson was that building a data pipeline is not primarily about moving data from one place to another.

It is about designing a reliable process for data to move through a system.