Colombia Open Data Pipeline

DATA ENGINEERING · CASE 01

Colombia Open Data Pipeline

Daily ingestion from datos.gov.co, layered data models and automated quality tests.

problem

Public datasets may arrive dirty, in different formats and without quality controls. [Adjust with the real case.]

solution

A reproducible pipeline with scheduled ingestion, bronze / silver / gold layers and tests that stop invalid data.

result

[Real measurable result: time saved, tables delivered, refresh frequency.]

pipeline-datos-co/
├─ ingest/
│  └─ socrata.py
├─ models/
│  ├─ bronze/
│  ├─ silver/
│  └─ gold/
├─ tests/
└─ .github/workflows/

def extraer(dataset: str, limite: int = 50_000):
    url = f"{BASE}/{dataset}.json"
    datos = requests.get(url, params={"$limit": limite}).json()
    df = pd.DataFrame(datos)
    df.to_parquet(f"bronze/{dataset}/{hoy()}.parquet")
    return len(df)