WORKER PROCESS POSTGRES DB EXTERNAL APIs READ STATE HTTP GET RESPONSE WRITE COMMIT HTTP POST RESPONSE PYTHON RUNNER STEP 1 FETCH DATA STEP 2 PROCESS & SEND CHECKPOINTS TABLE STEP 1: PENDING STEP 2: PENDING DATA API /v1/records INGEST API /v2/submit