Track 8 · Lesson 10 of 11

Small data pipelines

Structure file-processing programs as a pipeline of small functions: read, clean, transform and write, each testable on its own.

You will be able to

• Split a file-processing job into read, clean, transform and write stages, one function each
• Write a clean function that normalises one record or rejects it
• Combine the stages in a run function that reports how many records were used and skipped
• Make a pipeline safe to re-run: same input, same output