Research reproducibility checklist · Reviewed 25 October 2026
A chart is only as clear as the question, selection and calculation behind it. This guide gives a practical audit trail for a small local-data comparison. It does not claim that a dataset is complete or that an observed difference establishes a cause.
Save a selection recipe
Before changing filters, write the question in one sentence. Then record the dataset title, publishing agency, source link, retrieval date, field definitions and exact filters. Keep an untouched copy of the export separate from your working file. If the source changes, the original export explains what your published calculation used.
| Element | What to document |
|---|---|
| Observation unit | What one row represents and its identifier |
| Period | Chosen date field, boundaries and treatment of partial periods |
| Area | Exact geography and treatment of missing locations |
| Cleaning | Duplicate handling, missing values and excluded records |
| Calculation | Formula, denominator and rounding rule |
| Limitations | Coverage, reporting behavior and unresolved inconsistencies |
Check labels and coverage before joining files
NYC Open Data has documented changes to 311 labels and the division of current and historical request datasets. This illustrates why two similarly named downloads should not be combined without checking their coverage and structure. A renamed display column may not mean the underlying API field changed, and an older export may not include newly added fields.
Use a small calculation check
Suppose a fictional filtered file has 105 rows, of which five repeat identifiers already present. If the question counts distinct requests, the result is 100, assuming those identifiers correctly define the unit. If the question counts rows received, the answer remains 105. Neither number is meaningful without its definition.
Take a few records and trace them from the original export through the cleaning step into the result. Check that exclusions are intentional. Record whether blank values were removed, kept as unknown or assigned to a category. Never silently turn “unknown” into “none.”
Keep comparisons symmetrical
Use the same date field, category definitions and boundary method for both groups. Identify whether one period is incomplete. If you normalize counts, use a denominator with compatible geography and timing. Explain any mismatch instead of hiding it in a footnote or presenting extra decimal places as precision.
Write the conclusion last
Describe the result in terms of the defined measure. Then separate possible explanations from established facts. A difference in recorded requests could motivate further research; it does not by itself prove why the difference exists. Publish the selection recipe beside the result so another reader can challenge or repeat the analysis.
Primary reference: NYC Open Data: 311 Service Requests Updates. The recipe and arithmetic example are original editorial material; no live neighborhood statistics are asserted here.