Drawing from three principles of data science: predictability, computability, and stability (PCS), we propose the PCS framework to provide responsible, reliable, reproducible, and transparent results across the entire data science life cycle. It uses predictability as a reality check and considers the importance of computation in data collection/storage and algorithm design. We augment predictability and computability with an overarching stability principle. Stability expands on statistical uncertainty considerations to assess how human judgment calls impact data results through data and model/algorithm perturbations. We develop inference procedures that build on PCS, namely PCS perturbation intervals and PCS hypothesis testing, to investigate the stability of data results relative to problem formulation, data cleaning, and modeling decisions. We compare PCS inference with existing methods in high-dimensional sparse linear model simulations to demonstrate that our approach compares favorably to others in terms of ROC curves over a wide range of simulation settings. Finally, we propose PCS documentation based on R Markdown or Jupyter Notebook, with publicly available, reproducible codes and narratives to back up human choices made throughout an analysis. The PCS workflow and documentation are demonstrated in a genomics case study available on Zenodo.
View on arXiv