datamanifest

Keep track of the datasets used in a scientific project. You declare each dataset in a manifest — a plain TOML file committed alongside your code — and three clients read and write it the same way: the datamanifest command line and the Python library (both in the PyPI package datamanifestpy), and the Julia package DataManifest.jl. The manifest format itself is defined by a shared, language-agnostic spec, so a project's manifest works across all three.

• A transparent, trackable manifest. Every dataset a project depends on — URLs, DOIs, checksums (content hashes used to verify each file), formats — is listed in a single datamanifest.toml file (the manifest) that you can read directly and version with git. The format is defined by a language-agnostic spec (implemented in Python and Julia) and can be edited by hand, from code, or through the CLI.
• Fetch from a wide range of sources. Direct URLs, Zenodo/figshare and PANGAEA DOIs, git repos, object stores (s3://, gs://, …), and bulk imports from pooch, intake or DVC — all checksum-verified, extracted, and adopted in place when already on disk.
• Cache your own computed data too. The same tooling backs a @cached decorator that stores your own results with PID-lock, keyed by their inputs, to speed up calculations locally. Caching is a separate, local concern — nothing is fetched — but cached results are managed through the same CLI as datasets.
• A CLI for data download, local management and synchronization across machines. Add and download datasets, inspect and repair what's on disk, move or centralize where data is stored, and push/pull datasets and cached results between machines over rsync+ssh — all without touching your analysis code. A git-ignored state file (.datamanifest/state.toml) records where each object actually landed on this machine, keeping local location tracking separate from the portable, shareable manifest.

Get started

Code examples across this site come in tabs — pick your client once (CLI, Python or Julia) and every page follows.

CLIPythonJuliaManifest

pipx install datamanifestpy
datamanifest init
datamanifest add https://gml.noaa.gov/webdata/ccgg/trends/co2/co2_annmean_mlo.csv --name co2
datamanifest path co2              # the on-disk path, downloaded and verified

# pip install datamanifestpy
import datamanifest

db = datamanifest.Database("datamanifest.toml")
db.add("https://gml.noaa.gov/webdata/ccgg/trends/co2/co2_annmean_mlo.csv", name="co2")
df = db.load_dataset("co2")             # downloaded and verified, then loaded

# using Pkg; Pkg.add("DataManifest")
using DataManifest

db = read_dataset("datamanifest.toml")
add(db, "https://gml.noaa.gov/webdata/ccgg/trends/co2/co2_annmean_mlo.csv"; name="co2")
path = get_dataset_path(db, "co2")      # the on-disk path, downloaded and verified

[co2]
checksum = "sha256:0058b3788040b5c27b2b5c1dd6d26226b7e4deef85e34c153e64806c37df7c75"
uri = "https://gml.noaa.gov/webdata/ccgg/trends/co2/co2_annmean_mlo.csv"

Three ways in, one manifest:

• Use it from the shell — install the CLI and follow the quickstart; the CLI reference covers every command and flag.
• Use it from Python — Using it from your code walks through load_dataset, the @cached decorator and the file-less Database; the Python API reference lists everything.
• Use it from Julia — the quickstart and guide pages carry Julia tabs for the essentials; the Julia API reference lists everything, and the Julia walkthrough lives in the DataManifest.jl repository.

This site documents the CLI and the Python library in full, and the essentials for Julia.

Guide

• Use cases — the CLI workflows end to end: add, repair, store, sync.
• Storage model — where data lives on disk and how to centralize it.
• Configuration — the config scopes and every setting.
• Adding datasets — direct URLs, Zenodo / PANGAEA DOIs, object stores.
• Importing from other tools — pooch, intake, DVC, CSV/URL lists.
• Language bindings — one manifest across Python and Julia.
• Related projects — the DataManifest family, and Python alternatives.

From the same author

A few other open-source tools I maintain.

Scientific writing & data

• texmark — write scientific articles in Markdown and convert them to journal-ready LaTeX/PDF.
• papers — command-line BibTeX bibliography and PDF library manager.

Speech to Text (dictate) and Text to Speech (read-aloud) tools

• scribe — speech-to-text dictation.
• bard — text-to-speech reader.

Development

• Conformance — the shared manifest format and what this implementation supports.
• Roadmap — parked ideas and deferred decisions.