API reference¶

Note

This page mirrors the DataManifest.jl API reference; the source of truth lives in the DataManifest.jl repository. It is regenerated from that repository on every deploy of this site.

This page lists the public functions and types with their signatures. The long-form walkthrough is in doc.md; the storage layout and configuration model in storage.md; produce-or-load caching in caching.md; loaders and fetchers in language-bindings.md.

Default database: wherever the db::Database argument is shown in brackets, it may be omitted. The default database is then used, which requires an activated Julia project with a datasets TOML file that exists or can be inferred.

Core¶

`Database`¶

Database(;
    datasets_toml::String="",
    datasets_folder::String="",
    persist::Bool=true,
    skip_checksum::Bool=false,
    skip_checksum_folders::Bool=false,
    datasets::Dict{String, DatasetEntry}=Dict{String, DatasetEntry}(),
    storage_config::Union{Nothing,AbstractDict}=nothing
) -> Database

Database(datasets_toml::String, datasets_folder::String=""; kwargs...) -> Database

Create a dataset database — the in-memory counterpart of a datamanifest.toml manifest.

If datasets_folder is not provided, the fetched-datasets folder is resolved via Storage.datasets_dir (default: the machine-global shared store, $user_data_dir/datamanifest/shared/datasets). To override, set the datasets_dir config field or the DATAMANIFEST_DATASETS_DIR environment variable, or pass datasets_folder= explicitly. See storage.md for the full resolution ladder.
If datasets_toml is not provided and persist is true, a TOML file is inferred from the project or environment.
If the TOML file exists, datasets are loaded from it.
The configuration (storage fields, environment, host) is frozen at construction; see freeze_config! to re-read it.
An in-memory database (persist=false: nothing is ever written back to a manifest, and db.datasets_toml == "") keeps its state-file inventories under the storage roots it describes — the datasets folder for fetched datasets, the resolved datacache_dir for produced artifacts — never under the caller's project or working directory.

Arguments: - datasets_toml::String: Path to the TOML file for persistence. - datasets_folder::String: Path to the datasets folder. - persist::Bool: Whether to persist changes to disk. - skip_checksum::Bool: Skip checksum verification. - skip_checksum_folders::Bool: Skip checksum verification for folders. - datasets::Dict{String, DatasetEntry}: Initial datasets. - storage_config: an optional Dict setting the manifest-layer [_STORAGE] table directly — the way to name an in-memory database's cache bundle, e.g. Database(datasets_folder=..., persist=false, storage_config=Dict("project" => "mylib")) puts produced artifacts under …/projects/mylib/cached. It overrides a [_STORAGE] table read from datasets_toml, sits below the checkout config and DATAMANIFEST_* environment variables, and is never written back to the manifest. See the db= option of @cached.

Returns: A Database object.

`read_dataset`¶

read_dataset(datasets_toml::String, datasets_folder::String=""; kwargs...) -> Database

Read a manifest TOML file into a Database. Equivalent to Database(datasets_toml, datasets_folder; kwargs...); keyword arguments are the same as for Database.

`add`¶

add([db::Database], uri::String=""; skip_download::Bool=false, kwargs...) -> Pair{String, DatasetEntry}

Add a dataset to the database and download it.

This is a convenience function that calls register_dataset to register the dataset, then download_dataset unless skip_download=true. All other keyword arguments are passed to register_dataset — see there for the list.

add_dataset is an alias for add.

Returns: A pair (name => entry) where entry is the registered DatasetEntry.

`download_dataset`¶

download_dataset([db::Database], name::String; extract::Union{Nothing,Bool}=nothing, overwrite::Bool=false, kwargs...) -> String
download_dataset([db::Database], entry::DatasetEntry; extract::Union{Nothing,Bool}=nothing, overwrite::Bool=false) -> String

Download a dataset by name or entry, and return the local path.

If the dataset is already present on disk, it is not downloaded again (unless overwrite=true).
If overwrite=true, skips existence checks and re-downloads (overwrites in place; git and extracted dirs require removal first).
If extract=true, the dataset is extracted after download (if applicable).
Checksum verification is performed unless disabled (see skip_checksum on the entry or the database).
Datasets listed in the entry's requires field are downloaded first.

Returns: The local path as a String.

`download_datasets`¶

download_datasets([db::Database], names::Union{Nothing,Vector{<:Any}}=nothing; kwargs...) -> Nothing

Download multiple datasets by name.

If names is nothing, all datasets in the database are downloaded.
Each dataset is downloaded using download_dataset, with the same keyword arguments.
Datasets already present on disk are not downloaded again.

Returns: Nothing.

`get_dataset_path`¶

get_dataset_path([db::Database], name::String; extract::Union{Nothing,Bool}=nothing) -> String
get_dataset_path([db::Database], entry::DatasetEntry; extract::Union{Nothing,Bool}=nothing) -> String

Return the local path for a dataset, based on its storage_path (or the keyed default $datasets_dir/$key) — see storage.md for path resolution.

You can provide either the dataset name or a DatasetEntry object.
The extract keyword controls whether the path points to the extracted folder (if applicable).

Returns: The local path as a String.

`load_dataset`¶

load_dataset([db::Database], name::String; loader=nothing, kwargs...)
load_dataset([db::Database], entry::DatasetEntry; loader=nothing, kwargs...)

Download a dataset if needed, then open it and return the loaded value.

loader may be a function path -> value, the name of a loader defined in the manifest, or a built-in format name (csv, parquet, nc, dimstack, md, txt, json, yaml, yml, toml, zip, tar, tar.gz).
When loader is not given, the loader is resolved from the manifest: the dataset's own loader binding, then the manifest's per-format loaders, then the built-in format default. See language-bindings.md.
For a dataset with lazy_access=true, the URI is opened in place by the loader (no download); a loader is required in that case.

`delete_dataset`¶

delete_dataset([db::Database], name::String; keep_cache::Bool=false, persist::Bool=true)

Remove a dataset from the database (and optionally from disk).

Removes the entry from the manifest TOML (or the in-memory db).
If keep_cache=false (default), also removes the dataset files/directories from disk. For extracted datasets, removes both the archive and the extracted directory.
If keep_cache=true, keeps the dataset on disk but removes the entry from the database.
Datasets with skip_download or lazy_access set, and datasets with a user-managed storage_path (an exact path without $key), are never removed from disk.

Searching¶

`search_datasets`¶

search_datasets([db::Database], name::String; alt=true, partial=false) -> Vector{Pair{String, DatasetEntry}}

Search for datasets in the database by name or alternative keys, returning all matches as a vector of (name => entry) pairs.

The search is case-insensitive and proceeds in the following order:

Exact match on dataset name (the main key in the database).
Exact match on alternative keys (if alt=true):
- Any value in the aliases field (alternative names).
- The doi field (if present).
- The key field (unique key for the dataset).
- The path field (the URI path).
- For a git-repository dataset, the bare repository name (e.g. repo for https://github.com/org/repo).
Partial match on dataset name (if partial=true): Checks if name is a substring of any dataset name.
Partial match on alternative keys (if alt=true and partial=true): Checks if name is a substring of any value in the alternative keys listed above.

All matches found in the above order are returned.

Arguments: - db::Database (optional): The database to search in. - name::String: The name, alias, DOI, key, or path of the dataset. - alt::Bool: Whether to search alternative keys (default: true). - partial::Bool: Whether to allow partial (substring) matches (default: false).

Returns: A vector of (name => entry) pairs for all matching datasets.

`search_dataset`¶

search_dataset([db::Database], name::String; raise=true, alt=true, partial=false) -> Tuple{String, DatasetEntry}

Search for a dataset by name or alternative keys in the database, returning the first match as a tuple (name, entry).

The search logic and field priority are the same as in search_datasets.
If no match is found and raise=true (default), an error is thrown. If raise=false, returns nothing.
If the identifier is ambiguous (it matches more than one dataset — e.g. a doi shared by several entries), an error is thrown naming the candidate datasets; disambiguate by exact name. With raise=false the first match is returned.

Returns: A tuple (name, entry) where entry is the found DatasetEntry.

Note: You can also access a dataset entry directly by name using indexing syntax:

entry = db["dataset_name"]

This is equivalent to search_dataset(db, "dataset_name")[2].

Registering and editing entries¶

`register_dataset`¶

register_dataset([db::Database], uri::String="";
    name::String="",
    overwrite::Bool=false,
    persist::Bool=true,
    check_duplicate::Bool=true,
    kwargs...
) -> Pair{String, DatasetEntry}

register_dataset([db::Database], uris::Vector{String}; ...) -> Pair{String, DatasetEntry}

Register a dataset in the database, without downloading it.

If name is not provided, it is inferred from the URI or dataset entry.
uri may be passed as a Vector{String} (equivalent to the uris= keyword) to register a multi-file dataset; the key is auto-derived from the common host + path prefix, or can be set explicitly via key=.
The remaining keyword arguments set DatasetEntry fields: version, branch, doi, aliases, key, checksum (a legacy sha256= hex value is also accepted), skip_checksum, skip_download, extract, format, storage_path, and so on.
Duplicate entries are checked by default; set check_duplicate=false to disable.
If an entry with the same name or key exists, it is updated or overwritten according to the overwrite flag.

Returns: A pair (name => entry) where entry is the registered DatasetEntry.

`DatasetEntry`¶

struct DatasetEntry
    uri::String = ""
    uris::Vector{String} = []     # Multi-file dataset: several URIs under one key
    version::String = ""
    branch::String = ""           # For git repositories
    doi::String = ""
    aliases::Vector{String} = []
    description::String = ""      # Free-text description
    key::String = ""              # Unique key for the dataset, usually the DOI or a unique name
    storage_path::String = ""     # Per-dataset path expression; empty = "$datasets_dir/$key"
    checksum::String = ""         # Expected content digest, "<algo>:<hex>"
    skip_checksum::Bool = false   # Skip checksum checks for this dataset
    skip_download::Bool = false   # Never download (user-managed or custom-fetched files)
    lazy_access::Bool = false     # Open the uri in place via a loader; no local copy
    extract::Bool = false         # Extract the dataset after downloading
    format::String = ""           # File format (e.g., "zip", "tar")
    shell::String = ""            # Run this shell command instead of the built-in download
    julia::String = ""            # Run this Julia code instead (takes precedence over shell)
    julia_modules::Vector{String} = []
    loader::String = ""           # Per-dataset loader (Julia shorthand)
    requires::Vector{String} = [] # Datasets that must be present first
    # Julia bindings, parsed from [<ds>._LANG.julia] (see language-bindings.md):
    lang_julia_fetcher::String = ""
    lang_julia_fetcher_args::Vector{Any} = []
    lang_julia_fetcher_kwargs::Dict{String,Any} = Dict()
    lang_julia_loader::String = ""
    lang_julia_loader_args::Vector{Any} = []
    lang_julia_loader_kwargs::Dict{String,Any} = Dict()
    extra::Dict{String,Any} = Dict()  # Unknown keys, preserved verbatim for round-trip
end

A DatasetEntry holds metadata and configuration for a dataset. It is initialized via the add method (and internally, register_dataset and init_dataset_entry).

Fields: - uri::String: The dataset URI (required, unless the dataset is produced by a fetcher or is user-managed). - uris::Vector{String}: URIs of a multi-file dataset, stored under one key. - version::String: Version or tag for the dataset. - branch::String: Branch for git repositories. - doi::String: DOI for the dataset. - aliases::Vector{String}: Alternative names for the dataset. - description::String: Free-text description. - key::String: Unique key for the dataset. - storage_path::String: A path expression for where this dataset lives, defaulting to $datasets_dir/$key. A value containing $key is a tool-managed keyed location; an exact path without $key is a user-managed location used verbatim (maintenance never touches it, and a relative path resolves against the project root). The expression may interpolate $-symbols, environment variables, and ~ — see storage.md. - checksum::String: Expected content digest as <algo>:<hex> (e.g. sha256:…, md5:…); a bare hex value is read as sha256, and the legacy sha256 key is still accepted. Empty means the checksum is computed on first download. Used for fetch-time verification and change detection. - skip_checksum::Bool: Skip checksum verification for this dataset. - skip_download::Bool: Skip downloading this dataset — for files that must never be fetched, or entries materialized by custom shell/julia code that does not need a download step. For pointing at a user-managed local file, prefer an exact storage_path. - lazy_access::Bool: Open the uri in place via a loader (typically a remote object store) instead of materializing a local copy — no download, no checksum. Requires a loader. See language-bindings.md. - extract::Bool: Extract the dataset after download. - format::String: File format (e.g., "zip", "tar"). - shell::String: When set, run this shell command instead of the built-in download. Template placeholders: $download_path, $project_root, $uri, $key, $version, $doi, $format, $branch, plus $requires_paths and $path_<n> for dependencies. The command runs with working directory = project root when available. - julia::String: When set, run this Julia code in an isolated module (takes precedence over shell). Use julia_modules to load modules before the code. The code sees download_path, project_root, entry, uri, key, version, doi, format, branch, and requires (same names as the shell template placeholders). - julia_modules::Vector{String}: Module names for using X before running julia. - loader::String: Per-dataset loader — a name, a module:function ref, or inline code. Shorthand for the Julia binding in language-bindings.md. - requires::Vector{String}: Datasets that must be present before this one; see below. - lang_julia_*: Julia fetcher/loader bindings parsed from [<ds>._LANG.julia], with optional args/kwargs payloads for parameterized bindings — see language-bindings.md. - extra::Dict{String,Any}: Unknown per-dataset keys, kept verbatim so the manifest round-trips losslessly.

Note: Fields such as host, path, and scheme are internal and not documented here.

`requires` (DatasetEntry field)¶

The requires field specifies other datasets that must be present before this one. Each entry is resolved by name, DOI, or key via search_dataset. When downloading, dependencies are fetched first (in topological order). overwrite applies only to the main dataset, not to dependencies. Circular dependencies raise an error.

TOML example:

[downstream]
uri = "https://..."
requires = ["upstream_dataset", "10.1594/PANGAEA.123456"]

`write`¶

write(db::Database, datasets_toml::String; canonical::Union{Bool,Nothing}=nothing)

Write the database to a manifest TOML file. Functions that modify the database call this automatically when persist=true; use it directly to write an in-memory database to disk.

The output is sorted: structural _* tables first, then datasets, both alphabetical.
canonical=true pipes the output through the Python datamanifest format CLI for cross-tool byte-identical files; canonical=false forces native TOML output. When the keyword is omitted (nothing), the canonical config field decides (default false) — resolved on the ordinary ladder, see Storage.canonical_write and the write section in doc.md. If the CLI is not found, native TOML is written with a warning.

Loaders and checksums¶

`validate_loader` / `validate_loaders`¶

validate_loader(db::Database, name::String) -> Function
validate_loaders(db::Database) -> Nothing

Loaders are compiled lazily, on first use — which avoids circular dependencies when a loader's modules depend on a package that itself uses DataManifest, but also means a broken loader only fails when a dataset is loaded. To compile and check loaders eagerly:

validate_loader(db, name) resolves the loader name (a named loader from the manifest, a Module:function reference, or inline code), compiles it, and returns the resulting function; it errors if the loader cannot be resolved or does not evaluate to a function.
validate_loaders(db) runs validate_loader on every named loader declared in the manifest.

`verify_checksum`¶

verify_checksum(db::Database, entry::DatasetEntry; persist::Bool=true, extract::Union{Nothing,Bool}=nothing, skip_if_complete::Bool=false)

Verify the on-disk data of a dataset against its checksum field. Called automatically around downloads; call it directly to re-check a dataset.

If the entry's checksum is empty, the checksum is computed (as sha256) and stored in the entry — and persisted to the manifest when persist=true.
A declared checksum is verified in its own algorithm and never silently rewritten; an algorithm this implementation cannot compute (e.g. md5) is preserved but skipped with a warning.
A mismatch raises an error listing the possible resolutions (remove the file, reset the checksum field, use a different key, or set skip_checksum on the entry or the database).
Verification is skipped when the data is not on disk yet, when skip_checksum is set (entry or database), for directories when the database's skip_checksum_folders is set, or — with skip_if_complete=true — when the entry has a checksum and the data's completion marker is present.

Configuration¶

`freeze_config!`¶

freeze_config!(db::Database) -> db

Re-read a Database's frozen configuration snapshot.

Configuration (the checkout config .datamanifest/config.toml, the manifest's [_STORAGE], the user-global config, the environment, and the host name) is captured once, when the Database is created, so every config variable has one well-defined value for the Database's lifetime. Call freeze_config!(db) to re-read the config files and environment for an existing Database — e.g. after editing a config file. See storage.md for the resolution ladder.

Assigning db.storage_config or db.datasets_toml invalidates the snapshot; it is re-frozen (from the then-current files and environment) on next use.

Storage helpers¶

The Storage submodule resolves storage paths and config fields. The main entry points (all take the optional keywords storage_config=, env=ENV, host=gethostname(), and where relevant project_root=; see storage.md for the resolution ladder):

Storage.datasets_dir(; ...) -> String
Storage.datacache_dir(; ...) -> String
Storage.datasets_pools(; ...) -> Vector{String}
Storage.datacache_pools(; ...) -> Vector{String}
Storage.dataset_storage_path(storage_path, key; ...) -> String
Storage.resolve_symbol(name; ...) -> String
Storage.config_layers([storage_config]; project_root="", env=ENV) -> Vector{Dict}
Storage.canonical_write(; ...) -> Bool
Storage.lock_stale_age(; ...) -> Float64
Storage.ConfigSnapshot

datasets_dir: The resolved fetched-datasets folder (default $user_data_dir/datamanifest/shared/datasets). Overridable by DATAMANIFEST_DATASETS_DIR or the datasets_dir config field.
datacache_dir: The resolved produced-cache folder (default $user_cache_dir/datamanifest/projects/$project/cached). Overridable by DATAMANIFEST_DATACACHE_DIR or the datacache_dir config field. See caching.md.
datasets_pools: The resolved fetched-dataset read pools — extra read-only locations probed for an already-present <pool>/<key> before downloading. Set via the datasets_pools config field (a list of path expressions, host-composable via _HOST) or DATAMANIFEST_DATASETS_POOLS (pathsep-separated); when undefined, built-in defaults apply (Storage.POOL_DEFAULTS); an explicit empty list disables them.
datacache_pools: The resolved produced-artifact read pools, probed for an already-produced <pool>/<cachetype>[/<version>]/<hash> before recomputing. Set via datacache_pools or DATAMANIFEST_DATACACHE_POOLS; opt-in — undefined means no pools.
dataset_storage_path(storage_path, key; ...): Resolve a dataset's on-disk path from its storage_path field (empty means $datasets_dir/$key).
resolve_symbol(name; ...): Resolve a single $-symbol. Ladder: DATAMANIFEST_<NAME> environment variable → the configuration layers (per layer: _HOST glob match, then base value) → the predefined defaults (user_data_dir, user_cache_dir, repo, project). A resolved value is itself a path expression and may reference other symbols; an undefined non-predefined symbol is an error.
config_layers(storage_config=Dict(); project_root="", env=ENV): The full configuration chain as a vector of [_STORAGE]-shaped dicts, in precedence order — the checkout config (.datamanifest/config.toml), the manifest's [_STORAGE] (the storage_config argument), and the user-global config (~/.config/datamanifest/config.toml). Pass the result as storage_config to any resolver.
canonical_write: The canonical manifest-write directive (default false) — whether persisted manifests are piped through the Python datamanifest format CLI for byte-identical output. Resolved from DATAMANIFEST_CANONICAL or the canonical config field; truthy strings are "1"/"true"/"yes"/"on" (case-insensitive).
lock_stale_age: The age in seconds (default 30) after which another process's materialization lock is considered stale and may be broken. Resolved from DATAMANIFEST_LOCK_STALE_AGE or the lock_stale_age config field; non-positive or unparsable values fall back to the default.
ConfigSnapshot: A frozen configuration — the file-backed config layers together with the environment and host they are resolved against, captured at Database creation (see freeze_config!). Accepted anywhere storage_config is. A snapshot is authoritative: its captured environment and hostname replace the resolvers' own env=/host= inputs, so every ladder lookup against it — environment rung included — is deterministic for the snapshot's lifetime.

Caching: the `@cached` macro¶

@cached key=(args -> (; …)) [cachetype="…"] [version="…"] [ext="jls"] [basename="data"] [db=DB] function fn(pos…; kw…) … end

Wrap a function with produce-or-load disk caching: the result is computed once and saved under <datacache_dir>/<cachetype>/[<version>/]<hash>/; later calls with the same parameters load the artifact instead of recomputing. Positional and keyword arguments both feed the cache key by parameter name (splatted args... are rejected). The long-form guide is caching.md.

The cache key is the SHA-256 of the canonical JSON (RFC 8785) of the keyword arguments selected by key, computed identically by the Julia and Python tools, so caches are shared across languages. Key-table values may be strings, integers, booleans, finite floats, and arrays/objects of those; NaN, ±Inf, and nulls (nothing/missing) raise an error.

Macro options (all literals except key and db):

key (required): a function receiving the call's keyword arguments as a NamedTuple (every declared keyword except _-prefixed runtime knobs) and returning the key table — the parameters that identify the result. Two calls whose key tables are equal share one artifact.
cachetype (optional): the namespace artifacts are stored under. Defaults to the function's importable name, Module.func, so distinct functions never collide; a function with no stable importable name (defined in a script, the REPL, or via eval) should be given an explicit cachetype.
version (optional): a version string that becomes a path segment and part of the recipe identity (not part of the hash). Bump it to deliberately invalidate old results, e.g. after changing the function's code.
ext (default "jls"): the artifact serialization format. jls (stdlib Serialization) is the dependency-free built-in; register others (nc, jld2, …) with DataManifest.Cache.register_format!(ext, save, load) where save(data, path) writes and load(path) reads.
basename (default "data"): the artifact's file name (<basename>.<ext>).
db (optional): an expression whose value is a Database, evaluated in the caller's scope at call time (so a const LIBDB = Database(...) defined after the @cached function works). The entire cache context then derives from the database's frozen configuration: artifacts land under its datacache_dir (keyed with its $project), locks use its lock_stale_age, and the produced variation registers in its state file. An in-memory database (persist=false) keeps that inventory under the resolved datacache root itself (<datacache_dir>/.datamanifest/state.toml) — the caller's project / cwd never gains a .datamanifest/ from it. See the Database storage_config argument for naming the bundle.

Without db=, the bare form resolves over the default database when a manifest is discoverable (environment override or active project) — which anchors at the same project, so behavior there is unchanged — and falls back to the ambient derivation when none is, so caching keeps working in manifest-less projects.

Per-call behaviour and special keyword arguments. The macro injects a cached::Bool=true keyword: pass cached=false to run the body directly, with no disk reads or writes. If the wrapped function declares keyword arguments with these names, they get extra meaning:

any _-prefixed keyword is a runtime knob: visible in the body, excluded from the hash;
_metadata_extras (NamedTuple/Dict/nothing) is merged into the artifact's metadata.toml sidecar without affecting the hash;
cache_dir overrides the artifact location verbatim (<cache_dir>/<cachetype>/[<version>/]<hash>), bypassing datacache_dir entirely — useful for keeping one experiment's outputs in a folder of its own;
cached_toml overrides the state-file path the produced variation is registered in.

State file¶

The state file (.datamanifest/state.toml, next to the manifest; for an in-memory database, under its storage roots — see Database) records where each fetched dataset and produced artifact landed on this machine. It is maintained automatically by download_dataset and @cached; the following exported helpers read and write it directly (e.g. for custom maintenance tooling).

`CachedIndex`¶

CachedIndex(; recipes=Dict(), datasets=Dict(), path="")

The in-memory view of a state file: recipes (produced artifacts) maps a recipe identity (cachetype, version) to its ref, format, and an instances map hash => artifact directory; datasets (fetched) maps a storage key to its recorded storage_path and sha256; path is the file it was read from / will be written to.

`locate_state`¶

locate_state(base::AbstractString) -> String

The state file to read at base (a directory or file path): the canonical .datamanifest/state.toml when present, else a legacy sibling (.datamanifest-state.toml, cached.toml), else the canonical path (which may not exist yet). Resolution is local to base's project directory — a linked git worktree gets no main-checkout fallback.

`read_index` / `read_index_or_empty`¶

read_index(path::AbstractString) -> CachedIndex
read_index_or_empty(path::AbstractString) -> CachedIndex

Read a state file from path (a file, or a directory holding one under its canonical or legacy name). Older schemas are migrated forward on read. read_index_or_empty returns an empty index bound to the canonical path when no file exists, so the next write_index lands in the right place.

`register!` / `register_dataset!`¶

register!(index::CachedIndex; cachetype, hash, storage_path="", ref="", format="", version="") -> CachedIndex
register_dataset!(index::CachedIndex; key, storage_path="", sha256="") -> CachedIndex

Record a produced variation (register!: a new hash adds an instance under its (cachetype, version) recipe rather than replacing it) or a fetched dataset's resolved location and checksum (register_dataset!: a non-empty argument overwrites the recorded value, an empty one leaves it untouched).

`write_index`¶

write_index(index::CachedIndex, path::AbstractString="") -> String

Write the state file to path (or the index's bound path), canonically ordered, via a temp file and atomic rename. A legacy-named file is migrated to the canonical .datamanifest/state.toml on first write. Returns the path written.

Store maintenance¶

There is no automatic garbage collector: maintenance is always an explicit inspect, filter, act sequence, and only produced (cached) artifacts are eligible for deletion.

`inspect_store`¶

inspect_store(db::Database; cache_root="", cached_toml="") -> Vector{CacheObject}

Enumerate the produced artifacts under the resolved datacache_dir and the datasets present on disk as one list of maintenance objects, resolving referenced: a produced artifact is referenced iff its (cachetype, version, hash) identity is rooted by the project's state file; a present fetched dataset is always referenced (by its manifest entry). cache_root and cached_toml override the resolved cache folder and the state-file path (both default from db).

Each CacheObject has the fields:

kind: "cached" (produced artifact) or "datasets" (fetched dataset);
key: "<cachetype>/<hash>" (produced) or the dataset name (fetched);
hash, cachetype, version: produced-artifact identity;
format, size (bytes), location (absolute path);
created: when the object was produced/fetched — the reliable age signal;
last_access: read from the filesystem at inspect time (the access time, falling back to the modification time when atime is unreadable). Coarse and possibly stale — relatime advances atime at most once a day, and noatime/network/read-only filesystems record nothing — so never use it as the sole basis for deletion; prefer created;
referenced: true/false once resolved (nothing while unknown).

Pass the result through your own filter and act with delete_object / move_object — for example, pruning orphaned artifacts:

db = read_dataset("datamanifest.toml")
for o in inspect_store(db)
    o.kind == "cached" && o.referenced == false && delete_object(o)   # prune orphaned artifacts
end

`delete_object`¶

delete_object(obj::CacheObject) -> Nothing

Delete a produced artifact directory and its sibling completion/lock/tmp markers. Refuses anything that is not kind="cached" — fetched datasets are never removed by the maintenance surface (use delete_dataset for those).

`move_object`¶

move_object(obj::CacheObject, dest_root::AbstractString) -> String

Move a produced artifact to dest_root, preserving its <cachetype>/[<version>/]<hash> key path; returns the new location. Refuses anything that is not kind="cached".

Migration¶

`migrate`¶

migrate(path::String) -> Nothing

Rewrite a v0 manifest at path to schema v1 format, in-place.

Reads the TOML, detects the schema version, and exits early if the file is already v1 (_META.schema ≥ 1).
Moves [_LOADERS] entries that are module:function refs into [_LANG.julia.loaders] (format → ref map).
Moves per-dataset julia= / loader= fields that are refs into [<ds>._LANG.julia] (fetcher / loader).
Inline code that cannot be expressed as a ref is preserved verbatim with a log note — it is the user's responsibility to extract it into a module function.
Sets [_META].schema = 1 in the rewritten file.
Foreign keys and non-Julia _LANG subtrees are never modified.
The call is idempotent: running migrate on an already-v1 file is a no-op.

Returns: Nothing.

API reference¶

Core¶

Database¶

read_dataset¶

add¶

download_dataset¶

download_datasets¶

get_dataset_path¶

load_dataset¶

delete_dataset¶

Searching¶

search_datasets¶

search_dataset¶

Registering and editing entries¶

register_dataset¶

DatasetEntry¶

requires (DatasetEntry field)¶

write¶