Getting started

Getting started#

CADET-RDM manages computational research projects by separating project code and generated results into two coupled Git repositories:

  • the project repository, which contains source code, configuration files, documentation, and metadata

  • the output repository, which contains all results generated by executing the project code

Both repositories are created and managed automatically. They are independent Git repositories with separate histories and remotes, but CADET-RDM provides workflows that operate on both to ensure reproducibility and traceability of results.

CADET-RDM can be used through two interfaces:

  • a command line interface (CLI), e.g. for scripted or automated bash workflows

  • a Python interface, e.g. for direct context tracking of code within existing Python workflows

Additionally, CADET-RDM can be used within Jupyter Lab with some limitations.

Detailed descriptions of commands and APIs are provided in the dedicated interface documentation.

Initializing a project repository#

Create a new project repository or convert an existing directory into a CADET-RDM repository.

CLI:

rdm init <path_to_repo> [output_directory_name]

Python:

from cadetrdm import initialize_repo
initialize_repo(path_to_repo, [output_directory_name])

If output_directory_name is not provided, it defaults to output.

During initialization, the project repository is created or updated and an output repository is created inside the project repository. The two repositories are independent Git repositories.

Cookiecutter support#

CADET-RDM supports initializing repositories from Cookiecutter templates.

CLI:

rdm init <path_to_repo> --cookiecutter <template_url>

Python:

from cadetrdm import initialize_repo
initialize_repo(path_to_repo, cookiecutter_template="template_url")

Options:

  • If <path_to_repo> is provided as an absolute or relative path, it overrides any directory name specified by the Cookiecutter template.

  • If <path_to_repo> is omitted, the repository is initialized in the current working directory. No additional directory is created, even if the Cookiecutter template would normally create one.

For state-of-the-art Python package development, we recommend using the CADET Cookiecutter Template.

This template provides a standardized starting point for development projects, including project metadata, development tooling, and continuous integration configuration.

This Cookiecutter template automatically creates a repository with the following content:

README.md
LICENCE.md
AUTHORS.md
SECURITY.md
CODE_OF_CONDUCT.md
CONTRIBUTING.md
CITATION.bib
.zenodo.json
pyproject.toml
environment.yml
.pre-commit-config.yml
.github/dependabot.yml
.github/workflows/ruff.yml
.gitignore

Handling results with CADET-RDM#

Running code and tracking results#

Each execution of project code creates a new output branch that contains the generated results and associated metadata.

CLI (run a Python script):

cd <path_to_repo>
rdm run python <path/to/script.py> "commit message for the results"

CLI (run an arbitrary command):

rdm run command "bash run_simulation.sh" "commit message for the results"

Python (track results in code):

from cadetrdm import ProjectRepo

repo = ProjectRepo()
repo.commit("Commit code changes")

with repo.track_results(results_commit_message="Generate results"):
    data = generate_data()
    write_data_to_file(data, output_directory=repo.output_directory)

Pushing changes#

CLI (check consistency and stage changes):

rdm check

CLI (commit staged changes):

rdm commit -m <message>

CLI (push project and output repositories):

rdm push

Python (push both repositories):

repo.push()

Reusing results from earlier runs#

Results are referenced by a unique output branch name:

<timestamp>_<active_project_branch>_<project_repo_hash[:7]>

CLI:

rdm data cache <branch_name>

Python:

cached_folder_path = repo.input_data(branch_name="<branch_name>")

Using results from another repository#

Results from other CADET-RDM projects can be reused via the local cache mechanism.

CLI (fetch remote repositories defined in the cache file):

rdm data fetch

Python (import a remote repository and register it in the cache):

repo.import_remote_repo(
    source_repo_location="<URL>",
    source_repo_branch="<branch_name>"
)

Optionally specify a destination directory:

repo.import_remote_repo(
    source_repo_location="<URL>",
    source_repo_branch="<branch_name>",
    target_repo_location="<path/to/destination/repository>"
)

Python (load repositories listed in .cadet-rdm-cache.json):

repo.fill_data_from_cadet_rdm_json()

How result metadata is cached#

CADET-RDM uses the .cadet-rdm-cache.json file to keep track of external or previously generated results that can be reused as input data.

The file is located in the project repository root and is managed automatically by CADET-RDM. It should not be edited manually.

Conceptually, the cache file stores references to results, not the results themselves. It records:

  • the location of an output repository

  • the output branch containing the results

  • the exact commit hash used for reproducibility

A typical entry looks like this:

{
  "__example/path/to/repo__": {
    "source_repo_location": "git@github.com:cadet/example_output.git",
    "branch_name": "2024-10-25_main_3910c84",
    "commit_hash": "6e3c26527999036e9490d2d86251258fe81d46dc"
  }
}

Remote repositories#

To share both project code and results, remotes must be configured for both repositories.

Cloning an existing CADET-RDM project#

Use CADET-RDM cloning rather than git clone.

CLI:

rdm clone <project_url> <destination_path>

Python:

from cadetrdm import ProjectRepo
ProjectRepo.clone("<project_url>", "<destination_path>")

The destination directory must be empty.

Adding existing remotes manually#

CLI:

rdm remote add git@<my_git_server.foo>:<project>.git
cd output
rdm remote add git@<my_git_server.foo>:<project>_output.git

Python:

from cadetrdm import ProjectRepo

repo = ProjectRepo()
repo.add_remote("git@<my_git_server.foo>:<project>.git")
repo.output_repo.add_remote("git@<my_git_server.foo>:<project>_output.git")

Creating remotes automatically via GitHub or GitLab APIs#

CADET-RDM can create both remotes automatically if a Personal Access Token is available in the Python keyring.

The URL must match the GitHub or GitLab instance used for remote creation, for example:

Store a token (Python):

import keyring
keyring.set_password("https://jugit.fz-juelich.de/", "username", "token")

Store a token (CLI):

keyring set "https://jugit.fz-juelich.de/" <username>

Create remotes (Python):

from cadetrdm import ProjectRepo

repo = ProjectRepo()
repo.create_remotes(
    name="Workproject",
    namespace="githubusers_workproject",
    url="https://github.com/",
    username="githubuser"
)

Create remotes (CLI):

rdm remote create <url> <namespace> <name> <username>

Example:

rdm remote create https://github.com/ githubusers_workproject Workproject githubuser

The output repository name is derived automatically by appending _output to the project repository name.

Migrating a repository to another remote#

To migrate to a different remote, update the origin URL for both repositories and push.

CLI:

rdm remote set-url origin git@<my_git_server.foo>:<project>.git
cd output
rdm remote set-url origin git@<my_git_server.foo>:<project>_output.git
cd ..
rdm push
Contents