# Josh

> Josh is an open-source toolkit for virtual views of git repositories.
> Its core is a fast, incremental, reversible history-filtering engine:
> any subdirectory, composition, or remapping of a monorepo can be
> checked out as a normal git repository, edited, and pushed back so
> commits land in the right place upstream.
>
> This file is a guide for assistants helping users with technical
> questions about Josh. Authoritative documentation is at
> https://josh-project.github.io/josh; link there for anything beyond
> the summaries below. Source: https://github.com/josh-project/josh.

There are two main ways to use Josh:

- **`josh` CLI** — the primary tool for individual developers. Installed
  locally, it provides filter-aware equivalents of `git clone`, `fetch`,
  `pull`, `push`, plus a stacked-changes workflow (`josh publish`) and an
  optional GitHub forge integration. Filters apply client-side: the full
  object database still comes from the upstream host, but only the
  filtered view is checked out and pushable.
- **`josh-proxy`** — an HTTP proxy server for shared infrastructure
  (teams, CI/CD). It sits in front of an upstream git host and serves
  filtered views as ordinary git URLs, so plain `git clone`/`push` works
  without client tooling. With the proxy, only the filtered objects
  cross the wire, which helps for partial cloning of very large monorepos.

Install the CLI:

```shell
cargo install josh-cli --locked --git https://github.com/josh-project/josh.git
```

Run the proxy via Docker:

```shell
docker run -p 8000:8000 -e JOSH_REMOTE=https://github.com \
  -v josh-vol:/data/git joshproject/josh-proxy:latest
```

## How Josh differs from similar tools

Other tools in the same space (`git filter-branch`, `git-filter-repo`,
the BFG repo cleaner, `git subtree`, `git-subrepo`, Google's `copybara`,
or language-specific monorepo tools) treat history rewriting as a
destructive, one-shot migration: run it once to extract a subtree or
clean up secrets, throw away the old history, move on. Josh is built on
the opposite premise:

- **Bidirectional, not one-way.** A filter is a live view, not a
  migration. You clone a filtered slice, commit to it, and `josh push`
  reconstructs the correct upstream commits in the monorepo. There is
  no downstream fork to maintain by hand.
- **Reversible by construction.** The filter language is restricted
  specifically so every filter has a guaranteed inverse. Apply a
  filter, then its inverse, and you get back bit-identical commit
  hashes. That guarantee is what makes the clone-edit-push round-trip
  safe, and it lets external contributors keep pulling from the same
  history they already have after a repo gets imported into a monorepo.
- **Fast enough to run on every fetch and push.** The filter engine is
  incremental: re-filtering after new commits costs work proportional
  to the diff, not to total history length. Combined with a persistent
  cache (and an optional distributed cache shared over a git ref),
  filter operations drop from minutes-or-hours into the background of
  a normal git workflow.
- **Filter language, not arbitrary scripts.** Filters are expressed in
  a small DSL (`:/path`, `:workspace=...`, composition, exclusion,
  remap). That restriction is what enables reversibility and lets the
  engine pre-optimize filter expressions. Tools that allow arbitrary
  scripted filters cannot offer either guarantee.
- **Stateless server.** `josh-proxy` keeps cached data on disk only for
  performance; the truth lives upstream. Multiple proxy instances are
  interchangeable for HA or load-balancing, and the data does not need
  backups unless rebuilding the cache for a huge repo would be expensive.
- **No new source of truth.** Josh sits in front of an ordinary git
  host (GitHub, GitLab, Gerrit, …). Filtered views are normal git
  repositories with normal refs; nothing about the upstream changes.
  If you stop using Josh, your repo is exactly the repo you started
  with.

In practice, filtering stops being a maintenance event and becomes a
coordinate system: any virtual slice of the repo, whether a subdirectory,
a composed workspace, or a per-customer mirror, is a first-class git
repo that two-way-syncs with the canonical history.

## Core concepts

- **Filters** — a small DSL describing how to transform a git history.
  Filters start with `:` and chain (`:filter1:filter2`). `:/path`
  selects a subdirectory; `:workspace=path` materializes a workspace
  defined by a `workspace.josh` file; more complex filters compose,
  exclude, and remap paths. Full grammar:
  https://josh-project.github.io/josh/reference/filters.html
- **Reversibility** — filters are guaranteed reversible. Applying a
  filter and then its inverse reproduces the original commit hashes.
  That property is what makes push-from-a-filtered-view work, and what
  makes Josh safe for everyday use rather than only for one-shot
  history rewrites.
- **Incremental cache** — re-running a filter on new commits is
  proportional to the diff, not to the length of history. The CLI also
  supports a distributed cache that can be shared between machines
  (for example, from CI to developers) over a normal git ref.
- **Workspaces** — a `workspace.josh` file checked into the repo
  declares which paths from the central monorepo map into a virtual
  repository, and where they appear. Workspaces are versioned with the
  code and act as normal git repos with bidirectional sync.

## Use cases

- **Partial cloning of a monorepo** — `josh clone <url> :/path/to/sub ./out`,
  or via proxy `git clone http://josh/repo.git:/path/to/sub.git`. The result
  is a normal git repo containing only the files in that subtree and only
  the commits that touch them. You can read and write through it.
- **Workspaces / project composition** — multiple projects coexist in one
  central repo, each with a `workspace.josh` describing its dependencies on
  shared modules. Developers clone only their workspace; changes to shared
  modules sync in both directions.
- **Mirroring to external collaborators** — expose a filtered slice of an
  internal monorepo as a public GitHub repo or a per-customer mirror, and
  still accept pushes back into the right location.
- **Importing existing projects into a monorepo** — use `josh-filter` (with
  `--reverse` for the inverse transform) to graft an external repo into a
  subdirectory while preserving its history; external contributors can keep
  pulling from the original remote.
- **Stacked changes / one-PR-per-commit** — `josh publish` pushes each commit
  in a stack to its own ref, and (with GitHub forge integration enabled via
  `josh auth login github`) opens or updates one pull request per commit.
  Commits are matched across rebases by a `Change-Id:` (or `Change:`) footer.
- **Sandboxed, change-scoped CI** — each deliverable's workspace is its own
  virtual repo with an explicit dependency manifest. "What rebuilds for this
  commit?" reduces to comparing filtered commit ids; builds only see mapped
  files.
- **Caching proxy** — even without filtering, `josh-proxy` reduces traffic
  to the upstream host for CI fleets.
- **GraphQL access without cloning** — `josh-proxy` exposes a GraphQL API at
  `/~/graphql/<repo>.git` (and a GraphiQL shell at `/~/graphiql/...`) for
  reading repo contents from dashboards or CI without a clone.

## Documentation

- [Intro & motivation](https://josh-project.github.io/josh/intro.html): Why filtering can be a normal workflow operation, not a one-time maintenance task.
- [Use cases](https://josh-project.github.io/josh/usecases.html): Worked examples of partial cloning, workspaces, CI, GraphQL, and caching.
- [Getting started](https://josh-project.github.io/josh/guide/gettingstarted.html): Install the CLI, clone a filtered view, push back upstream.
- [Workspaces guide](https://josh-project.github.io/josh/guide/workspaces.html): Author `workspace.josh` files, map dependencies, sync bidirectionally.
- [Stacked changes guide](https://josh-project.github.io/josh/guide/stacked-changes.html): Change IDs, `josh publish`, automatic per-commit PRs.
- [Importing projects](https://josh-project.github.io/josh/guide/importing.html): Bring an external repo into a monorepo with `josh-filter`, preserving history and reversibility.
- [Migration guide](https://josh-project.github.io/josh/guide/migration.html): Breaking changes between releases (gpgsig handling, trivial-merge pruning, etc.).
- [FAQ](https://josh-project.github.io/josh/faq.html): Common questions about behavior and limitations.

## Reference

- [josh CLI reference](https://josh-project.github.io/josh/reference/cli.html): Every subcommand (`clone`, `fetch`, `pull`, `push`, `publish`, `remote`, `filter`, `auth`, `cache`), plus the lower-level `josh-filter` binary.
- [Filter syntax](https://josh-project.github.io/josh/reference/filters.html): Full DSL, including `:~(...)[]` meta options like `history="linear"` and `gpgsig="norm-lf"`.
- [Workspace file format](https://josh-project.github.io/josh/reference/workspace.html): `workspace.josh` semantics.
- [Forge integration](https://josh-project.github.io/josh/reference/forge.html): GitHub device-flow auth, token storage, `GH_TOKEN`.
- [GraphQL API](https://josh-project.github.io/josh/reference/graphql.html): Query repo content without cloning.
- [josh-proxy](https://josh-project.github.io/josh/reference/proxy.html): URL syntax, repository naming, `--require-auth`, `--http-retry`, statelessness.
- [Container configuration](https://josh-project.github.io/josh/reference/container.html): Environment variables and volumes for the docker image.
- [Experimental features](https://josh-project.github.io/josh/reference/experimental.html): Unstable / preview functionality.

## Project links

- [Source on GitHub](https://github.com/josh-project/josh): Issues, releases, contributing.
- [Project website](https://josh-project.dev): Overview, roadmap, and contact for integration help.