---
title: "Get started with tarpolyglot"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Get started with tarpolyglot}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = FALSE)
```

tarpolyglot lets you run **Python**, **Julia**, and **Rust** code as ordinary steps of a [`targets`](https://docs.ropensci.org/targets/) pipeline. It adds three pairs of target constructors that mirror `targets::tar_target()` / `targets::tar_target_raw()`:

| Language | Non-standard eval (in `_targets.R`) | Raw (inside factories) | Bridge |
|------------------|------------------|------------------|------------------|
| Python | `tar_target_py()` | `tar_target_py_raw()` | reticulate |
| Julia | `tar_target_jl()` | `tar_target_jl_raw()` | JuliaCall |
| Rust | `tar_target_rs()` | `tar_target_rs_raw()` | rextendr / extendr |

Every code block below is shown for illustration and **is not executed** when the vignette is built (running it would need a live Python/Julia/Rust toolchain and a `_targets.R` project). Copy the pieces into your own pipeline.

## Motivation

If you already work in `targets`, you usually want to keep *all* of its machinery (dynamic branching, cues and invalidation, storage formats, `crew`-based parallelism, `tarchetypes`) while occasionally dropping into Python or Julia for a step or two, or writing a hot function in Rust. [rixpress](https://github.com/ropensci/rixpress) is a really good take on polyglot, reproducible pipelines, but it builds those pipelines on **Nix**-managed environments which could be difficult to implement in some case (e.g. a public cluster on which you can't install nix, a windows computer on which you don't wish to install wsl).

Doing that by hand means, in every single step, repeating the same plumbing: initialise reticulate against the right environment, push your R objects into Python, run the script, convert the results back, and the equivalent dance for JuliaCall, or the `rextendr::rust_source()` setup for Rust. tarpolyglot moves that boilerplate into the constructor. You point at a script, say which upstream targets feed it and what to read back, and you get a normal `targets` target:

``` r
library(targets)
library(tarpolyglot)

list(
  tar_target(values, c(2, 4, 6, 8)),
  tar_target_py(
    name = py_mean,
    script = "py/mean.py",           # your Python, unchanged
    inputs = c(x = "values"),        # wire in an upstream target
    pre_script = "R/push.R",         # hand `x` to Python
    retrieve = "result"              # read the Python `result` back into R
  )
)
```

No `reticulate::use_*()`, no manual conversion, no bespoke wrapper per step, and the environment/version selection (virtualenv, venv, uv, poetry, conda for Python; juliaup and Julia projects for Julia; rustup toolchains for Rust) is just another argument. You keep every `targets` feature and pick your own environment manager, without adopting a separate build tool such as Nix.

## Installation

``` r
# install.packages("remotes")
remotes::install_local("path/to/tarpolyglot", build_vignettes = TRUE)
```

You then need a working toolchain only for the language(s) you actually use (each is independent):

- **Python**: any recent CPython, plus `install.packages("reticulate")`. reticulate can also provision an ephemeral Python for you via [uv](https://docs.astral.sh/uv/).
- **Julia**: install via [juliaup](https://github.com/JuliaLang/juliaup), plus `install.packages("JuliaCall")`.
- **Rust**: install the toolchain with [rustup](https://rust-lang.org/tools/install/) (on **Windows use the GNU toolchain**), plus `install.packages("rextendr")`.

See the per-language vignettes for the details: `vignette("python")`, `vignette("julia")`, `vignette("rust")`.

## Limitations to know first

tarpolyglot is a thin bridge, so it inherits the constraints of the runtimes it wraps. The important ones:

1.  **One interpreter per R session.** reticulate binds a single Python per R process, and JuliaCall a single Julia, fixed on the first call. A second step in the *same* session that asks for a different environment silently runs in the first one. To use different environments in one pipeline, put those steps on **separate `crew` workers** (see below). *(Rust is exempt: each Rust step compiles its own library, so different steps can use different crates freely.)*

2.  **Foreign global state is shared within a session.** Your Python script runs in Python's top-level module, `__main__`, and your Julia script in Julia's, `Main`. Those namespaces belong to the *interpreter*, not to the step, so steps sharing one R session share them too: a later step does not start from a clean slate, it reuses the interpreter an earlier step already bound, variables and all. Beware of three similar-looking names here: Python's `__main__` and Julia's `Main` are foreign-language modules, while `deployment = "main"` refers to the main **R** process, and that is where this bites most often, because `targets` runs every `main`-deployed step in that one session. Giving each step its own `crew` worker avoids it (point 3). Either way, never rely on those leftovers: pass data in explicitly (`inputs` + `to_py`/`to_jl`) and read results back with `retrieve` or a post-script.

3.  **Start-up cost vs isolation.** A fresh process re-initialises the interpreter (and, on the default ephemeral env, may fetch packages). The recommended `polyglot_controller()` gives each target a fresh interpreter (`tasks_max = 1`) for maximum isolation; raise `tasks_max` to reuse interpreters for throughput, at the cost of the shared-session caveats in points 1–2.

4.  **Only serialisable R values cross target boundaries.** A step can return an R object produced by conversion, or **file paths** (`output = "file"`). A live Python/Julia object (an open handle, an in-memory model) cannot be a target value. Or you can instead write it to disk in one step and load it in the next.

5.  **Reproducibility is on you.** The default ephemeral Python drifts over time; pin a real environment (`env` + `env_manager`, or `python`) and, for Julia, a `julia_project` with a committed `Manifest.toml`. You should also make sure to use some targets steps to keep track of your script files.

6.  **A `crew` worker is a full process, not a slice of one.** It is not restricted to a single CPU core the way a lightweight thread might be, it sees every core the machine has, exactly like the main session. This matters when a step itself does internal multi-core work (RcppParallel, Python's `multiprocessing`, Julia's `Distributed`): running several such steps at once on separate workers means each one may try to claim every core, so `N` concurrent workers each spawning a full-width thread or process pool can genuinely oversubscribe the machine, degrading performance or, on a machine with few cores, exhausting it. `deployment = "main"` on the specific step that does the internal parallel work is the fix: `targets` only ever builds one `main`-deployed target at a time, so those steps never compound with each other, while ordinary worker-deployed steps keep running alongside them. See `vignette("cpp")`, `vignette("python")`, and `vignette("julia")` for the working pattern in each language.

## A worked pipeline (Python + iris)

This section builds one small pipeline that exercises the features you will use most: passing data (including a reproducible **seed**) into Python, reading variables back in a post-script, tracking the scripts as dependencies, and returning results either as an in-pipeline object or as files on disk.

### The three-script model

A Python (or Julia) step is up to three scripts:

```         
 upstream targets ─► pre_script (R) ─► script (.py) ─► post_script (R) ─► target value
                     builds `to_py`     computes         reads `py$name` /
                                        `result`         `py_get("name")`
```

- **`script`** (required): your Python/Julia file.
- **`pre_script`** (optional R): runs first; the `inputs` are already bound by name. Assign a named list `to_py` (Python) / `to_jl` (Julia) to push objects into the foreign session.
- **`post_script`** (optional R): runs last; reads results back. Its last expression becomes the target value (object mode), or it returns file paths (file mode).

### Object output: return a value into the pipeline

`py/iris_kmeans.py` (a small, seeded computation on iris):

``` python
import numpy as np

# `df` (a pandas DataFrame) and `seed` (an int) were pushed from R.
rng = np.random.default_rng(seed)
X = df[["Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"]].to_numpy()

# pick 3 random initial centres, reproducibly (same seed -> same pick)
idx = rng.choice(X.shape[0], size=3, replace=False)
result = {"n": int(X.shape[0]), "centers": X[idx].tolist()}
```

`R/iris_pre.R`: the pre-script builds `to_py`, and gets the target's **reproducible seed** straight from `targets`:

``` r
# `df` is bound from inputs = c(df = "iris_data").
# tar_seed_get() returns THIS target's deterministic seed (derived from the
# pipeline seed and the target name), so the Python RNG is reproducible.
to_py <- list(
  df   = df,
  seed = targets::tar_seed_get()
)
```

`R/iris_post.R`: the post-script reads variables back. tarpolyglot binds `py` (the reticulate `__main__` proxy) and `py_get(name)` (an explicit `reticulate::py_to_r()` shortcut). The last expression is the target's value:

``` r
res <- py_get("result")               # or: res <- py$result
data.frame(n = res$n, k = length(res$centers))
```

`_targets.R`:

``` r
library(targets)
library(tarpolyglot)

list(
  tar_target(iris_data, iris),

  tar_target_py(
    name = km,
    script = "py/iris_kmeans.py",
    inputs = c(df = "iris_data"),
    pre_script = "R/iris_pre.R",
    post_script = "R/iris_post.R"       # returns a data.frame into the pipeline
  )
)
```

If all you need is to read one or more variables verbatim (no reshaping), drop the post-script and use `retrieve` instead: it does the `py_get()` lookup for you:

``` r
tar_target_py(
  name = km,
  script = "py/iris_kmeans.py",
  inputs = c(df = "iris_data"),
  pre_script = "R/iris_pre.R",
  retrieve = "result"                   # returns the Python `result` as an R list
)
```

### File output: return paths to files on disk

When the script *writes* its output (a CSV, a model, a figure), set `output = "file"`. The target `format` becomes `"file"` automatically, and `targets` tracks the returned paths by hash.

`py/iris_write.py`:

``` python
import pandas as pd
out_path = "out/iris_summary.csv"
df.groupby("Species").mean().to_csv(out_path)   # `df` pushed from R
```

`R/iris_files_post.R` (returns the path(s) the step produced):

``` r
py_get("out_path")                      # a character vector of file paths
```

``` r
tar_target_py(
  name = iris_csv,
  script = "py/iris_write.py",
  inputs = c(df = "iris_data"),
  pre_script = "R/iris_pre.R",
  post_script = "R/iris_files_post.R",
  output = "file"                        # target tracks the CSV on disk
)
```

Use file mode for anything that does not round-trip cleanly through conversion (large arrays, models, non-convertible objects): write it in one step, load it in the next step's `pre_script`.

### Tracking the scripts as dependencies

By default `script` / `pre_script` / `post_script` are plain literal paths: `targets` embeds them as constants, so **editing the file does not invalidate the target**. To have the step re-run when a script changes, track that script with its own `format = "file"` target and reference it with `tar_target_path()`:

``` r
list(
  tar_target(iris_data, iris),

  # each script tracked as a file target
  tar_target(kmeans_py,  "py/iris_kmeans.py", format = "file"),
  tar_target(pre_R,      "R/iris_pre.R",      format = "file"),
  tar_target(post_R,     "R/iris_post.R",     format = "file"),

  tar_target_py(
    name = km,
    script      = tar_target_path("kmeans_py"),   # re-runs when the .py changes
    pre_script  = tar_target_path("pre_R"),
    post_script = tar_target_path("post_R"),
    inputs = c(df = "iris_data")
  )
)
```

`tar_target_path("name")` takes the *name* of the upstream file target (exactly like `inputs = c(x = "some_target")`), so `targets`' normal dependency detection picks it up, so there is no need to also list it in `deps`.

### Parallelism and isolation with crew

Because of limitation 1 (one interpreter per session), the clean way to give each step a fresh interpreter, and to let different steps use different environments, is to run them in separate **worker processes** with [crew](https://wlandau.github.io/crew/). tarpolyglot ships a controller preconfigured for this:

``` r
tar_option_set(controller = polyglot_controller(workers = 2L))
```

`polyglot_controller()` wraps `crew::crew_controller_local()` with **`tasks_max = 1L`**: each worker runs one task then retires, so every foreign step starts from a brand-new interpreter. With a controller set, tarpolyglot targets (whose `deployment` defaults to `"worker"`) run on workers automatically.

**Foreign steps on workers, ordinary R steps on the main process.** This is the common setup: the heavy R work stays in-process, and only the Python/Julia/Rust steps go to the controller. Flip the global default to `deployment = "main"` and override just the foreign steps back to `"worker"`:

``` r
tar_option_set(
  controller = polyglot_controller(workers = 1L),
  deployment = "main"                    # every ordinary tar_target() stays on main
)

list(
  tar_target(iris_data, iris),           # main process
  tar_target_py(
    name = km,
    script = "py/iris_kmeans.py",
    inputs = c(df = "iris_data"),
    pre_script = "R/iris_pre.R",
    retrieve = "result",
    deployment = "worker"                # THIS step runs on the polyglot worker
  )
)
```

The per-step `deployment = "worker"` is required here: without it the Python step would also inherit `"main"` and run in-process, with no interpreter isolation.

**Foreign and R steps on different workers.** Use a named `crew::crew_controller_group()` and route each target by controller name; set the R controller as the default via `resources` so ordinary targets need no extra argument:

``` r
tar_option_set(
  controller = crew::crew_controller_group(
    polyglot_controller(name = "poly", workers = 2L),          # Python/Julia/Rust
    crew::crew_controller_local(name = "rmain", workers = 4L)  # ordinary R targets
  ),
  resources = tar_resources(crew = tar_resources_crew(controller = "rmain"))
)

list(
  tar_target(iris_data, iris),           # -> "rmain" (the default)
  tar_target_py(
    name = km,
    script = "py/iris_kmeans.py",
    inputs = c(df = "iris_data"),
    pre_script = "R/iris_pre.R",
    retrieve = "result",
    resources = tar_resources(crew = tar_resources_crew(controller = "poly"))
  )
)
```

**Dynamic branching** works unchanged: `inputs` become real dependencies, so `pattern` branches them, and crew spreads the branches across workers (each a fresh interpreter under `tasks_max = 1`):

``` r
list(
  tar_target(iris_groups, split(iris, iris$Species), iteration = "list"),  # 3 groups
  tar_target_py(
    name = per_species,
    script = "py/iris_kmeans.py",
    inputs = c(df = "iris_groups"),
    pre_script = "R/iris_pre.R",
    retrieve = "result",
    pattern = map(iris_groups),          # one branch per species, in parallel
    iteration = "list"
  )
)
```

**Per-step log files.** `polyglot_controller()` accepts a `log` argument (built with `tar_polyglot_log()`) that writes one stdout/stderr file per Python or Julia step, named after the target:

``` r
tar_option_set(
  controller = polyglot_controller(
    workers = 2L,
    log = tar_polyglot_log(stdout = "logs/out", stderr = "logs/err")
  )
)
```

The `km` step above would then get `logs/out/km.out` / `logs/err/km.err`, starting with a header (the step name, `date()`, the resolved interpreter's version and path, and whether an explicit environment was used) followed by whatever the script itself printed. By default (`append = FALSE`) a step's log is truncated at the start of its run, so it only ever holds the latest attempt; set `append = TRUE` to accumulate history instead, with new content separated from old by two blank lines -- this also means that if `crew` silently retries a step after a worker crash, both attempts show up in the same file, which can be useful for debugging.

`tar_polyglot_log()` covers Python and Julia only, not Rust: `rextendr`-compiled code writes straight to the OS file descriptor, bypassing the redirection reticulate/JuliaCall provide for their embedded interpreters -- and Rust doesn't pay a per-branch interpreter start-up cost the way Python/Julia do to begin with, so there is less need for a per-step Rust log (see `tarpolyglot_map()` in `vignette("rust")` for how a Rust step's compiled library is instead built once and reused across branches). Use `crew`'s own `options_local(log_directory = ...)` for Rust step output; it also works for Python/Julia when `log` is left unset, since tarpolyglot never spawns a subprocess for them -- reticulate/JuliaCall's output shares the worker process's own stdout/stderr, so `crew`'s process-level capture sees it too. Setting *both* is fine, and the two are complementary rather than redundant: `options_local(log_directory = ...)` also captures ordinary R-level output (pre/post-script messages, warnings) that `tar_polyglot_log()` does not touch, while `tar_polyglot_log()`'s file is named after the step rather than a `crew`-generated worker name. They do **split** the foreign output between the two files rather than duplicate it, though: while a step's script body is running, `tar_polyglot_log()` diverts the foreign print output specifically into its own file, so for that window it will not also appear in `crew`'s log.

One caveat under **dynamic branching**: every branch of a pattern target shares the *same* target name, so with `append = FALSE` (the default) each branch's log write truncates whatever the previous branch wrote -- only the last-finishing branch's log survives. Set `append = TRUE` if you need every branch's output; they land in one file, interleaved by finish order, not split into one file per branch.

See `vignette("python")`, `vignette("julia")`, and `vignette("rust")` for the full per-language reference, and `?polyglot_controller` / `?tar_polyglot_log` for the controller and logging options.

## Pre- and post-scripts: moving variables in and out

The pre- and post-scripts are the two hooks where you move data between R and the foreign session. They are ordinary R scripts, evaluated in the step's environment.

### Sending variables in (the pre-script)

The pre-script runs **before** the foreign script, with the `inputs` already bound by name. If it assigns a named list called `to_py` (Python) or `to_jl` (Julia), tarpolyglot pushes each element into the foreign session:

- **Python**: each element becomes a top-level variable in the `__main__` module (via reticulate's conversion rules).
- **Julia**: each element is `julia_assign()`ed as a variable in `Main`.

``` r
# R/iris_pre.R  (inputs = c(df = "iris_data"))
to_py <- list(
  df   = df,                       # R data.frame -> pandas DataFrame
  seed = targets::tar_seed_get(),  # reproducible integer for the Python RNG
  k    = 3L                        # note the L: Python wants an int, not a double
)
```

Check the reticulate / JuliaCall conversion rules for which R types round-trip; for anything that does not, use file mode.

### Reading variables out (the post-script)

The post-script runs **after** the foreign script. tarpolyglot binds a few helpers into its environment:

- **Python steps**: `py`, the reticulate `__main__` proxy (`py$result`), and `py_get("result")`, a shortcut for `reticulate::py_to_r(py[["result"]])` that forces an explicit conversion.
- **Julia steps**: `jl_get("name")`, a shortcut for `JuliaCall::julia_eval("name")` (JuliaCall has no `py`-style proxy), and `jl_call`, an alias of `JuliaCall::julia_call()`.

``` r
# R/iris_post.R  (Python)
res <- py_get("result")
data.frame(n = res$n, k = length(res$centers))   # last expression = target value
```

Rust is different: it has **no pre-script**. The `#[extendr]` functions are compiled and then called directly from the post-script, with the `inputs` in scope. See `vignette("rust")`.

## Where to next

- `vignette("python")`: every `tar_target_py()` argument, choosing an environment (system / virtualenv / venv / uv / poetry / conda) or a `python_version`, with use-case examples.
- `vignette("julia")`: the same for Julia (`julia_version`, `julia_project`, `julia_packages`).
- `vignette("rust")`: Rust steps via rextendr/extendr.
- `?tar_target_py`, `?tar_target_jl`, `?tar_target_rs`, `?tar_target_path`, `?polyglot_controller`: the function reference.