Get started with tarpolyglot

The three-script model

A Python (or Julia) step is up to three scripts:

 upstream targets ─► pre_script (R) ─► script (.py) ─► post_script (R) ─► target value
                     builds `to_py`     computes         reads `py$name` /
                                        `result`         `py_get("name")`

• script (required): your Python/Julia file.
• pre_script (optional R): runs first; the inputs are already bound by name. Assign a named list to_py (Python) / to_jl (Julia) to push objects into the foreign session.
• post_script (optional R): runs last; reads results back. Its last expression becomes the target value (object mode), or it returns file paths (file mode).

Object output: return a value into the pipeline

py/iris_kmeans.py (a small, seeded computation on iris):

import numpy as np

# `df` (a pandas DataFrame) and `seed` (an int) were pushed from R.
rng = np.random.default_rng(seed)
X = df[["Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"]].to_numpy()

# pick 3 random initial centres, reproducibly (same seed -> same pick)
idx = rng.choice(X.shape[0], size=3, replace=False)
result = {"n": int(X.shape[0]), "centers": X[idx].tolist()}

R/iris_pre.R: the pre-script builds to_py, and gets the target’s reproducible seed straight from targets:

# `df` is bound from inputs = c(df = "iris_data").
# tar_seed_get() returns THIS target's deterministic seed (derived from the
# pipeline seed and the target name), so the Python RNG is reproducible.
to_py <- list(
  df   = df,
  seed = targets::tar_seed_get()
)

R/iris_post.R: the post-script reads variables back. tarpolyglot binds py (the reticulate __main__ proxy) and py_get(name) (an explicit reticulate::py_to_r() shortcut). The last expression is the target’s value:

res <- py_get("result")               # or: res <- py$result
data.frame(n = res$n, k = length(res$centers))

_targets.R:

library(targets)
library(tarpolyglot)

list(
  tar_target(iris_data, iris),

  tar_target_py(
    name = km,
    script = "py/iris_kmeans.py",
    inputs = c(df = "iris_data"),
    pre_script = "R/iris_pre.R",
    post_script = "R/iris_post.R"       # returns a data.frame into the pipeline
  )
)

If all you need is to read one or more variables verbatim (no reshaping), drop the post-script and use retrieve instead: it does the py_get() lookup for you:

tar_target_py(
  name = km,
  script = "py/iris_kmeans.py",
  inputs = c(df = "iris_data"),
  pre_script = "R/iris_pre.R",
  retrieve = "result"                   # returns the Python `result` as an R list
)

File output: return paths to files on disk

When the script writes its output (a CSV, a model, a figure), set output = "file". The target format becomes "file" automatically, and targets tracks the returned paths by hash.

py/iris_write.py:

import pandas as pd
out_path = "out/iris_summary.csv"
df.groupby("Species").mean().to_csv(out_path)   # `df` pushed from R

R/iris_files_post.R (returns the path(s) the step produced):

py_get("out_path")                      # a character vector of file paths

tar_target_py(
  name = iris_csv,
  script = "py/iris_write.py",
  inputs = c(df = "iris_data"),
  pre_script = "R/iris_pre.R",
  post_script = "R/iris_files_post.R",
  output = "file"                        # target tracks the CSV on disk
)

Use file mode for anything that does not round-trip cleanly through conversion (large arrays, models, non-convertible objects): write it in one step, load it in the next step’s pre_script.

Tracking the scripts as dependencies

By default script / pre_script / post_script are plain literal paths: targets embeds them as constants, so editing the file does not invalidate the target. To have the step re-run when a script changes, track that script with its own format = "file" target and reference it with tar_target_path():

list(
  tar_target(iris_data, iris),

  # each script tracked as a file target
  tar_target(kmeans_py,  "py/iris_kmeans.py", format = "file"),
  tar_target(pre_R,      "R/iris_pre.R",      format = "file"),
  tar_target(post_R,     "R/iris_post.R",     format = "file"),

  tar_target_py(
    name = km,
    script      = tar_target_path("kmeans_py"),   # re-runs when the .py changes
    pre_script  = tar_target_path("pre_R"),
    post_script = tar_target_path("post_R"),
    inputs = c(df = "iris_data")
  )
)

tar_target_path("name") takes the name of the upstream file target (exactly like inputs = c(x = "some_target")), so targets’ normal dependency detection picks it up, so there is no need to also list it in deps.

Parallelism and isolation with crew

Because of limitation 1 (one interpreter per session), the clean way to give each step a fresh interpreter, and to let different steps use different environments, is to run them in separate worker processes with crew. tarpolyglot ships a controller preconfigured for this:

tar_option_set(controller = polyglot_controller(workers = 2L))

polyglot_controller() wraps crew::crew_controller_local() with tasks_max = 1L: each worker runs one task then retires, so every foreign step starts from a brand-new interpreter. With a controller set, tarpolyglot targets (whose deployment defaults to "worker") run on workers automatically.

Foreign steps on workers, ordinary R steps on the main process. This is the common setup: the heavy R work stays in-process, and only the Python/Julia/Rust steps go to the controller. Flip the global default to deployment = "main" and override just the foreign steps back to "worker":

tar_option_set(
  controller = polyglot_controller(workers = 1L),
  deployment = "main"                    # every ordinary tar_target() stays on main
)

list(
  tar_target(iris_data, iris),           # main process
  tar_target_py(
    name = km,
    script = "py/iris_kmeans.py",
    inputs = c(df = "iris_data"),
    pre_script = "R/iris_pre.R",
    retrieve = "result",
    deployment = "worker"                # THIS step runs on the polyglot worker
  )
)

The per-step deployment = "worker" is required here: without it the Python step would also inherit "main" and run in-process, with no interpreter isolation.

Foreign and R steps on different workers. Use a named crew::crew_controller_group() and route each target by controller name; set the R controller as the default via resources so ordinary targets need no extra argument:

tar_option_set(
  controller = crew::crew_controller_group(
    polyglot_controller(name = "poly", workers = 2L),          # Python/Julia/Rust
    crew::crew_controller_local(name = "rmain", workers = 4L)  # ordinary R targets
  ),
  resources = tar_resources(crew = tar_resources_crew(controller = "rmain"))
)

list(
  tar_target(iris_data, iris),           # -> "rmain" (the default)
  tar_target_py(
    name = km,
    script = "py/iris_kmeans.py",
    inputs = c(df = "iris_data"),
    pre_script = "R/iris_pre.R",
    retrieve = "result",
    resources = tar_resources(crew = tar_resources_crew(controller = "poly"))
  )
)

Dynamic branching works unchanged: inputs become real dependencies, so pattern branches them, and crew spreads the branches across workers (each a fresh interpreter under tasks_max = 1):

list(
  tar_target(iris_groups, split(iris, iris$Species), iteration = "list"),  # 3 groups
  tar_target_py(
    name = per_species,
    script = "py/iris_kmeans.py",
    inputs = c(df = "iris_groups"),
    pre_script = "R/iris_pre.R",
    retrieve = "result",
    pattern = map(iris_groups),          # one branch per species, in parallel
    iteration = "list"
  )
)

Per-step log files. polyglot_controller() accepts a log argument (built with tar_polyglot_log()) that writes one stdout/stderr file per Python or Julia step, named after the target:

tar_option_set(
  controller = polyglot_controller(
    workers = 2L,
    log = tar_polyglot_log(stdout = "logs/out", stderr = "logs/err")
  )
)

The km step above would then get logs/out/km.out / logs/err/km.err, starting with a header (the step name, date(), the resolved interpreter’s version and path, and whether an explicit environment was used) followed by whatever the script itself printed. By default (append = FALSE) a step’s log is truncated at the start of its run, so it only ever holds the latest attempt; set append = TRUE to accumulate history instead, with new content separated from old by two blank lines – this also means that if crew silently retries a step after a worker crash, both attempts show up in the same file, which can be useful for debugging.

tar_polyglot_log() covers Python and Julia only, not Rust: rextendr-compiled code writes straight to the OS file descriptor, bypassing the redirection reticulate/JuliaCall provide for their embedded interpreters – and Rust doesn’t pay a per-branch interpreter start-up cost the way Python/Julia do to begin with, so there is less need for a per-step Rust log (see tarpolyglot_map() in vignette("rust") for how a Rust step’s compiled library is instead built once and reused across branches). Use crew’s own options_local(log_directory = ...) for Rust step output; it also works for Python/Julia when log is left unset, since tarpolyglot never spawns a subprocess for them – reticulate/JuliaCall’s output shares the worker process’s own stdout/stderr, so crew’s process-level capture sees it too. Setting both is fine, and the two are complementary rather than redundant: options_local(log_directory = ...) also captures ordinary R-level output (pre/post-script messages, warnings) that tar_polyglot_log() does not touch, while tar_polyglot_log()’s file is named after the step rather than a crew-generated worker name. They do split the foreign output between the two files rather than duplicate it, though: while a step’s script body is running, tar_polyglot_log() diverts the foreign print output specifically into its own file, so for that window it will not also appear in crew’s log.

One caveat under dynamic branching: every branch of a pattern target shares the same target name, so with append = FALSE (the default) each branch’s log write truncates whatever the previous branch wrote – only the last-finishing branch’s log survives. Set append = TRUE if you need every branch’s output; they land in one file, interleaved by finish order, not split into one file per branch.

See vignette("python"), vignette("julia"), and vignette("rust") for the full per-language reference, and ?polyglot_controller / ?tar_polyglot_log for the controller and logging options.

Sending variables in (the pre-script)

The pre-script runs before the foreign script, with the inputs already bound by name. If it assigns a named list called to_py (Python) or to_jl (Julia), tarpolyglot pushes each element into the foreign session:

• Python: each element becomes a top-level variable in the __main__ module (via reticulate’s conversion rules).
• Julia: each element is julia_assign()ed as a variable in Main.

# R/iris_pre.R  (inputs = c(df = "iris_data"))
to_py <- list(
  df   = df,                       # R data.frame -> pandas DataFrame
  seed = targets::tar_seed_get(),  # reproducible integer for the Python RNG
  k    = 3L                        # note the L: Python wants an int, not a double
)

Check the reticulate / JuliaCall conversion rules for which R types round-trip; for anything that does not, use file mode.

Reading variables out (the post-script)

The post-script runs after the foreign script. tarpolyglot binds a few helpers into its environment:

• Python steps: py, the reticulate __main__ proxy (py$result), and py_get("result"), a shortcut for reticulate::py_to_r(py[["result"]]) that forces an explicit conversion.
• Julia steps: jl_get("name"), a shortcut for JuliaCall::julia_eval("name") (JuliaCall has no py-style proxy), and jl_call, an alias of JuliaCall::julia_call().

# R/iris_post.R  (Python)
res <- py_get("result")
data.frame(n = res$n, k = length(res$centers))   # last expression = target value

Rust is different: it has no pre-script. The #[extendr] functions are compiled and then called directly from the post-script, with the inputs in scope. See vignette("rust").