4. Working with Environment

The following discusses Pkg's interaction with environments. For more on the role, environments play in code loading, including the "stack" of environments from which code can be loaded, see this section in the Julia manual.

Creating your own environments

So far we have added packages to the default environment at ~/.julia/environments/v1.8. It is however easy to create other, independent, projects. This approach has the benefit of allowing you to check in a Project.toml, and even a Manifest.toml if you wish, into version control (e.g. git) alongside your code. It should be pointed out that when two projects use the same package at the same version, the content of this package is not duplicated. In order to create a new project, create a directory for it and then activate that directory to make it the "active project", which package operations manipulate:

(@v1.8) pkg> activate MyProject
Activating new environment at `~/MyProject/Project.toml`

(MyProject) pkg> st
    Status `~/MyProject/Project.toml` (empty project)

Note that the REPL prompt changes when the new project is activated. Until a package is added, there are no files in this environment and the directory to the environment might not even be created:

julia> isdir("MyProject")
false

(MyProject) pkg> add Example
   Resolving package versions...
   Installed Example ─ v0.5.3
    Updating `~/MyProject/Project.toml`
  [7876af07] + Example v0.5.3
    Updating `~~/MyProject/Manifest.toml`
  [7876af07] + Example v0.5.3
Precompiling environment...
  1 dependency successfully precompiled in 2 seconds

julia> readdir("MyProject")
2-element Vector{String}:
 "Manifest.toml"
 "Project.toml"

julia> print(read(joinpath("MyProject", "Project.toml"), String))
[deps]
Example = "7876af07-990d-54b4-ab0e-23690620f79a"

julia> print(read(joinpath("MyProject", "Manifest.toml"), String))
# This file is machine-generated - editing it directly is not advised

julia_version = "1.8.2"
manifest_format = "2.0"
project_hash = "2ca1c6c58cb30e79e021fb54e5626c96d05d5fdc"

[[deps.Example]]
git-tree-sha1 = "46e44e869b4d90b96bd8ed1fdcf32244fddfb6cc"
uuid = "7876af07-990d-54b4-ab0e-23690620f79a"
version = "0.5.3"

This new environment is completely separate from the one we used earlier. See Project.toml and Manifest.toml for a more detailed explanation.

Using someone else's project

Simply clone their project using e.g. git clone, cd to the project directory and call

shell> git clone https://github.com/JuliaLang/Example.jl.git
Cloning into 'Example.jl'...
...

(@v1.8) pkg> activate Example.jl
Activating project at `~/Example.jl`

(Example) pkg> instantiate
  No Changes to `~/Example.jl/Project.toml`
  No Changes to `~/Example.jl/Manifest.toml`

If the project contains a manifest, this will install the packages in the same state that is given by that manifest. Otherwise, it will resolve the latest versions of the dependencies compatible with the project.

Note that activate by itself does not install missing dependencies. If you only have a Project.toml, a Manifest.toml must be generated by "resolving" the environment, then any missing packages must be installed and precompiled. instantiate does all this for you.

If you already have a resolved Manifest.toml, then you will still need to ensure that the packages are installed and with the correct versions. Again instantiate does this for you.

In short, instantiate is your friend to make sure an environment is ready to use. If there's nothing to do, instantiate does nothing.

$ julia --project=. myscript.jl

Temporary environments

Temporary environments make it easy to start an environment from a blank slate to test a package or set of packages, and have Pkg automatically delete the environment when you're done. For instance, when writing a bug report, you may want to test your minimal reproducible example in a 'clean' environment to ensure it's actually reproducible as written. You might also want a scratch space to try out a new package, or a sandbox to resolve version conflicts between several incompatible packages.

(@v1.8) pkg> activate --temp # requires Julia 1.5 or later
  Activating new environment at `/var/folders/34/km3mmt5930gc4pzq1d08jvjw0000gn/T/jl_a31egx/Project.toml`

(jl_a31egx) pkg> add Example
    Updating registry at `~/.julia/registries/General`
   Resolving package versions...
    Updating `/private/var/folders/34/km3mmt5930gc4pzq1d08jvjw0000gn/T/jl_a31egx/Project.toml`
  [7876af07] + Example v0.5.3
    Updating `/private/var/folders/34/km3mmt5930gc4pzq1d08jvjw0000gn/T/jl_a31egx/Manifest.toml`
  [7876af07] + Example v0.5.3

Shared environments

A "shared" environment is simply an environment that exists in ~/.julia/environments. The default v1.8 environment is therefore a shared environment:

(@v1.8) pkg> st
Status `~/.julia/environments/v1.8/Project.toml`

Shared environments can be activated with the --shared flag to activate:

(@v1.8) pkg> activate --shared mysharedenv
  Activating project at `~/.julia/environments/mysharedenv`

(@mysharedenv) pkg>

Shared environments have a @ before their name in the Pkg REPL prompt.

Environment Precompilation

Before a package can be imported, Julia will "precompile" the source code into an intermediate more efficient cache on disc. This precompilation can be triggered via code loading if the un-imported package is new or has changed since the last cache

julia> using Example
[ Info: Precompiling Example [7876af07-990d-54b4-ab0e-23690620f79a]

or using Pkg's precompile option, which can precompile the entire environment, or a given dependency, and do so in parallel, which can be significantly faster than the code-load route above.

(@v1.8) pkg> precompile
Precompiling environment...
  23 dependencies successfully precompiled in 36 seconds

However, neither of these should be routinely required thanks to Pkg's automatic precompilation.

Automatic Precompilation

By default, any package that is added to a project or updated in a Pkg action will be automatically precompiled, along with its dependencies.

(@v1.8) pkg> add Images
   Resolving package versions...
    Updating `~/.julia/environments/v1.9/Project.toml`
  [916415d5] + Images v0.25.2
    Updating `~/.julia/environments/v1.9/Manifest.toml`
    ...
Precompiling environment...
  Progress [===================>                     ]  45/97
  ✓ NaNMath
  ✓ IntervalSets
  ◐ CoordinateTransformations
  ◑ ArnoldiMethod
  ◑ IntegralArrays
  ◒ RegionTrees
  ◐ ChangesOfVariables
  ◓ PaddedViews

The exception is the develop command, which neither builds nor precompiles the package. When that happens is left up to the user to decide.

If a given package version errors during auto-precompilation, Pkg will remember for the following times it automatically tries and will skip that package with a brief warning. Manual precompilation can be used to force these packages to be retried, as pkg> precompile will always retry all packages.

To disable the auto-precompilation, set ENV["JULIA_PKG_PRECOMPILE_AUTO"]=0.

Precompiling new versions of loaded packages

If a package that has been updated is already loaded in the session, the precompilation process will go ahead and precompile the new version, and any packages that depend on it, but will note that the package cannot be used until session restart.