From Documentation
Jump to: navigation, search
(→‎Version Selection)
(→‎Installing a package as a new user)
 
(64 intermediate revisions by 3 users not shown)
Line 2: Line 2:
 
|package_name=NIX
 
|package_name=NIX
 
|package_description=User Level Purely Functional Package Manager
 
|package_description=User Level Purely Functional Package Manager
−
|package_idnumber=22
+
|package_idnumber=171
 
}}
 
}}
  
−
=Introduction=
+
= Introduction =
  
−
=Version Selection=
+
Nix is a package manager system that allows users to manage their own software environment.  This environment is persistent and is shared across all clusters.
  
−
Sets the path to nix scripts (coming soon)
+
* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
−
<pre>module load nix</pre>
+
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
 +
* Previous environments can be switched back to at any point.
 +
* Users can add their own packages and share them with other users.
  
−
=Job Submission=
+
The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
 +
== Enabling and disabling the Nix environment ==
  
−
=Example Job=
+
The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.
  
−
=General Notes=
+
<pre class="sh">module load nix
 +
ls -ld .nix*
 +
env | fgrep -i nix</pre>
 +
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.
 +
 
 +
<pre class="sh">module unload nix
 +
ls -ld .nix*
 +
env | fgrep -i nix
 +
module load nix</pre>
 +
== Completely reseting the Nix environment ==
 +
 
 +
Most operations can be undone with the <code>--rollback</code> option (i.e.,  <code>nix-env --rollback</code> or  <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.
 +
 
 +
<pre class="sh">module unload nix
 +
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
 +
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
 +
module load nix</pre>
 +
 
 +
= Installing and remove packages =
 +
 
 +
The <code>nix-env</code> command is used to setup your Nix environment.
 +
== What do I have installed and what can I install ==
 +
 
 +
Lets first see what we currently have installed.
 +
 
 +
<nowiki>nix-env --query</nowiki>
 +
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.
 +
 
 +
<nowiki>nix-env --query --available --attr-path --description</nowiki>
 +
== Installing packages ==
 +
 
 +
Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.
 +
 
 +
<nowiki>git --version
 +
which git</nowiki>
 +
Let's tell Nix to install its version in our environment.
 +
 
 +
<nowiki>nix-env --install --attr nixpkgs.git
 +
nix-env --query</nowiki>
 +
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).
 +
 
 +
<nowiki>git --version
 +
which git</nowiki>
 +
== Removing packages ==
 +
 
 +
For completeness, lets add in the other usual version-control suspects.
 +
 
 +
<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
 +
nix-env --query</nowiki>
 +
Actually, we probably don't really want subversion any more. Let's remove that.
 +
 
 +
<nowiki>nix-env --uninstall subversion
 +
nix-env --query</nowiki>
 +
= Environments =
 +
 
 +
Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
 +
== Switching between previous environments ==
 +
 
 +
This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.
 +
 
 +
<nowiki>nix-env --rollback
 +
nix-env --query</nowiki>
 +
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.
 +
 
 +
<nowiki>nix-env --list-generations
 +
nix-env --switch-generation 4
 +
nix-env --query</nowiki>
 +
== Operations are atomic ==
 +
 
 +
Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.
 +
 
 +
Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.
 +
 
 +
<nowiki>nix-env --switch-generation 1
 +
nix-env --install --attr nixpkgs.tla</nowiki>
 +
<blockquote>CTRL+c
 +
</blockquote>
 +
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.
 +
 
 +
<nowiki>nix-env --query
 +
nix-env --list-generations</nowiki>
 +
== Nix only does things once ==
 +
 
 +
The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.
 +
 
 +
<nowiki>nix-env --list-generations
 +
nix-env --install nixpkgs.git nixpkgs.mercurial
 +
nix-env --list-generations</nowiki>
 +
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
 +
== Garbage collection ==
 +
 
 +
Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.
 +
 
 +
<nowiki>nix-env --delete-generations 30d</nowiki>
 +
= Using channels to obtain Nix expressions =
 +
 
 +
Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
 +
== Subscribing to a channel ==
 +
 
 +
By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.
 +
 
 +
<nowiki>nix-channel --list
 +
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
 +
nix-channel --list
 +
nix-channel --update nixpkgs_old</nowiki>
 +
 
 +
== Installing packages from a channel ==
 +
 
 +
Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.
 +
 
 +
<nowiki>nix-env --query --available --attr-path --description git</nowiki>
 +
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).
 +
 
 +
<nowiki>nix-env --install --attr nixpkgs_old.git
 +
nix-env --query
 +
git --version</nowiki>
 +
 
 +
== What about dependency conflicts ==
 +
 
 +
There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.
 +
 
 +
<nowiki>ldd $(readlink ($which git))</nowiki>
 +
Switching between environments continue to work as before.
 +
 
 +
<nowiki>nix-env --rollback
 +
git --version
 +
nix-env --list-generations
 +
nix-env --switch-generation 5
 +
git --version</nowiki>
 +
== Switching between previous updates ==
 +
 
 +
The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.
 +
 
 +
<nowiki>nix-channel --rollback
 +
nix-env --query --available --attr-path git
 +
nix-channel --rollback 2</nowiki>
 +
 
 +
= Development with Nix (e.g., Python, R, etc.) =
 +
 
 +
Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
 +
== Wrapper scripts for setting paths ==
 +
 
 +
Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.
 +
 
 +
The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].
 +
 
 +
<pre class="nix">with import &lt;nixpkgs&gt; { };
 +
 
 +
rec {
 +
  myPython = python3.buildEnv.override rec {
 +
    extraLibs = with python3Packages; [ numpy scipy ];
 +
  };
 +
 
 +
  myR = rWrapper.override rec {
 +
    packages = with rPackages; [ rgeos rgdal ];
 +
  };
 +
 
 +
  myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
 +
    with ghcpkgs; [ pipes lens cabal-install ]
 +
  );
 +
}</pre>
 +
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.
 +
 
 +
<nowiki>nix-env --install --attr mypkgs.myPython
 +
cat $(which python3)
 +
python3</nowiki>
 +
<nowiki>import scipy</nowiki>
 +
<blockquote>CTRL+d
 +
</blockquote>
 +
== Development environments ==
 +
 
 +
A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.
 +
 
 +
The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.
 +
 
 +
<nowiki>mkdir devel
 +
cd devel</nowiki>
 +
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs
 +
 
 +
<pre class="nix">with import &lt;nixpkgs&gt; { };
 +
 
 +
stdenv.mkDerivation {
 +
  name = &quot;my-env&quot;;
 +
  buildInputs = [
 +
    ( python3.buildEnv.override rec {
 +
        extraLibs = with python3Packages; [ numpy scipy ];
 +
      } )
 +
    clang
 +
  ];
 +
}</pre>
 +
save it in ''default.nix'' and then run <code>nix-shell</code>.
 +
 
 +
<nowiki>nix-shell</nowiki>
 +
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.
 +
 
 +
<nowiki>cat $(which python3)
 +
clang --version
 +
python3</nowiki>
 +
<nowiki>import scipy</nowiki>
 +
<blockquote>CTRL+d
 +
</blockquote>
 +
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.
 +
 
 +
<nowiki>exit
 +
cd ..</nowiki>
 +
== One-off environments ==
 +
 
 +
Quite frequently we need a one-off development environment with a few packages.  Say CLang and git.  Rather than have to write the following boilerplate ''default.nix'' file
 +
 
 +
<pre class="nix">with import &lt;nixpkgs&gt; { };
 +
 
 +
stdenv.mkDerivation {
 +
  name = &quot;my-env&quot;;
 +
  buildInputs = [
 +
    clang
 +
    git
 +
  ];
 +
}</pre>
 +
we can just get <code>nix-shell</code> to do it for us.
 +
 
 +
<pre class="sh">nix-shell --packages clang git</pre>
 +
<blockquote>CTRL+d
 +
</blockquote>
 +
 
 +
= Submitting Jobs =
 +
 
 +
== With sqsub ==
 +
 
 +
Loading the Nix module before submitting a job makes the Nix environment available to the job.  Note that these jobs will see the current Nix environment including any changes made after submission.  Compiled binaries should not require the Nix module to be loaded to run.
 +
 
 +
<pre class="sh">module load nix
 +
sqsub ...</pre>
 +
 
 +
= Internals =
 +
 
 +
This section details some of the internals of how Nix works and how to create your own packages.  It is more advanced material not required for the basic usage.
 +
== The Nix store ==
 +
 
 +
A Nix environment is just a collection of symlinks to all the packages that exist in that environment.
 +
 
 +
<pre class="sh">readlink $(which git)
 +
readlink -f ~/.nix-profile</pre>
 +
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.
 +
 
 +
Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
 +
== Nix expressions and instantiation ==
 +
 
 +
Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.
 +
 
 +
<pre class="sh">nix-env --install --attr nix-repl
 +
nix-repl</pre>
 +
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.
 +
 
 +
<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
 +
stdenv.mkDerivation rec {
 +
  name = &quot;cabextract-1.6&quot;
 +
 
 +
  src = fetchurl {
 +
    url = &quot;http://www.cabextract.org.uk/${name}.tar.gz&quot;;
 +
    sha256 = &quot;1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf&quot;;
 +
  };
 +
 
 +
  meta = with stdenv.lib; {
 +
    homepage = http://www.cabextract.org.uk/;
 +
    description = &quot;Free Software for extracting Microsoft cabinet files&quot;;
 +
    platforms = platforms.all;
 +
    license = licenses.gpl3;
 +
    maintainers = with maintainers; [ pSub ];
 +
  };
 +
}</pre>
 +
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages.  For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].
 +
 
 +
The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.
 +
 
 +
<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
 +
echo $cabdrv</pre>
 +
 
 +
== Nix derivations and realization ==
 +
 
 +
The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.
 +
 
 +
<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
 +
pp-aterm -i $cabdrv</pre>
 +
<pre>Derive(
 +
  [(&quot;out&quot;, &quot;/home/nixbld/store/...-cabextract-1.6&quot;, &quot;&quot;, &quot;&quot;)]
 +
, [ (&quot;/home/nixbld/store/...-stdenv.drv&quot;, [&quot;out&quot;])
 +
  , (&quot;/home/nixbld/store/...-cabextract-1.6.tar.gz.drv&quot;, [&quot;out&quot;])
 +
  , (&quot;/home/nixbld/store/...-bash-4.3-p42.drv&quot;, [&quot;out&quot;])
 +
  ]
 +
, [&quot;/home/nixbld/store/...-default-builder.sh&quot;]
 +
, &quot;x86_64-linux&quot;
 +
, &quot;/home/nixbld/store/...-bash-4.3-p42/bin/bash&quot;
 +
, [&quot;-e&quot;, &quot;...-default-builder.sh&quot;]
 +
, [ (&quot;buildInputs&quot;, &quot;&quot;)
 +
  , (&quot;builder&quot;, &quot;/home/nixbld/store/...-bash-4.3-p42/bin/bash&quot;)
 +
  , (&quot;name&quot;, &quot;cabextract-1.6&quot;)
 +
  , (&quot;nativeBuildInputs&quot;, &quot;&quot;)
 +
  , (&quot;out&quot;, &quot;/home/nixbld/store/...-cabextract-1.6&quot;)
 +
  , (&quot;propagatedBuildInputs&quot;, &quot;&quot;)
 +
  , (&quot;propagatedNativeBuildInputs&quot;, &quot;&quot;)
 +
  , (&quot;src&quot;, &quot;/home/nixbld/store/...-cabextract-1.6.tar.gz&quot;)
 +
  , (&quot;stdenv&quot;, &quot;/home/nixbld/store/...-stdenv&quot;)
 +
  , (&quot;system&quot;, &quot;x86_64-linux&quot;)
 +
  ]
 +
)</pre>
 +
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.
 +
 
 +
The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.
 +
 
 +
<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
 +
echo $cabpath</pre>
 +
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.
 +
 
 +
<pre class="sh">nix-env --install $cabpath</pre>
 +
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.
 +
 
 +
<pre class="sh">nix-store --read-log $cabdrv
 +
nix-store --read-log $cabpath
 +
nix-store --read-log $(which git)</pre>
 +
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.
 +
 
 +
<pre class="sh">nix-store --query --requisites $cabpath
 +
nix-store --export $(nix-store --query --requisites $(which git)) | gzip &gt; git.nar.gz</pre>
 +
== Nix default expression ==
 +
 
 +
Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.
 +
 
 +
The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.
 +
 
 +
<pre class="nix">args@{ ... }: with import ./nixpkgs args;
 +
rec {
 +
  cabextract = stdenv.mkDerivation rec {
 +
    name = &quot;cabextract-1.6&quot;;
 +
 
 +
    src = fetchurl {
 +
      url = &quot;http://www.cabextract.org.uk/${name}.tar.gz&quot;;
 +
      sha256 = &quot;1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf&quot;;
 +
    };
 +
 
 +
    meta = with stdenv.lib; {
 +
      homepage = http://www.cabextract.org.uk/;
 +
      description = &quot;Free Software for extracting Microsoft cabinet files&quot;;
 +
      platforms = platforms.all;
 +
      license = licenses.gpl3;
 +
      maintainers = with maintainers; [ pSub ];
 +
    };
 +
  };
 +
}</pre>
 +
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.
 +
 
 +
<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
 +
 
 +
= Examples =
 +
 
 +
== Installing a package as a new user ==
 +
 
 +
<pre>
 +
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
 +
[roberpj@gra-login1:~] module load nix
 +
[roberpj@gra-login1:~] nix-env --query
 +
atop-2.3.0
 +
nix-index-0.1.0
 +
[roberpj@gra-login1:~] nix-env --uninstall atop
 +
uninstalling 'atop-2.3.0'
 +
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
 +
uninstalling 'nixpkgs'
 +
 
 +
2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
 +
[roberpj@gra-login1:~] module load nix
 +
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
 +
[roberpj@gra-login1:~] nix-channel --update
 +
unpacking channels...
 +
 
 +
3) Next install package "atop" or any other nix package for that matter !!!
 +
[roberpj@gra-login1:~] module load nix
 +
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
 +
replacing old 'atop-2.3.0'
 +
installing 'atop-2.3.0'
 +
[roberpj@gra-login1:~] which atop
 +
~/.nix-profile/bin/atop
 +
 
 +
4) For future logins, simply load the nix module and atop will be in your path ;)
 +
[roberpj@laptop:~] ssh graham.sharcnet.ca
 +
[roberpj@gra-login1:~] module load nix
 +
[roberpj@gra-login1:~] which atop
 +
~/.nix-profile/bin/atop
 +
</pre>
 +
 
 +
The above steps can also be followed to start over if a previously installed nix package is not showing up in your path as expected.
  
 
=References=
 
=References=
Line 25: Line 416:
 
o Official Nix/Nixpkgs/NixOS<br>
 
o Official Nix/Nixpkgs/NixOS<br>
 
https://github.com/NixOS
 
https://github.com/NixOS
 +
 +
o Nix on SHARCNET (slides by Tyson Whitehead)<br>
 +
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf
 +
 +
o Exploring a new approach to package management (youtube video by Tyson Whitehead)<br>
 +
https://www.youtube.com/watch?v=pQE9WTLAPHQ

Latest revision as of 21:12, 12 December 2018

NIX
Description: User Level Purely Functional Package Manager
SHARCNET Package information: see NIX software page in web portal
Full list of SHARCNET supported software


Introduction

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

  • Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
  • Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
  • Previous environments can be switched back to at any point.
  • Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.

Enabling and disabling the Nix environment

The user's current Nix environment is enabled by loading the nix module. This creates some .nix* files and sets some environment variables. 

module load nix
ls -ld .nix*
env | fgrep -i nix

It is disabled by unloading the nix module. This unsets the environment variables but leaves the .nix* files alone. 

module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix

Completely reseting the Nix environment

Most operations can be undone with the --rollback option (i.e., nix-env --rollback or nix-channel --rollback). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file. 

module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix

Installing and remove packages

The nix-env command is used to setup your Nix environment.

What do I have installed and what can I install

Lets first see what we currently have installed. 

nix-env --query

Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future. 

nix-env --query --available --attr-path --description

Installing packages

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with. 

git --version
which git

Let's tell Nix to install its version in our environment. 

nix-env --install --attr nixpkgs.git
nix-env --query

Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with hash -r so it notices the new one). 

git --version
which git

Removing packages

For completeness, lets add in the other usual version-control suspects. 

nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query

Actually, we probably don't really want subversion any more. Let's remove that. 

nix-env --uninstall subversion
nix-env --query

Environments

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.

Switching between previous environments

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment. 

nix-env --rollback
nix-env --query

Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback. 

nix-env --list-generations
nix-env --switch-generation 4
nix-env --query

Operations are atomic

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with CTRL+c partway through. 

nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla
CTRL+c

Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever. 

nix-env --query
nix-env --list-generations

Nix only does things once

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial. 

nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations

Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.

Garbage collection

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old. 

nix-env --delete-generations 30d

Using channels to obtain Nix expressions

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for nix-env is ~/.nix-defexp and ~/.nix-defexp/channels. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.

Subscribing to a channel

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions. 

nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old

Installing packages from a channel

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions. 

nix-env --query --available --attr-path --description git

Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal). 

nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version

What about dependency conflicts

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under /nix/store versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need. 

ldd $(readlink ($which git))

Switching between environments continue to work as before. 

nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version

Switching between previous updates

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken. 

nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2

Development with Nix (e.g., Python, R, etc.)

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.

Wrapper scripts for setting paths

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the Nix Expression Language. 

with import <nixpkgs> { };

rec {
  myPython = python3.buildEnv.override rec {
    extraLibs = with python3Packages; [ numpy scipy ];
  };

  myR = rWrapper.override rec {
    packages = with rPackages; [ rgeos rgdal ];
  };

  myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
    with ghcpkgs; [ pipes lens cabal-install ]
  );
}

Saving it as ~/.nix-defexpr/mypkgs.nix makes these available for installation with nix-env. 

nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3
import scipy
CTRL+d

Development environments

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The nix-shell command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process. 

mkdir devel
cd devel

Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs 

with import <nixpkgs> { };

stdenv.mkDerivation {
  name = "my-env";
  buildInputs = [
    ( python3.buildEnv.override rec {
        extraLibs = with python3Packages; [ numpy scipy ];
      } )
    clang
  ];
}

save it in default.nix and then run nix-shell. 

nix-shell

Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs. 

cat $(which python3)
clang --version
python3
import scipy
CTRL+d

To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies. 

exit
cd ..

One-off environments

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate default.nix file 

with import <nixpkgs> { };

stdenv.mkDerivation {
  name = "my-env";
  buildInputs = [
    clang
    git
  ];
}

we can just get nix-shell to do it for us. 

nix-shell --packages clang git
CTRL+d

Submitting Jobs

With sqsub

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run. 

module load nix
sqsub ...

Internals

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.

The Nix store

A Nix environment is just a collection of symlinks to all the packages that exist in that environment. 

readlink $(which git)
readlink -f ~/.nix-profile

Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.

Nix expressions and instantiation

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The nix-repl program can be used to interactively experiment with the Nix language. 

nix-env --install --attr nix-repl
nix-repl

The following cabextract.nix Nix expression evaluates to a derivation fully specifying how to build the cabextract utility. 

with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
  name = "cabextract-1.6"

  src = fetchurl {
    url = "http://www.cabextract.org.uk/${name}.tar.gz";
    sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
  };

  meta = with stdenv.lib; {
    homepage = http://www.cabextract.org.uk/;
    description = "Free Software for extracting Microsoft cabinet files";
    platforms = platforms.all;
    license = licenses.gpl3;
    maintainers = with maintainers; [ pSub ];
  };
}

It is written using the stdenv.mkDerivation function from nixpkgs. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the nixpkgs manual.

The Nix expression is instantiate to a derivation using nix-instantiate. The leading ./ is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate. 

cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv

Nix derivations and realization

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the pp-aterm program. 

nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv
Derive(
  [("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
  , ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
  , ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
  ]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
  , ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
  , ("name", "cabextract-1.6")
  , ("nativeBuildInputs", "")
  , ("out", "/home/nixbld/store/...-cabextract-1.6")
  , ("propagatedBuildInputs", "")
  , ("propagatedNativeBuildInputs", "")
  , ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
  , ("stdenv", "/home/nixbld/store/...-stdenv")
  , ("system", "x86_64-linux")
  ]
)

The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The nix-store program --realize option is used to signal the build server to realize the derivation. 

cabpath=$(nix-store --realize $cabdrv)
echo $cabpath

The nix-env program will add the realized derivation to the users environment with the --install (-i) option. Technically it creates a new realization of the user-environment package that symlinks in the entire contents of the path path and switches to it. 

nix-env --install $cabpath

The build log associated with a realization can be viewed with the --read-log (-l) option. 

nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)

The nix-store program also supports numerous other store related items such as computing the transitive closure of a package with the --query (-q) --requisites (-R) and exporting them via the --export option to a Nix archive for import into another store. 

nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz

Nix default expression

Nix has a default expression created from the contents of ~/.nix-defexpr. The nix-env --install operation normally works by selecting an attribute in this expression (fast) or matching against the name attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The nix-channel command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ~/.nix-defexprs/channels to then. Adding additional expressions to ~/.nix-defexpr makes them available for use with nix-env as well. For example, the following ~/.nix-defexpr/mypkgs expression packages up the above cabextract.nix example. 

args@{ ... }: with import ./nixpkgs args;
rec {
  cabextract = stdenv.mkDerivation rec {
    name = "cabextract-1.6";

    src = fetchurl {
      url = "http://www.cabextract.org.uk/${name}.tar.gz";
      sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
    };

    meta = with stdenv.lib; {
      homepage = http://www.cabextract.org.uk/;
      description = "Free Software for extracting Microsoft cabinet files";
      platforms = platforms.all;
      license = licenses.gpl3;
      maintainers = with maintainers; [ pSub ];
    };
  };
}

This makes it possible to install packages from mypkgs as easily as those from the official nixpkgs. 

nix-env --install --attr mypkgs.cabextract

Examples

Installing a package as a new user

1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) For future logins, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

The above steps can also be followed to start over if a previously installed nix package is not showing up in your path as expected.

References

o Nix Package Manager
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead)
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead)
https://www.youtube.com/watch?v=pQE9WTLAPHQ

Retrieved from "https://www.sharcnet.ca/help/index.php?title=NIX&oldid=17221"