From Documentation
Jump to: navigation, search
(Version Selection)
(Flush out content from CO mini TECC talk)
Line 6: Line 6:
  
 
=Introduction=
 
=Introduction=
 +
 +
Nix is an atomic pure non-mutable package manager system that allows users to manage their own software environment.  This environment is persistent and is shared across all clusters.
 +
 +
* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users
 +
* Operations are atomic, they either succeed and create a new environment or fail leaving the previous environment in place.  Previous environments can be switched back to at any point.
 +
 +
The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
  
 
=Version Selection=
 
=Version Selection=
  
Sets the path to nix scripts (orca only):
+
This section details how to enable your Nix environment and install and remove packages from it.  See the [[#General Notes]] section for a more advanced discussion of how Nix works and how to create your own packages.
<pre>module load nix</pre>
+
  
Tab complete then gives the following:
+
== Enabling and disabling the Nix environment ==
<pre>
+
 
[roberpj@orc-login2:~] nix-
+
The user's current Nix environment is enabled by loading the nix module. This creates some *.nix** files and sets some environment variables.
nix-build            nix-env               nix-log2xml          nix-store
+
 
nix-channel          nix-generate-patches  nix-prefetch-url      nix-worker
+
<pre class="sh">module load nix
nix-collect-garbage  nix-hash              nix-pull             
+
ls -ld .nix*
nix-copy-closure      nix-install-package  nix-push             
+
env | fgrep -i nix</pre>
nix-daemon            nix-instantiate      nix-shell
+
It is disabled by unloading the nix module. This unsets the environment variables but leaves the *.nix** files alone.
</pre>
+
 
 +
<pre class="sh">module unload nix
 +
ls -ld .nix*
 +
env | fgrep -i nix
 +
module load nix</pre>
 +
 
 +
== Installing, upgrading, uninstalling, and querying software ==
 +
 
 +
The nix environment is manipulated via the <code>nix-env</code> command. The <code>--install</code> (<code>-i</code>) option is used to install software
 +
 
 +
<pre class="sh">git --version
 +
which git</pre>
 +
<pre class="sh">nix-env --install git
 +
git --version
 +
which git</pre>
 +
This takes a long time as it has to open many many files to enumerate all available software to locate the requested one by name. A faster way is to directly specify what to install by the attribute instead using the <code>--attr</code> (<code>-a</code>) option
 +
 
 +
<pre class="sh">nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial</pre>
 +
The <code>--upgrade</code> (<code>-u</code>) option upgrades installed software to the latest version. The <code>--install</code> (<code>-i</code>) option can also be used for this purpose.
 +
 
 +
<pre class="sh">nix-env --upgrade --attr nixpkgs.git</pre>
 +
The <code>--uninstall</code> (<code>-e</code>) option uninstalls software. It can be done by name or path.
 +
 
 +
<pre class="sh">nix-env --uninstall $(which svn)</pre>
 +
The contents of the environment is queried via the <code>--query</code> (<code>-q</code>) option.
 +
 
 +
<pre class="sh">nix-env --query</pre>
 +
All possible packages can be queried via the <code>--available</code> (<code>-a</code>) option. This is quite slow again due to having to enumerate all available software. Adding <code>--attr-path</code> (<code>-P</code>) includes the faster attribute paths and <code>--description</code> includes the descriptions. It is a good idea to save this to a file for future reference.
 +
 
 +
<pre class="sh">nix-env --query --available --description --attr-path &gt; nix-packages.txt
 +
grep git nix-packages.txt | less -S</pre>
 +
 
 +
== Environment transactions ==
 +
 
 +
Environments are non-mutable. All <code>nix-env</code> commands do not actually modify the current environment. Rather they create a new environment based on the existing environment and switch to it. This means all previous environments continue to exist and can be re-activated.
 +
 
 +
The <code>--rollback</code> option can be used to restore the previous environment.
 +
 
 +
<pre class="sh">nix-env --query
 +
nix-env --rollback
 +
nix-env --query</pre>
 +
Previous generations of the environment can be displayed with the <code>--list-generations</code> option and any one can be switched to with the <code>--switch-generation</code> option.
 +
 
 +
<pre class="sh">nix-env --list-generations
 +
nix-env --switch-generation 3
 +
nix-env --query</pre>
 +
Nix preserves any software this is accessible for any user environment. This means it is important to periodically delete older environment generations so Nix can remove the software associated with them. This is done with the <code>--delete-generations</code> options.
 +
 
 +
<pre class="sh">nix-env --delete-generations 30d</pre>
 +
Environments are technically realizations of the Nix user-environment package.
  
 
=Job Submission=
 
=Job Submission=
 +
 +
Loading the Nix module before submitting a job makes the Nix environment available to the job.  Note that these jobs will see the current Nix environment including any changes made after submission.  Compiled binaries should not require the Nix module to be loaded to run..
  
 
=Example Job=
 
=Example Job=
  
=General Notes=
+
<pre class="sh">module load nix
 +
sqsub ...</pre>
 +
 
 +
= General Notes =
 +
 
 +
This section details some of the internals of how Nix works and how to create your own packages.  It is more advanced material not required for the basic usage detailed under the [[#Version Selection]] section above.
 +
 
 +
== The Nix store ==
 +
 
 +
A Nix environment is just a collection of symlinks to all the packages that exist in that environment.
 +
 
 +
<pre class="sh">readlink $(which git)
 +
readlink -f ~/.nix-profile</pre>
 +
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.
 +
 
 +
Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
 +
 
 +
== Nix expressions and instantiation ==
 +
 
 +
Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.
 +
 
 +
<pre class="sh">nix-env --install --attr nix-repl
 +
nix-repl</pre>
 +
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.
 +
 
 +
<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
 +
stdenv.mkDerivation rec {
 +
  name = &quot;cabextract-1.6&quot;
 +
 
 +
  src = fetchurl {
 +
    url = &quot;http://www.cabextract.org.uk/${name}.tar.gz&quot;;
 +
    sha256 = &quot;1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf&quot;;
 +
  };
 +
 
 +
  meta = with stdenv.lib; {
 +
    homepage = http://www.cabextract.org.uk/;
 +
    description = &quot;Free Software for extracting Microsoft cabinet files&quot;;
 +
    platforms = platforms.all;
 +
    license = licenses.gpl3;
 +
    maintainers = with maintainers; [ pSub ];
 +
  };
 +
}</pre>
 +
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages.
 +
 
 +
The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.
 +
 
 +
<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
 +
echo $cabdrv</pre>
 +
 
 +
== Nix derivations and realization ==
 +
 
 +
The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.
 +
 
 +
<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
 +
pp-aterm -i $cabdrv</pre>
 +
<pre>Derive(
 +
  [(&quot;out&quot;, &quot;/home/nixbld/store/...-cabextract-1.6&quot;, &quot;&quot;, &quot;&quot;)]
 +
, [ (&quot;/home/nixbld/store/...-stdenv.drv&quot;, [&quot;out&quot;])
 +
  , (&quot;/home/nixbld/store/...-cabextract-1.6.tar.gz.drv&quot;, [&quot;out&quot;])
 +
  , (&quot;/home/nixbld/store/...-bash-4.3-p42.drv&quot;, [&quot;out&quot;])
 +
  ]
 +
, [&quot;/home/nixbld/store/...-default-builder.sh&quot;]
 +
, &quot;x86_64-linux&quot;
 +
, &quot;/home/nixbld/store/...-bash-4.3-p42/bin/bash&quot;
 +
, [&quot;-e&quot;, &quot;...-default-builder.sh&quot;]
 +
, [ (&quot;buildInputs&quot;, &quot;&quot;)
 +
  , (&quot;builder&quot;, &quot;/home/nixbld/store/...-bash-4.3-p42/bin/bash&quot;)
 +
  , (&quot;name&quot;, &quot;cabextract-1.6&quot;)
 +
  , (&quot;nativeBuildInputs&quot;, &quot;&quot;)
 +
  , (&quot;out&quot;, &quot;/home/nixbld/store/...-cabextract-1.6&quot;)
 +
  , (&quot;propagatedBuildInputs&quot;, &quot;&quot;)
 +
  , (&quot;propagatedNativeBuildInputs&quot;, &quot;&quot;)
 +
  , (&quot;src&quot;, &quot;/home/nixbld/store/...-cabextract-1.6.tar.gz&quot;)
 +
  , (&quot;stdenv&quot;, &quot;/home/nixbld/store/...-stdenv&quot;)
 +
  , (&quot;system&quot;, &quot;x86_64-linux&quot;)
 +
  ]
 +
)</pre>
 +
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.
 +
 
 +
The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.
 +
 
 +
<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
 +
echo $cabpath</pre>
 +
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.
 +
 
 +
<pre class="sh">nix-env --install $cabpath</pre>
 +
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.
 +
 
 +
<pre class="sh">nix-store --read-log $cabdrv
 +
nix-store --read-log $cabpath
 +
nix-store --read-log $(which git)</pre>
 +
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.
 +
 
 +
<pre class="sh">nix-store --query --requisites $cabpath
 +
nix-store --export $(nix-store --query --requisites $(which git)) | gzip &gt; git.nar.gz</pre>
 +
 
 +
== Nix default expression ==
 +
 
 +
Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.
 +
 
 +
The <code>nix</code> module sets up a default ''~/.nix-defexpr/nixpkgs'' that symlinks to a slightly modified version of the official Nix ''nixpkgs''. This can be entirely replace this or augment. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.
 +
 
 +
<pre class="nix">args@{ ... }: with import ./nixpkgs args;
 +
rec {
 +
  cabextract = stdenv.mkDerivation rec {
 +
    name = &quot;cabextract-1.6&quot;;
 +
 
 +
    src = fetchurl {
 +
      url = &quot;http://www.cabextract.org.uk/${name}.tar.gz&quot;;
 +
      sha256 = &quot;1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf&quot;;
 +
    };
 +
 
 +
    meta = with stdenv.lib; {
 +
      homepage = http://www.cabextract.org.uk/;
 +
      description = &quot;Free Software for extracting Microsoft cabinet files&quot;;
 +
      platforms = platforms.all;
 +
      license = licenses.gpl3;
 +
      maintainers = with maintainers; [ pSub ];
 +
    };
 +
  };
 +
}</pre>
 +
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.
  
 +
<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
 
=References=
 
=References=
  

Revision as of 17:52, 2 December 2015

NIX
Description: User Level Purely Functional Package Manager
SHARCNET Package information: see NIX software page in web portal
Full list of SHARCNET supported software


Introduction

Nix is an atomic pure non-mutable package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

  • Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users
  • Operations are atomic, they either succeed and create a new environment or fail leaving the previous environment in place. Previous environments can be switched back to at any point.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.

Version Selection

This section details how to enable your Nix environment and install and remove packages from it. See the #General Notes section for a more advanced discussion of how Nix works and how to create your own packages.

Enabling and disabling the Nix environment

The user's current Nix environment is enabled by loading the nix module. This creates some *.nix** files and sets some environment variables.

module load nix
ls -ld .nix*
env | fgrep -i nix

It is disabled by unloading the nix module. This unsets the environment variables but leaves the *.nix** files alone.

module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix

Installing, upgrading, uninstalling, and querying software

The nix environment is manipulated via the nix-env command. The --install (-i) option is used to install software

git --version
which git
nix-env --install git
git --version
which git

This takes a long time as it has to open many many files to enumerate all available software to locate the requested one by name. A faster way is to directly specify what to install by the attribute instead using the --attr (-a) option

nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial

The --upgrade (-u) option upgrades installed software to the latest version. The --install (-i) option can also be used for this purpose.

nix-env --upgrade --attr nixpkgs.git

The --uninstall (-e) option uninstalls software. It can be done by name or path.

nix-env --uninstall $(which svn)

The contents of the environment is queried via the --query (-q) option.

nix-env --query

All possible packages can be queried via the --available (-a) option. This is quite slow again due to having to enumerate all available software. Adding --attr-path (-P) includes the faster attribute paths and --description includes the descriptions. It is a good idea to save this to a file for future reference.

nix-env --query --available --description --attr-path > nix-packages.txt
grep git nix-packages.txt | less -S

Environment transactions

Environments are non-mutable. All nix-env commands do not actually modify the current environment. Rather they create a new environment based on the existing environment and switch to it. This means all previous environments continue to exist and can be re-activated.

The --rollback option can be used to restore the previous environment.

nix-env --query
nix-env --rollback
nix-env --query

Previous generations of the environment can be displayed with the --list-generations option and any one can be switched to with the --switch-generation option.

nix-env --list-generations
nix-env --switch-generation 3
nix-env --query

Nix preserves any software this is accessible for any user environment. This means it is important to periodically delete older environment generations so Nix can remove the software associated with them. This is done with the --delete-generations options.

nix-env --delete-generations 30d

Environments are technically realizations of the Nix user-environment package.

Job Submission

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run..

Example Job

module load nix
sqsub ...

General Notes

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage detailed under the #Version Selection section above.

The Nix store

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

readlink $(which git)
readlink -f ~/.nix-profile

Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.

Nix expressions and instantiation

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The nix-repl program can be used to interactively experiment with the Nix language.

nix-env --install --attr nix-repl
nix-repl

The following cabextract.nix Nix expression evaluates to a derivation fully specifying how to build the cabextract utility.

with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
  name = "cabextract-1.6"

  src = fetchurl {
    url = "http://www.cabextract.org.uk/${name}.tar.gz";
    sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
  };

  meta = with stdenv.lib; {
    homepage = http://www.cabextract.org.uk/;
    description = "Free Software for extracting Microsoft cabinet files";
    platforms = platforms.all;
    license = licenses.gpl3;
    maintainers = with maintainers; [ pSub ];
  };
}

It is written using the stdenv.mkDerivation function from nixpkgs. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages.

The Nix expression is instantiate to a derivation using nix-instantiate. The leading ./ is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv

Nix derivations and realization

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the pp-aterm program.

nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv
Derive(
  [("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
  , ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
  , ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
  ]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
  , ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
  , ("name", "cabextract-1.6")
  , ("nativeBuildInputs", "")
  , ("out", "/home/nixbld/store/...-cabextract-1.6")
  , ("propagatedBuildInputs", "")
  , ("propagatedNativeBuildInputs", "")
  , ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
  , ("stdenv", "/home/nixbld/store/...-stdenv")
  , ("system", "x86_64-linux")
  ]
)

The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The nix-store program --realize option is used to signal the build server to realize the derivation.

cabpath=$(nix-store --realize $cabdrv)
echo $cabpath

The nix-env program will add the realized derivation to the users environment with the --install (-i) option. Technically it creates a new realization of the user-environment package that symlinks in the entire contents of the path path and switches to it.

nix-env --install $cabpath

The build log associated with a realization can be viewed with the --read-log (-l) option.

nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)

The nix-store program also supports numerous other store related items such as computing the transitive closure of a package with the --query (-q) --requisites (-R) and exporting them via the --export option to a Nix archive for import into another store.

nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz

Nix default expression

Nix has a default expression created from the contents of ~/.nix-defexpr. The nix-env --install operation normally works by selecting an attribute in this expression (fast) or matching against the name attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The nix module sets up a default ~/.nix-defexpr/nixpkgs that symlinks to a slightly modified version of the official Nix nixpkgs. This can be entirely replace this or augment. For example, the following ~/.nix-defexpr/mypkgs expression packages up the above cabextract.nix example.

args@{ ... }: with import ./nixpkgs args;
rec {
  cabextract = stdenv.mkDerivation rec {
    name = "cabextract-1.6";

    src = fetchurl {
      url = "http://www.cabextract.org.uk/${name}.tar.gz";
      sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
    };

    meta = with stdenv.lib; {
      homepage = http://www.cabextract.org.uk/;
      description = "Free Software for extracting Microsoft cabinet files";
      platforms = platforms.all;
      license = licenses.gpl3;
      maintainers = with maintainers; [ pSub ];
    };
  };
}

This makes it possible to install packages from mypkgs as easily as those from the official nixpkgs.

nix-env --install --attr mypkgs.cabextract

References

o Nix Package Manager
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead)
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead)
https://www.youtube.com/watch?v=pQE9WTLAPHQ