Documentation - User contributions [en]

NIX

2018-01-13T20:16:47Z

Tyson: Add directions to entirely reset nix

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>
== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-01-12T20:09:11Z

Tyson: Info is accurate for graham too

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>
== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

File:Git.svg

2017-02-15T19:17:05Z

Tyson: Tyson uploaded a new version of File:Git.svg

Git command and flow diagram

File:Git.svg

2017-02-15T19:13:18Z

Tyson: Tyson uploaded a new version of File:Git.svg

Git command and flow diagram

NIX

2016-10-05T19:14:26Z

Tyson: Add link to the nixpkgs manual

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>
== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

File:Git.svg

2016-05-11T21:19:40Z

Tyson: Git command and flow diagram

Git command and flow diagram

NIX

2016-04-18T18:40:01Z

Tyson: Simplified intro recommended by Fraser

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>
== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages.

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>
== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2016-04-14T21:12:39Z

Tyson: Add section on one-off environments

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is an atomic pure non-mutable package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users
* Operations are atomic, they either succeed and create a new environment or fail leaving the previous environment in place. Previous environments can be switched back to at any point.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>
== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages.

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>
== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2016-04-14T20:42:57Z

Tyson: Merge in material prepared for TECC

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is an atomic pure non-mutable package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users
* Operations are atomic, they either succeed and create a new environment or fail leaving the previous environment in place. Previous environments can be switched back to at any point.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>
== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages.

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>
== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2015-12-14T21:29:36Z

Tyson: Correct formating on file names

NIX

2015-12-10T22:03:21Z

Tyson: Clean up in page links a bit

NIX

2015-12-02T21:53:45Z

Tyson: Typo

NIX

2015-12-02T21:52:15Z

Tyson: Flush out content from CO mini TECC talk

VALGRIND

2015-09-22T15:25:21Z

Tyson: Clarify what is the valgrind enabled openmpi debug library

{{Software
|package_name=VALGRIND
|package_description=Memory debugger
|package_idnumber=132
}}
'''Valgrind''' is a powerful tool for analyzing programs, memory debugging, memory leak detection and profiling. It is freely available under GNU license. Version 3.5.0 is available on most SHARCNET systems.

'''NOTE''' To avoid spurious warnings it is important to not use too new of a version of GCC or OpenMPI. We recommend

* gcc/4.8.2
* openmpi/gcc-debug/1.8.3 modules

= Overview =

Valgrind is a dynamic binary instrumentation framework that dynamically translates executables to add instrumentation and track all memory and register usages by a program. The advantages of this approach are that

* it can be directly run on any executable, and
* dynamic translation allows ultimate instrumentation

while the disadvantages are

* 5-100 x slow down depending on tool,
* 12-18 x increase in size of translated code, and
* corner cases may exist between translated code and original.

Several tools have been built upon this framework. These include

* ''memcheck'' - memory error detector
* ''cachegrind'' - cache and branch-prediction profiler
* ''callgrind'' - call-graph generating cache and branch prediction profiler
* ''helgrind'' - thread error detector
* ''DRD'' - thread error detector
* ''Massif'' - heap profiler
* ''DHAT'' - dynamic heap analysis tool
* ''SGCheck'' - experimental stack and global array overrun detector
* ''BBV'' - experimental basic block vector generation tool

You are welcome to use any or all of these, but we have only used ''memcheck'' and ''cachegrind'' and only support ''memcheck''.

= Usage =

The primary used tool is ''memcheck''. This is the default tool and the only one we discuss here. Documentation for other tools can be found on the [http://www.valgrind.org valgrind website]. The memcheck tool detects several common memory errors

* overrunning and underrunning heap blocks,
* overrunning top of stack,
* continuing to access released memory,
* using uninitialized values,
* incorrectly using memory copying routines,
* incorrectly paired allocation/release calls,
* relasing unallocated memory, and
* not releasing memory.

We recommend running all new code under valgrind on small test cases (small due to the aforementioned ~10x slowdown). This can save hours and hours of debugging. Running the program under valgrind can be as simple as compiling with debugging information (adding the <tt>-g</tt> flag) and running as <tt>valgrind <program> <arguements></tt>.

== Serial code ==

Consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

int main() {
double array[10];

// Execution depends on uninitialized value
if (array[4] < 0.0)
printf("the results is negative\n");
else if (array[4] == 0.0)
printf("the results is zero\n");
else if (array[4] > 0.0)
printf("the results is positive\n");
else
printf("the results are special\n");

return 0;
}
</source>

It has an uninitialized value bug. Running this under valgrind

<source lang="bash">
cc -Wall -g bug.c -o bug
valgrind ./bug
</source>

reports this

==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400511: main (bug.c:7)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x40052C: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400536: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400551: main (bug.c:11)

Note that valgrind only reports uninitialized values usage once they lead to non-determinism (i.e., when the program encounters a branch whose choice depends on the result of an uninitiated value). This means you the first report you get is frequently not in the calculation done on the uninitialized values but rather the convergence test or print statement at the end of calculations (print statements contains a bunch of branches in order to handling printing of each different digit).

=== Tracking down uninitialized values ===

More typically a program will be composed of multiple routines that all work on the data. To this end, consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

void initialize_sequence(double* array, const int array_length) {
int i;

for (i=1; i<array_length+1; ++i)
array[i] = i;
}

double sum(const double* array, const int array_length) {
double array_sum;
int i;

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}

int main() {
double array[10];
const int array_length = sizeof(array)/sizeof(array[0]);

double array_sum;

initialize_sequence(array,array_length);
array_sum = sum(array,array_length);

printf("the sum of 0..%d is %f\n", array_length-1, array_sum);

return 0;
}
</source>

It has both indexing and uninitialized value bugs. Despite this, directly running the code

<source lang="bash">
cc -Wall -g bug.c -o bug
./bug
</source>

will mostly likely produce the correct answer on most machines most of the time

the sum of 0..9 is 45.000000

Running under valgrind

<source lang="bash">
valgrind ./bug
</source>

reliably gives many warnings of the following form

==15930== Conditional jump or move depends on uninitialised value(s)
==15930== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==15930== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==15930== by 0x5318189: printf (in /lib64/libc-2.12.so)
==15930== by 0x400722: main (bug.c:29)

This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is random as it is taking or does not taking a branch based on a result computed from something that was not set by the programmer).

We now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program. This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.

Fortunately Valgrind provides a <tt>--track-origins=yes</tt> flag to ease our search by telling us which variables are the source of the problem. Re-running with this flag

<source lang="bash">
valgrind --track-origins=yes ./bug
</source>

gives warning messages augmented to include the source of the uninitialized value that went into the computation of ''array_sum''

==17589== Conditional jump or move depends on uninitialised value(s)
==17589== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==17589== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==17589== by 0x5318189: printf (in /lib64/libc-2.12.so)
==17589== by 0x400722: main (bug.c:29)
==17589== Uninitialised value was created by a stack allocation
==17589== at 0x400632: sum (bug.c:10)

Now we know the source of the uninitialized value in the ''array_sum'' computation was a local variable in the ''sum'' routine. Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero. If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.

Correcting the ''sum'' routine

<source lang="bash">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

array_sum = 0.0;
for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation. Now (using the <tt>--track-origins=yes</tt>) they are of the form

==18613== Conditional jump or move depends on uninitialised value(s)
==18613== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==18613== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==18613== by 0x5318189: printf (in /lib64/libc-2.12.so)
==18613== by 0x40072C: main (bug.c:30)
==18613== Uninitialised value was created by a stack allocation
==18613== at 0x400690: main (bug.c:21)

Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set. This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed <tt>sizeof(array)/sizeof(array[0])</tt> value.

Looking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''. Careful examination of this routine reveals the final error. The initialization loop was done using Fortran indices (1...''array_length'') instead of C (0...''array_length-1''). Correcting this

<source lang="bash">
void initialize_sequence(double* array, const int array_length) {
int i;

for (i=0; i<array_length; ++i)
array[i] = i;
}
</source>

finally produces code the runs under valgrind without any warnings.

'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.

=== Calling Valgrind from your program ===

In some cases it can be helpful to call Valgrind directly from your program in order to check the status of various bits of memory. This is easily done by including the ''valgrind/memcheck.h'' header file

<source lang="C">
#include <valgrind/memcheck.h>
</source>

and then adding calls to the appropriate [http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs client check routines]. For example, here is a modified ''sum'' routine for the above program that prints a warning if it is called with an ''array'' that is not fully initialized

<source lang="C">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

if (VALGRIND_CHECK_MEM_IS_DEFINED(array,sizeof(double)*array_length))
fprintf(stderr,"sum called with an array that is not fully defined...\n");

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

Running the ''bug.c'' code under Valgrind with this modification produces the following output

==29765== Uninitialised byte(s) found during client check request
==29765== at 0x400745: sum (bug.c:15)
==29765== by 0x400832: main (bug.c:31)
==29765== Address 0x7fefff8e8 is on thread 1's stack
==29765==
sum called with an array that is not fully defined...

== MPI code ==

Consider the following ''bug.c'' MPI code

<source lang="C">
#include <mpi.h>

int main(int argc,char *argv[]){
int rank, size;
int value;

MPI_Init(&argc, &argv);

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

if (rank == 0 && size > 1) {
MPI_Request request;

MPI_Irecv(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);
value = 0;
MPI_Wait (&request, MPI_STATUS_IGNORE);
}
else if (rank == 1) {
MPI_Request request;

MPI_Isend(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);
value = 1;
MPI_Wait(&request, MPI_STATUS_IGNORE);
}

MPI_Finalize();

return 0;
}
</source>

It has an uninitialized value bug and two race condition bugs around the use of value.

# The first race condition is that rank==0 sets value=0 while at the same time doing a non-blocking receive into value (bug:16).
# The uninitialized value problem is that rank==1 starts a send of value to rank==0 without ever setting value (bug:22).
# The second race condition is that rank==1 sets value=1 while at the same time doing a non-blocking send of value (bug:23).

=== Basic Functionality ===

If we just straight up compile and run this

<source lang="bash">
module unload intel openmpi
module load gcc/4.8.2 openmpi/gcc/1.8.3
mpicc -Wall -g bug.c -o bug
mpirun -np 2 valgrind ./bug
</source>

we presumably get a report about the uninitialized value, but it is buried in tens of thousands of other bogus error messages.

This solution is to link against the valgrind openmpi debug wrapper library too

<source lang="bash">
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
mpirun -np 2 valgrind ./bug
</source>

Now there are only a few bogus errors reported of the form

==12598== Syscall param write(buf) points to uninitialised byte(s)
==12598== at 0x53916FD: ??? (in /lib64/libpthread-2.12.so)
==12598== by 0x8AC8E40: send_bytes (oob_tcp_sendrecv.c:84)
==12598== by 0x8AC9471: mca_oob_tcp_send_handler (oob_tcp_sendrecv.c:205)
==12598== by 0x616FA23: opal_libevent2021_event_base_loop (in /opt/sharcnet/openmpi/1.8.1/gcc-debug/lib/libopen-pal.so.6.1.1)
==12598== by 0x5B7E78D: orte_progress_thread_engine (ess_base_std_app.c:456)
==12598== by 0x538A9D0: start_thread (in /lib64/libpthread-2.12.so)
==12598== by 0x8AB86FF: ???

We know this isn't an issue with ''bug.c'' as the backtrace shows it is occurring in a pure OpenMPI thread (the backtrace starts with ''start_thread'' and all subsequent routines are not from ''bug.c''). Skipping over these we now see Valgrind is picking up on sending the uninitialized value

==12599== Uninitialised byte(s) found during client check request
==12599== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==12599== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==12599== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==12599== by 0x400B02: main (bug.c:22)
==12599== Address 0x7fefff5c4 is on thread 1's stack

'''NOTE''' If we want to avoid recompiling our program, we can also preload the ''mpiwrap-amd64-linux'' library instead of linking against it

<source lang="bash">
mpicc -Wall -g bug.c -o bug
LD_LIBRARY_PATH="$(which mpirun | sed -e 's|/bin/.*$|/lib|')${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \
LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug
</source>

=== Breaking out output by rank ===

By default the Valgrind output from all the MPI ranks gets intermingled together on stderr. Although they can be distinguished for the most part as each line is prefixed with the process number, it is frequently nice to write them out to individual files. This can be done by using the ''mpirun'' <tt>--output-filename <filename></tt> option. It captures the output of each rank to ''<filename>'' suffixed with the ''MPI_COMM_WORLD'' rank

<source lang="bash">
mpirun --output-filename bug.log -np 2 valgrind ./bug
</source>

To redirect just the Valgrind output we need to use the Valgrind <tt>--log-file=<filename></tt> option with the special <tt>%q{<environment-variable>}</tt> syntax

<source lang="bash">
mpirun -np 2 valgrind --log-file=bug.log-%q{OMPI_COMM_WORLD_RANK} ./bug
</source>

Both of the above can also be done by wrapping the Valgrind call with Bash to perform redirection and/or environment variable expansion

<source lang="bash">
mpirun -np 2 /bin/bash -c 'exec valgrind ./bug > bug.log-$OMPI_COMM_WORLD_RANK 2>&1'
</source>

and

<source lang="bash">
mpirun -np 2 /bin/bash -c 'exec valgrind --log-file=bug.log-$OMPI_COMM_WORLD_RANK ./bug'
</source>

=== Advanced Functionality ===

Now, if on top of this, we also bring in the valgrind enabled openmpi debug library (i.e., the ''debug'' version of our ''openmpi'' module), things get really sweet

<source lang="bash">
module unload gcc openmpi
module load gcc/4.8.2 openmpi/gcc-debug/1.8.3
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
</source>

Now all the bugs in the code are detected and reported

==27774== Uninitialised byte(s) found during client check request
==27774== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27774== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27774== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27774== by 0x402713: main (bug.c:22)
==27774== Address 0x7fefff6f4 is on thread 1's stack

==27773== Invalid write of size 4
==27773== at 0x4026A0: main (bug.c:16)
==27773== Address 0x7fefff6f4 is on thread 1's stack

==27774== Invalid write of size 4
==27774== at 0x40271B: main (bug.c:23)
==27774== Address 0x7fefff6f4 is on thread 1's stack

=== Suppression File ===

There is also a valgrind suppression option <tt>--suppressions="$(which mpirun | sed -e 's|/bin/.*|/share/openmpi/openmpi-valgrind.supp|')"</tt>. We have not observed any cases where this makes a difference yet though.

=References=

o Package website 
http://www.valgrind.org

[[Category:Software packages]]

VALGRIND

2015-09-08T20:32:46Z

Tyson: Add details about splitting out output by ranks

{{Software
|package_name=VALGRIND
|package_description=Memory debugger
|package_idnumber=132
}}
'''Valgrind''' is a powerful tool for analyzing programs, memory debugging, memory leak detection and profiling. It is freely available under GNU license. Version 3.5.0 is available on most SHARCNET systems.

'''NOTE''' To avoid spurious warnings it is important to not use too new of a version of GCC or OpenMPI. We recommend

* gcc/4.8.2
* openmpi/gcc-debug/1.8.3 modules

= Overview =

Valgrind is a dynamic binary instrumentation framework that dynamically translates executables to add instrumentation and track all memory and register usages by a program. The advantages of this approach are that

* it can be directly run on any executable, and
* dynamic translation allows ultimate instrumentation

while the disadvantages are

* 5-100 x slow down depending on tool,
* 12-18 x increase in size of translated code, and
* corner cases may exist between translated code and original.

Several tools have been built upon this framework. These include

* ''memcheck'' - memory error detector
* ''cachegrind'' - cache and branch-prediction profiler
* ''callgrind'' - call-graph generating cache and branch prediction profiler
* ''helgrind'' - thread error detector
* ''DRD'' - thread error detector
* ''Massif'' - heap profiler
* ''DHAT'' - dynamic heap analysis tool
* ''SGCheck'' - experimental stack and global array overrun detector
* ''BBV'' - experimental basic block vector generation tool

You are welcome to use any or all of these, but we have only used ''memcheck'' and ''cachegrind'' and only support ''memcheck''.

= Usage =

The primary used tool is ''memcheck''. This is the default tool and the only one we discuss here. Documentation for other tools can be found on the [http://www.valgrind.org valgrind website]. The memcheck tool detects several common memory errors

* overrunning and underrunning heap blocks,
* overrunning top of stack,
* continuing to access released memory,
* using uninitialized values,
* incorrectly using memory copying routines,
* incorrectly paired allocation/release calls,
* relasing unallocated memory, and
* not releasing memory.

We recommend running all new code under valgrind on small test cases (small due to the aforementioned ~10x slowdown). This can save hours and hours of debugging. Running the program under valgrind can be as simple as compiling with debugging information (adding the <tt>-g</tt> flag) and running as <tt>valgrind <program> <arguements></tt>.

== Serial code ==

Consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

int main() {
double array[10];

// Execution depends on uninitialized value
if (array[4] < 0.0)
printf("the results is negative\n");
else if (array[4] == 0.0)
printf("the results is zero\n");
else if (array[4] > 0.0)
printf("the results is positive\n");
else
printf("the results are special\n");

return 0;
}
</source>

It has an uninitialized value bug. Running this under valgrind

<source lang="bash">
cc -Wall -g bug.c -o bug
valgrind ./bug
</source>

reports this

==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400511: main (bug.c:7)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x40052C: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400536: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400551: main (bug.c:11)

Note that valgrind only reports uninitialized values usage once they lead to non-determinism (i.e., when the program encounters a branch whose choice depends on the result of an uninitiated value). This means you the first report you get is frequently not in the calculation done on the uninitialized values but rather the convergence test or print statement at the end of calculations (print statements contains a bunch of branches in order to handling printing of each different digit).

=== Tracking down uninitialized values ===

More typically a program will be composed of multiple routines that all work on the data. To this end, consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

void initialize_sequence(double* array, const int array_length) {
int i;

for (i=1; i<array_length+1; ++i)
array[i] = i;
}

double sum(const double* array, const int array_length) {
double array_sum;
int i;

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}

int main() {
double array[10];
const int array_length = sizeof(array)/sizeof(array[0]);

double array_sum;

initialize_sequence(array,array_length);
array_sum = sum(array,array_length);

printf("the sum of 0..%d is %f\n", array_length-1, array_sum);

return 0;
}
</source>

It has both indexing and uninitialized value bugs. Despite this, directly running the code

<source lang="bash">
cc -Wall -g bug.c -o bug
./bug
</source>

will mostly likely produce the correct answer on most machines most of the time

the sum of 0..9 is 45.000000

Running under valgrind

<source lang="bash">
valgrind ./bug
</source>

reliably gives many warnings of the following form

==15930== Conditional jump or move depends on uninitialised value(s)
==15930== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==15930== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==15930== by 0x5318189: printf (in /lib64/libc-2.12.so)
==15930== by 0x400722: main (bug.c:29)

This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is random as it is taking or does not taking a branch based on a result computed from something that was not set by the programmer).

We now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program. This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.

Fortunately Valgrind provides a <tt>--track-origins=yes</tt> flag to ease our search by telling us which variables are the source of the problem. Re-running with this flag

<source lang="bash">
valgrind --track-origins=yes ./bug
</source>

gives warning messages augmented to include the source of the uninitialized value that went into the computation of ''array_sum''

==17589== Conditional jump or move depends on uninitialised value(s)
==17589== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==17589== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==17589== by 0x5318189: printf (in /lib64/libc-2.12.so)
==17589== by 0x400722: main (bug.c:29)
==17589== Uninitialised value was created by a stack allocation
==17589== at 0x400632: sum (bug.c:10)

Now we know the source of the uninitialized value in the ''array_sum'' computation was a local variable in the ''sum'' routine. Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero. If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.

Correcting the ''sum'' routine

<source lang="bash">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

array_sum = 0.0;
for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation. Now (using the <tt>--track-origins=yes</tt>) they are of the form

==18613== Conditional jump or move depends on uninitialised value(s)
==18613== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==18613== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==18613== by 0x5318189: printf (in /lib64/libc-2.12.so)
==18613== by 0x40072C: main (bug.c:30)
==18613== Uninitialised value was created by a stack allocation
==18613== at 0x400690: main (bug.c:21)

Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set. This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed <tt>sizeof(array)/sizeof(array[0])</tt> value.

Looking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''. Careful examination of this routine reveals the final error. The initialization loop was done using Fortran indices (1...''array_length'') instead of C (0...''array_length-1''). Correcting this

<source lang="bash">
void initialize_sequence(double* array, const int array_length) {
int i;

for (i=0; i<array_length; ++i)
array[i] = i;
}
</source>

finally produces code the runs under valgrind without any warnings.

'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.

=== Calling Valgrind from your program ===

In some cases it can be helpful to call Valgrind directly from your program in order to check the status of various bits of memory. This is easily done by including the ''valgrind/memcheck.h'' header file

<source lang="C">
#include <valgrind/memcheck.h>
</source>

and then adding calls to the appropriate [http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs client check routines]. For example, here is a modified ''sum'' routine for the above program that prints a warning if it is called with an ''array'' that is not fully initialized

<source lang="C">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

if (VALGRIND_CHECK_MEM_IS_DEFINED(array,sizeof(double)*array_length))
fprintf(stderr,"sum called with an array that is not fully defined...\n");

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

Running the ''bug.c'' code under Valgrind with this modification produces the following output

==29765== Uninitialised byte(s) found during client check request
==29765== at 0x400745: sum (bug.c:15)
==29765== by 0x400832: main (bug.c:31)
==29765== Address 0x7fefff8e8 is on thread 1's stack
==29765==
sum called with an array that is not fully defined...

== MPI code ==

Consider the following ''bug.c'' MPI code

<source lang="C">
#include <mpi.h>

int main(int argc,char *argv[]){
int rank, size;
int value;

MPI_Init(&argc, &argv);

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

if (rank == 0 && size > 1) {
MPI_Request request;

MPI_Irecv(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);
value = 0;
MPI_Wait (&request, MPI_STATUS_IGNORE);
}
else if (rank == 1) {
MPI_Request request;

MPI_Isend(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);
value = 1;
MPI_Wait(&request, MPI_STATUS_IGNORE);
}

MPI_Finalize();

return 0;
}
</source>

It has an uninitialized value bug and two race condition bugs around the use of value.

# The first race condition is that rank==0 sets value=0 while at the same time doing a non-blocking receive into value (bug:16).
# The uninitialized value problem is that rank==1 starts a send of value to rank==0 without ever setting value (bug:22).
# The second race condition is that rank==1 sets value=1 while at the same time doing a non-blocking send of value (bug:23).

=== Basic Functionality ===

If we just straight up compile and run this

<source lang="bash">
module unload intel openmpi
module load gcc/4.8.2 openmpi/gcc/1.8.3
mpicc -Wall -g bug.c -o bug
mpirun -np 2 valgrind ./bug
</source>

we presumably get a report about the uninitialized value, but it is buried in tens of thousands of other bogus error messages.

This solution is to link against the valgrind openmpi debug wrapper library too

<source lang="bash">
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
mpirun -np 2 valgrind ./bug
</source>

Now there are only a few bogus errors reported of the form

==12598== Syscall param write(buf) points to uninitialised byte(s)
==12598== at 0x53916FD: ??? (in /lib64/libpthread-2.12.so)
==12598== by 0x8AC8E40: send_bytes (oob_tcp_sendrecv.c:84)
==12598== by 0x8AC9471: mca_oob_tcp_send_handler (oob_tcp_sendrecv.c:205)
==12598== by 0x616FA23: opal_libevent2021_event_base_loop (in /opt/sharcnet/openmpi/1.8.1/gcc-debug/lib/libopen-pal.so.6.1.1)
==12598== by 0x5B7E78D: orte_progress_thread_engine (ess_base_std_app.c:456)
==12598== by 0x538A9D0: start_thread (in /lib64/libpthread-2.12.so)
==12598== by 0x8AB86FF: ???

We know this isn't an issue with ''bug.c'' as the backtrace shows it is occurring in a pure OpenMPI thread (the backtrace starts with ''start_thread'' and all subsequent routines are not from ''bug.c''). Skipping over these we now see Valgrind is picking up on sending the uninitialized value

==12599== Uninitialised byte(s) found during client check request
==12599== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==12599== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==12599== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==12599== by 0x400B02: main (bug.c:22)
==12599== Address 0x7fefff5c4 is on thread 1's stack

'''NOTE''' If we want to avoid recompiling our program, we can also preload the ''mpiwrap-amd64-linux'' library instead of linking against it

<source lang="bash">
mpicc -Wall -g bug.c -o bug
LD_LIBRARY_PATH="$(which mpirun | sed -e 's|/bin/.*$|/lib|')${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \
LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug
</source>

=== Breaking out output by rank ===

By default the Valgrind output from all the MPI ranks gets intermingled together on stderr. Although they can be distinguished for the most part as each line is prefixed with the process number, it is frequently nice to write them out to individual files. This can be done by using the ''mpirun'' <tt>--output-filename <filename></tt> option. It captures the output of each rank to ''<filename>'' suffixed with the ''MPI_COMM_WORLD'' rank

<source lang="bash">
mpirun --output-filename bug.log -np 2 valgrind ./bug
</source>

To redirect just the Valgrind output we need to use the Valgrind <tt>--log-file=<filename></tt> option with the special <tt>%q{<environment-variable>}</tt> syntax

<source lang="bash">
mpirun -np 2 valgrind --log-file=bug.log-%q{OMPI_COMM_WORLD_RANK} ./bug
</source>

Both of the above can also be done by wrapping the Valgrind call with Bash to perform redirection and/or environment variable expansion

<source lang="bash">
mpirun -np 2 /bin/bash -c 'exec valgrind ./bug > bug.log-$OMPI_COMM_WORLD_RANK 2>&1'
</source>

and

<source lang="bash">
mpirun -np 2 /bin/bash -c 'exec valgrind --log-file=bug.log-$OMPI_COMM_WORLD_RANK ./bug'
</source>

=== Advanced Functionality ===

Now, if on top of this, we also bring in the valgrind enabled openmpi debug library, things get really sweet

<source lang="bash">
module unload gcc openmpi
module load gcc/4.8.2 openmpi/gcc-debug/1.8.3
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
</source>

Now all the bugs in the code are detected and reported

==27774== Uninitialised byte(s) found during client check request
==27774== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27774== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27774== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27774== by 0x402713: main (bug.c:22)
==27774== Address 0x7fefff6f4 is on thread 1's stack

==27773== Invalid write of size 4
==27773== at 0x4026A0: main (bug.c:16)
==27773== Address 0x7fefff6f4 is on thread 1's stack

==27774== Invalid write of size 4
==27774== at 0x40271B: main (bug.c:23)
==27774== Address 0x7fefff6f4 is on thread 1's stack

=== Suppression File ===

There is also a valgrind suppression option <tt>--suppressions="$(which mpirun | sed -e 's|/bin/.*|/share/openmpi/openmpi-valgrind.supp|')"</tt>. We have not observed any cases where this makes a difference yet though.

=References=

o Package website 
http://www.valgrind.org

[[Category:Software packages]]

VALGRIND

2015-09-08T19:38:23Z

Tyson: Missed version number change and various formatting cleanups

{{Software
|package_name=VALGRIND
|package_description=Memory debugger
|package_idnumber=132
}}
'''Valgrind''' is a powerful tool for analyzing programs, memory debugging, memory leak detection and profiling. It is freely available under GNU license. Version 3.5.0 is available on most SHARCNET systems.

'''NOTE''' To avoid spurious warnings it is important to not use too new of a version of GCC or OpenMPI. We recommend

* gcc/4.8.2
* openmpi/gcc-debug/1.8.3 modules

= Overview =

Valgrind is a dynamic binary instrumentation framework that dynamically translates executables to add instrumentation and track all memory and register usages by a program. The advantages of this approach are that

* it can be directly run on any executable, and
* dynamic translation allows ultimate instrumentation

while the disadvantages are

* 5-100 x slow down depending on tool,
* 12-18 x increase in size of translated code, and
* corner cases may exist between translated code and original.

Several tools have been built upon this framework. These include

* ''memcheck'' - memory error detector
* ''cachegrind'' - cache and branch-prediction profiler
* ''callgrind'' - call-graph generating cache and branch prediction profiler
* ''helgrind'' - thread error detector
* ''DRD'' - thread error detector
* ''Massif'' - heap profiler
* ''DHAT'' - dynamic heap analysis tool
* ''SGCheck'' - experimental stack and global array overrun detector
* ''BBV'' - experimental basic block vector generation tool

You are welcome to use any or all of these, but we have only used ''memcheck'' and ''cachegrind'' and only support ''memcheck''.

= Usage =

The primary used tool is ''memcheck''. This is the default tool and the only one we discuss here. Documentation for other tools can be found on the [http://www.valgrind.org valgrind website]. The memcheck tool detects several common memory errors

* overrunning and underrunning heap blocks,
* overrunning top of stack,
* continuing to access released memory,
* using uninitialized values,
* incorrectly using memory copying routines,
* incorrectly paired allocation/release calls,
* relasing unallocated memory, and
* not releasing memory.

We recommend running all new code under valgrind on small test cases (small due to the aforementioned ~10x slowdown). This can save hours and hours of debugging. Running the program under valgrind can be as simple as compiling with debugging information (adding the <tt>-g</tt> flag) and running as <tt>valgrind <program> <arguements></tt>.

== Serial code ==

Consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

int main() {
double array[10];

// Execution depends on uninitialized value
if (array[4] < 0.0)
printf("the results is negative\n");
else if (array[4] == 0.0)
printf("the results is zero\n");
else if (array[4] > 0.0)
printf("the results is positive\n");
else
printf("the results are special\n");

return 0;
}
</source>

It has an uninitialized value bug. Running this under valgrind

<source lang="bash">
cc -Wall -g bug.c -o bug
valgrind ./bug
</source>

reports this

==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400511: main (bug.c:7)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x40052C: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400536: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400551: main (bug.c:11)

Note that valgrind only reports uninitialized values usage once they lead to non-determinism (i.e., when the program encounters a branch whose choice depends on the result of an uninitiated value). This means you the first report you get is frequently not in the calculation done on the uninitialized values but rather the convergence test or print statement at the end of calculations (print statements contains a bunch of branches in order to handling printing of each different digit).

=== Tracking down uninitialized values ===

More typically a program will be composed of multiple routines that all work on the data. To this end, consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

void initialize_sequence(double* array, const int array_length) {
int i;

for (i=1; i<array_length+1; ++i)
array[i] = i;
}

double sum(const double* array, const int array_length) {
double array_sum;
int i;

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}

int main() {
double array[10];
const int array_length = sizeof(array)/sizeof(array[0]);

double array_sum;

initialize_sequence(array,array_length);
array_sum = sum(array,array_length);

printf("the sum of 0..%d is %f\n", array_length-1, array_sum);

return 0;
}
</source>

It has both indexing and uninitialized value bugs. Despite this, directly running the code

<source lang="bash">
cc -Wall -g bug.c -o bug
./bug
</source>

will mostly likely produce the correct answer on most machines most of the time

the sum of 0..9 is 45.000000

Running under valgrind

<source lang="bash">
valgrind ./bug
</source>

reliably gives many warnings of the following form

==15930== Conditional jump or move depends on uninitialised value(s)
==15930== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==15930== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==15930== by 0x5318189: printf (in /lib64/libc-2.12.so)
==15930== by 0x400722: main (bug.c:29)

This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is random as it is taking or does not taking a branch based on a result computed from something that was not set by the programmer).

We now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program. This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.

Fortunately Valgrind provides a <tt>--track-origins=yes</tt> flag to ease our search by telling us which variables are the source of the problem. Re-running with this flag

<source lang="bash">
valgrind --track-origins=yes ./bug
</source>

gives warning messages augmented to include the source of the uninitialized value that went into the computation of ''array_sum''

==17589== Conditional jump or move depends on uninitialised value(s)
==17589== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==17589== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==17589== by 0x5318189: printf (in /lib64/libc-2.12.so)
==17589== by 0x400722: main (bug.c:29)
==17589== Uninitialised value was created by a stack allocation
==17589== at 0x400632: sum (bug.c:10)

Now we know the source of the uninitialized value in the ''array_sum'' computation was a local variable in the ''sum'' routine. Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero. If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.

Correcting the ''sum'' routine

<source lang="bash">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

array_sum = 0.0;
for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation. Now (using the <tt>--track-origins=yes</tt>) they are of the form

==18613== Conditional jump or move depends on uninitialised value(s)
==18613== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==18613== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==18613== by 0x5318189: printf (in /lib64/libc-2.12.so)
==18613== by 0x40072C: main (bug.c:30)
==18613== Uninitialised value was created by a stack allocation
==18613== at 0x400690: main (bug.c:21)

Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set. This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed <tt>sizeof(array)/sizeof(array[0])</tt> value.

Looking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''. Careful examination of this routine reveals the final error. The initialization loop was done using Fortran indices (1...''array_length'') instead of C (0...''array_length-1''). Correcting this

<source lang="bash">
void initialize_sequence(double* array, const int array_length) {
int i;

for (i=0; i<array_length; ++i)
array[i] = i;
}
</source>

finally produces code the runs under valgrind without any warnings.

'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.

=== Calling Valgrind from your program ===

In some cases it can be helpful to call Valgrind directly from your program in order to check the status of various bits of memory. This is easily done by including the ''valgrind/memcheck.h'' header file

<source lang="C">
#include <valgrind/memcheck.h>
</source>

and then adding calls to the appropriate [http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs client check routines]. For example, here is a modified ''sum'' routine for the above program that prints a warning if it is called with an ''array'' that is not fully initialized

<source lang="C">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

if (VALGRIND_CHECK_MEM_IS_DEFINED(array,sizeof(double)*array_length))
fprintf(stderr,"sum called with an array that is not fully defined...\n");

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

Running the ''bug.c'' code under Valgrind with this modification produces the following output

==29765== Uninitialised byte(s) found during client check request
==29765== at 0x400745: sum (bug.c:15)
==29765== by 0x400832: main (bug.c:31)
==29765== Address 0x7fefff8e8 is on thread 1's stack
==29765==
sum called with an array that is not fully defined...

== MPI code ==

Consider the following ''bug.c'' MPI code

<source lang="C">
#include <mpi.h>

int main(int argc,char *argv[]){
int rank, size;
int value;

MPI_Init(&argc, &argv);

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

if (rank == 0 && size > 1) {
MPI_Request request;

MPI_Irecv(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);
value = 0;
MPI_Wait (&request, MPI_STATUS_IGNORE);
}
else if (rank == 1) {
MPI_Request request;

MPI_Isend(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);
value = 1;
MPI_Wait(&request, MPI_STATUS_IGNORE);
}

MPI_Finalize();

return 0;
}
</source>

It has an uninitialized value bug and two race condition bugs around the use of value.

# The first race condition is that rank==0 sets value=0 while at the same time doing a non-blocking receive into value (bug:16).
# The uninitialized value problem is that rank==1 starts a send of value to rank==0 without ever setting value (bug:22).
# The second race condition is that rank==1 sets value=1 while at the same time doing a non-blocking send of value (bug:23).

=== Basic Functionality ===

If we just straight up compile and run this

<source lang="bash">
module unload intel openmpi
module load gcc/4.8.2 openmpi/gcc/1.8.3
mpicc -Wall -g bug.c -o bug
mpirun -np 2 valgrind ./bug
</source>

we presumably get a report about the uninitialized value, but it is buried in tens of thousands of other bogus error messages.

This solution is to link against the valgrind openmpi debug wrapper library too

<source lang="bash">
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
mpirun -np 2 valgrind ./bug
</source>

Now there are only a few bogus errors reported of the form

==12598== Syscall param write(buf) points to uninitialised byte(s)
==12598== at 0x53916FD: ??? (in /lib64/libpthread-2.12.so)
==12598== by 0x8AC8E40: send_bytes (oob_tcp_sendrecv.c:84)
==12598== by 0x8AC9471: mca_oob_tcp_send_handler (oob_tcp_sendrecv.c:205)
==12598== by 0x616FA23: opal_libevent2021_event_base_loop (in /opt/sharcnet/openmpi/1.8.1/gcc-debug/lib/libopen-pal.so.6.1.1)
==12598== by 0x5B7E78D: orte_progress_thread_engine (ess_base_std_app.c:456)
==12598== by 0x538A9D0: start_thread (in /lib64/libpthread-2.12.so)
==12598== by 0x8AB86FF: ???

We know this isn't an issue with ''bug.c'' as the backtrace shows it is occurring in a pure OpenMPI thread (the backtrace starts with ''start_thread'' and all subsequent routines are not from ''bug.c''). Skipping over these we now see Valgrind is picking up on sending the uninitialized value

==12599== Uninitialised byte(s) found during client check request
==12599== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==12599== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==12599== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==12599== by 0x400B02: main (bug.c:22)
==12599== Address 0x7fefff5c4 is on thread 1's stack

'''NOTE''' If we want to avoid recompiling our program, we can also preload the ''mpiwrap-amd64-linux'' library instead of linking against it

<source lang="bash">
mpicc -Wall -g bug.c -o bug
LD_LIBRARY_PATH="$(which mpirun | sed -e 's|/bin/.*$|/lib|')${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \
LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug
</source>

=== Advanced Functionality ===

Now, if on top of this, we also bring in the valgrind enabled openmpi debug library, things get really sweet

<source lang="bash">
module unload gcc openmpi
module load gcc/4.8.2 openmpi/gcc-debug/1.8.3
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
</source>

Now all the bugs in the code are detected and reported

==27774== Uninitialised byte(s) found during client check request
==27774== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27774== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27774== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27774== by 0x402713: main (bug.c:22)
==27774== Address 0x7fefff6f4 is on thread 1's stack

==27773== Invalid write of size 4
==27773== at 0x4026A0: main (bug.c:16)
==27773== Address 0x7fefff6f4 is on thread 1's stack

==27774== Invalid write of size 4
==27774== at 0x40271B: main (bug.c:23)
==27774== Address 0x7fefff6f4 is on thread 1's stack

=== Suppression File ===

There is also a valgrind suppression option <tt>--suppressions="$(which mpirun | sed -e 's|/bin/.*|/share/openmpi/openmpi-valgrind.supp|')"</tt>. We have not observed any cases where this makes a difference yet though.

=References=

o Package website 
http://www.valgrind.org

[[Category:Software packages]]

VALGRIND

2015-09-08T19:34:50Z

Tyson: Update MPI section for recommended GCC and OpenMPI versions

{{Software
|package_name=VALGRIND
|package_description=Memory debugger
|package_idnumber=132
}}
'''Valgrind''' is a powerful tool for analyzing programs, memory debugging, memory leak detection and profiling. It is freely available under GNU license. Version 3.5.0 is available on most SHARCNET systems.

'''NOTE''' To avoid spurious warnings it is important to not use too new of a version of GCC or OpenMPI. We recommend

* gcc/4.8.2
* openmpi/gcc-debug/1.8.1 modules

= Overview =

Valgrind is a dynamic binary instrumentation framework that dynamically translates executables to add instrumentation and track all memory and register usages by a program. The advantages of this approach are that

* it can be directly run on any executable, and
* dynamic translation allows ultimate instrumentation

while the disadvantages are

* 5-100 x slow down depending on tool,
* 12-18 x increase in size of translated code, and
* corner cases may exist between translated code and original.

Several tools have been built upon this framework. These include

* ''memcheck'' - memory error detector
* ''cachegrind'' - cache and branch-prediction profiler
* ''callgrind'' - call-graph generating cache and branch prediction profiler
* ''helgrind'' - thread error detector
* ''DRD'' - thread error detector
* ''Massif'' - heap profiler
* ''DHAT'' - dynamic heap analysis tool
* ''SGCheck'' - experimental stack and global array overrun detector
* ''BBV'' - experimental basic block vector generation tool

You are welcome to use any or all of these, but we have only used ''memcheck'' and ''cachegrind'' and only support ''memcheck''.

= Usage =

The primary used tool is ''memcheck''. This is the default tool and the only one we discuss here. Documentation for other tools can be found on the [http://www.valgrind.org valgrind website]. The memcheck tool detects several common memory errors

* overrunning and underrunning heap blocks,
* overrunning top of stack,
* continuing to access released memory,
* using uninitialized values,
* incorrectly using memory copying routines,
* incorrectly paired allocation/release calls,
* relasing unallocated memory, and
* not releasing memory.

We recommend running all new code under valgrind on small test cases (small due to the aforementioned ~10x slowdown). This can save hours and hours of debugging. Running the program under valgrind can be as simple as compiling with debugging information (adding the <tt>-g</tt> flag) and running as <tt>valgrind <program> <arguements></tt>.

== Serial code ==

Consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

int main() {
double array[10];

// Execution depends on uninitialized value
if (array[4] < 0.0)
printf("the results is negative\n");
else if (array[4] == 0.0)
printf("the results is zero\n");
else if (array[4] > 0.0)
printf("the results is positive\n");
else
printf("the results are special\n");

return 0;
}
</source>

It has an uninitialized value bug. Running this under valgrind

<source lang="bash">
cc -Wall -g bug.c -o bug
valgrind ./bug
</source>

reports this

==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400511: main (bug.c:7)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x40052C: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400536: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400551: main (bug.c:11)

Note that valgrind only reports uninitialized values usage once they lead to non-determinism (i.e., when the program encounters a branch whose choice depends on the result of an uninitiated value). This means you the first report you get is frequently not in the calculation done on the uninitialized values but rather the convergence test or print statement at the end of calculations (print statements contains a bunch of branches in order to handling printing of each different digit).

=== Tracking down uninitialized values ===

More typically a program will be composed of multiple routines that all work on the data. To this end, consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

void initialize_sequence(double* array, const int array_length) {
int i;

for (i=1; i<array_length+1; ++i)
array[i] = i;
}

double sum(const double* array, const int array_length) {
double array_sum;
int i;

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}

int main() {
double array[10];
const int array_length = sizeof(array)/sizeof(array[0]);

double array_sum;

initialize_sequence(array,array_length);
array_sum = sum(array,array_length);

printf("the sum of 0..%d is %f\n", array_length-1, array_sum);

return 0;
}
</source>

It has both indexing and uninitialized value bugs. Despite this, directly running the code

<source lang="bash">
cc -Wall -g bug.c -o bug
./bug
</source>

will mostly likely produce the correct answer on most machines most of the time

the sum of 0..9 is 45.000000

Running under valgrind

<source lang="bash">
valgrind ./bug
</source>

reliably gives many warnings of the following form

==15930== Conditional jump or move depends on uninitialised value(s)
==15930== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==15930== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==15930== by 0x5318189: printf (in /lib64/libc-2.12.so)
==15930== by 0x400722: main (bug.c:29)

This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is random as it is taking or does not taking a branch based on a result computed from something that was not set by the programmer).

We now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program. This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.

Fortunately Valgrind provides a ''--track-origins=yes'' flag to ease our search by telling us which variables are the source of the problem. Re-running with this flag

<source lang="bash">
valgrind --track-origins=yes ./bug
</source>

gives warning messages augmented to include the source of the uninitialized value that went into the computation of ''array_sum''

==17589== Conditional jump or move depends on uninitialised value(s)
==17589== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==17589== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==17589== by 0x5318189: printf (in /lib64/libc-2.12.so)
==17589== by 0x400722: main (bug.c:29)
==17589== Uninitialised value was created by a stack allocation
==17589== at 0x400632: sum (bug.c:10)

Now we know the source of the uninitialized value in the ''array_sum'' computation was a local variable in the ''sum'' routine. Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero. If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.

Correcting the ''sum'' routine

<source lang="bash">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

array_sum = 0.0;
for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation. Now (using the ''--track-origins=yes'') they are of the form

==18613== Conditional jump or move depends on uninitialised value(s)
==18613== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==18613== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==18613== by 0x5318189: printf (in /lib64/libc-2.12.so)
==18613== by 0x40072C: main (bug.c:30)
==18613== Uninitialised value was created by a stack allocation
==18613== at 0x400690: main (bug.c:21)

Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set. This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed ''sizeof(array)/sizeof(array[0])'' value.

Looking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''. Careful examination of this routine reveals the final error. The initialization loop was done using Fortran indices (1...''array_length'') instead of C (0...''array_length-1''). Correcting this

<source lang="bash">
void initialize_sequence(double* array, const int array_length) {
int i;

for (i=0; i<array_length; ++i)
array[i] = i;
}
</source>

finally produces code the runs under valgrind without any warnings.

'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.

=== Calling Valgrind from your program ===

In some cases it can be helpful to call Valgrind directly from your program in order to check the status of various bits of memory. This is easily done by including the ''valgrind/memcheck.h'' header file

<source lang="C">
#include <valgrind/memcheck.h>
</source>

and then adding calls to the appropriate [http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs client check routines]. For example, here is a modified ''sum'' routine for the above program that prints a warning if it is called with an ''array'' that is not fully initialized

<source lang="C">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

if (VALGRIND_CHECK_MEM_IS_DEFINED(array,sizeof(double)*array_length))
fprintf(stderr,"sum called with an array that is not fully defined...\n");

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

Running the ''bug.c'' code under Valgrind with this modification produces the following output

==29765== Uninitialised byte(s) found during client check request
==29765== at 0x400745: sum (bug.c:15)
==29765== by 0x400832: main (bug.c:31)
==29765== Address 0x7fefff8e8 is on thread 1's stack
==29765==
sum called with an array that is not fully defined...

== MPI code ==

Consider the following ''bug.c'' MPI code

<source lang="C">
#include <mpi.h>

int main(int argc,char *argv[]){
int rank, size;
int value;

MPI_Init(&argc, &argv);

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

if (rank == 0 && size > 1) {
MPI_Request request;

MPI_Irecv(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);
value = 0;
MPI_Wait (&request, MPI_STATUS_IGNORE);
}
else if (rank == 1) {
MPI_Request request;

MPI_Isend(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);
value = 1;
MPI_Wait(&request, MPI_STATUS_IGNORE);
}

MPI_Finalize();

return 0;
}
</source>

It has an uninitialized value bug and two race condition bugs around the use of value.

# The first race condition is that rank==0 sets value=0 while at the same time doing a non-blocking receive into value (bug:16).
# The uninitialized value problem is that rank==1 starts a send of value to rank==0 without ever setting value (bug:22).
# The second race condition is that rank==1 sets value=1 while at the same time doing a non-blocking send of value (bug:23).

=== Basic Functionality ===

If we just straight up compile and run this

<source lang="bash">
module unload intel openmpi
module load gcc/4.8.2 openmpi/gcc/1.8.3
mpicc -Wall -g bug.c -o bug
mpirun -np 2 valgrind ./bug
</source>

we presumably get a report about the uninitialized value, but it is buried in tens of thousands of other bogus error messages.

This solution is to link against the valgrind openmpi debug wrapper library too

<source lang="bash">
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
mpirun -np 2 valgrind ./bug
</source>

Now there are only a few bogus errors reported of the form

==12598== Syscall param write(buf) points to uninitialised byte(s)
==12598== at 0x53916FD: ??? (in /lib64/libpthread-2.12.so)
==12598== by 0x8AC8E40: send_bytes (oob_tcp_sendrecv.c:84)
==12598== by 0x8AC9471: mca_oob_tcp_send_handler (oob_tcp_sendrecv.c:205)
==12598== by 0x616FA23: opal_libevent2021_event_base_loop (in /opt/sharcnet/openmpi/1.8.1/gcc-debug/lib/libopen-pal.so.6.1.1)
==12598== by 0x5B7E78D: orte_progress_thread_engine (ess_base_std_app.c:456)
==12598== by 0x538A9D0: start_thread (in /lib64/libpthread-2.12.so)
==12598== by 0x8AB86FF: ???

We know this isn't an issue with ''bug.c'' as the backtrace shows it is occurring in a pure OpenMPI thread (the backtrace starts with ''start_thread'' and all subsequent routines are not from ''bug.c''). Skipping over these we now see Valgrind is picking up on sending the uninitialized value

==12599== Uninitialised byte(s) found during client check request
==12599== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==12599== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==12599== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==12599== by 0x400B02: main (bug.c:22)
==12599== Address 0x7fefff5c4 is on thread 1's stack

'''NOTE''' If we want to avoid recompiling our program, we can also preload the ''mpiwrap-amd64-linux'' library instead of linking against it

<source lang="bash">
mpicc -Wall -g bug.c -o bug
LD_LIBRARY_PATH="$(which mpirun | sed -e 's|/bin/.*$|/lib|')${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \
LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug
</source>

=== Advanced Functionality ===

Now, if on top of this, we also bring in the valgrind enabled openmpi debug library, things get really sweet

<source lang="bash">
module unload gcc openmpi
module load gcc/4.8.2 openmpi/gcc-debug/1.8.3
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
</source>

Now all the bugs in the code are detected and reported

==27774== Uninitialised byte(s) found during client check request
==27774== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27774== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27774== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27774== by 0x402713: main (bug.c:22)
==27774== Address 0x7fefff6f4 is on thread 1's stack

==27773== Invalid write of size 4
==27773== at 0x4026A0: main (bug.c:16)
==27773== Address 0x7fefff6f4 is on thread 1's stack

==27774== Invalid write of size 4
==27774== at 0x40271B: main (bug.c:23)
==27774== Address 0x7fefff6f4 is on thread 1's stack

=== Suppression File ===

There is also a valgrind suppression option <tt>--suppressions="$(which mpirun | sed -e 's|/bin/.*|/share/openmpi/openmpi-valgrind.supp|')"</tt>, however we have not observed any cases where this makes a difference yet.

=References=

o Package website 
http://www.valgrind.org

[[Category:Software packages]]

VALGRIND

2015-09-08T18:03:09Z

Tyson: Minor correction

{{Software
|package_name=VALGRIND
|package_description=Memory debugger
|package_idnumber=132
}}
'''Valgrind''' is a powerful tool for analyzing programs, memory debugging, memory leak detection and profiling. It is freely available under GNU license. Version 3.5.0 is available on most SHARCNET systems.

'''NOTE''' To avoid spurious warnings it is important to not use too new of a version of GCC or OpenMPI. We recommend

* gcc/4.8.2
* openmpi/gcc-debug/1.8.1 modules

= Overview =

Valgrind is a dynamic binary instrumentation framework that dynamically translates executables to add instrumentation and track all memory and register usages by a program. The advantages of this approach are that

* it can be directly run on any executable, and
* dynamic translation allows ultimate instrumentation

while the disadvantages are

* 5-100 x slow down depending on tool,
* 12-18 x increase in size of translated code, and
* corner cases may exist between translated code and original.

Several tools have been built upon this framework. These include

* ''memcheck'' - memory error detector
* ''cachegrind'' - cache and branch-prediction profiler
* ''callgrind'' - call-graph generating cache and branch prediction profiler
* ''helgrind'' - thread error detector
* ''DRD'' - thread error detector
* ''Massif'' - heap profiler
* ''DHAT'' - dynamic heap analysis tool
* ''SGCheck'' - experimental stack and global array overrun detector
* ''BBV'' - experimental basic block vector generation tool

You are welcome to use any or all of these, but we have only used ''memcheck'' and ''cachegrind'' and only support ''memcheck''.

= Usage =

The primary used tool is ''memcheck''. This is the default tool and the only one we discuss here. Documentation for other tools can be found on the [http://www.valgrind.org valgrind website]. The memcheck tool detects several common memory errors

* overrunning and underrunning heap blocks,
* overrunning top of stack,
* continuing to access released memory,
* using uninitialized values,
* incorrectly using memory copying routines,
* incorrectly paired allocation/release calls,
* relasing unallocated memory, and
* not releasing memory.

We recommend running all new code under valgrind on small test cases (small due to the aforementioned ~10x slowdown). This can save hours and hours of debugging. Running the program under valgrind can be as simple as compiling with debugging information (adding the <tt>-g</tt> flag) and running as <tt>valgrind <program> <arguements></tt>.

== Serial code ==

Consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

int main() {
double array[10];

// Execution depends on uninitialized value
if (array[4] < 0.0)
printf("the results is negative\n");
else if (array[4] == 0.0)
printf("the results is zero\n");
else if (array[4] > 0.0)
printf("the results is positive\n");
else
printf("the results are special\n");

return 0;
}
</source>

It has an uninitialized value bug. Running this under valgrind

<source lang="bash">
cc -Wall -g bug.c -o bug
valgrind ./bug
</source>

reports this

==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400511: main (bug.c:7)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x40052C: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400536: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400551: main (bug.c:11)

Note that valgrind only reports uninitialized values usage once they lead to non-determinism (i.e., when the program encounters a branch whose choice depends on the result of an uninitiated value). This means you the first report you get is frequently not in the calculation done on the uninitialized values but rather the convergence test or print statement at the end of calculations (print statements contains a bunch of branches in order to handling printing of each different digit).

=== Tracking down uninitialized values ===

More typically a program will be composed of multiple routines that all work on the data. To this end, consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

void initialize_sequence(double* array, const int array_length) {
int i;

for (i=1; i<array_length+1; ++i)
array[i] = i;
}

double sum(const double* array, const int array_length) {
double array_sum;
int i;

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}

int main() {
double array[10];
const int array_length = sizeof(array)/sizeof(array[0]);

double array_sum;

initialize_sequence(array,array_length);
array_sum = sum(array,array_length);

printf("the sum of 0..%d is %f\n", array_length-1, array_sum);

return 0;
}
</source>

It has both indexing and uninitialized value bugs. Despite this, directly running the code

<source lang="bash">
cc -Wall -g bug.c -o bug
./bug
</source>

will mostly likely produce the correct answer on most machines most of the time

the sum of 0..9 is 45.000000

Running under valgrind

<source lang="bash">
valgrind ./bug
</source>

reliably gives many warnings of the following form

==15930== Conditional jump or move depends on uninitialised value(s)
==15930== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==15930== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==15930== by 0x5318189: printf (in /lib64/libc-2.12.so)
==15930== by 0x400722: main (bug.c:29)

This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is random as it is taking or does not taking a branch based on a result computed from something that was not set by the programmer).

We now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program. This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.

Fortunately Valgrind provides a ''--track-origins=yes'' flag to ease our search by telling us which variables are the source of the problem. Re-running with this flag

<source lang="bash">
valgrind --track-origins=yes ./bug
</source>

gives warning messages augmented to include the source of the uninitialized value that went into the computation of ''array_sum''

==17589== Conditional jump or move depends on uninitialised value(s)
==17589== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==17589== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==17589== by 0x5318189: printf (in /lib64/libc-2.12.so)
==17589== by 0x400722: main (bug.c:29)
==17589== Uninitialised value was created by a stack allocation
==17589== at 0x400632: sum (bug.c:10)

Now we know the source of the uninitialized value in the ''array_sum'' computation was a local variable in the ''sum'' routine. Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero. If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.

Correcting the ''sum'' routine

<source lang="bash">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

array_sum = 0.0;
for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation. Now (using the ''--track-origins=yes'') they are of the form

==18613== Conditional jump or move depends on uninitialised value(s)
==18613== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==18613== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==18613== by 0x5318189: printf (in /lib64/libc-2.12.so)
==18613== by 0x40072C: main (bug.c:30)
==18613== Uninitialised value was created by a stack allocation
==18613== at 0x400690: main (bug.c:21)

Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set. This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed ''sizeof(array)/sizeof(array[0])'' value.

Looking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''. Careful examination of this routine reveals the final error. The initialization loop was done using Fortran indices (1...''array_length'') instead of C (0...''array_length-1''). Correcting this

<source lang="bash">
void initialize_sequence(double* array, const int array_length) {
int i;

for (i=0; i<array_length; ++i)
array[i] = i;
}
</source>

finally produces code the runs under valgrind without any warnings.

'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.

=== Calling Valgrind from your program ===

In some cases it can be helpful to call Valgrind directly from your program in order to check the status of various bits of memory. This is easily done by including the ''valgrind/memcheck.h'' header file

<source lang="C">
#include <valgrind/memcheck.h>
</source>

and then adding calls to the appropriate [http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs client check routines]. For example, here is a modified ''sum'' routine for the above program that prints a warning if it is called with an ''array'' that is not fully initialized

<source lang="C">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

if (VALGRIND_CHECK_MEM_IS_DEFINED(array,sizeof(double)*array_length))
fprintf(stderr,"sum called with an array that is not fully defined...\n");

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

Running the ''bug.c'' code under Valgrind with this modification produces the following output

==29765== Uninitialised byte(s) found during client check request
==29765== at 0x400745: sum (bug.c:15)
==29765== by 0x400832: main (bug.c:31)
==29765== Address 0x7fefff8e8 is on thread 1's stack
==29765==
sum called with an array that is not fully defined...

== MPI code ==

Consider the following ''bug.c'' MPI code

<source lang="C">
#include <mpi.h>

int main(int argc,char *argv[]){
int rank, size;
int value;

MPI_Init(&argc, &argv);

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

if (rank == 0 && size > 1) {
MPI_Request request;

MPI_Irecv(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);
value = 0;
MPI_Wait (&request, MPI_STATUS_IGNORE);
}
else if (rank == 1) {
MPI_Request request;

MPI_Isend(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);
value = 1;
MPI_Wait(&request, MPI_STATUS_IGNORE);
}

MPI_Finalize();

return 0;
}
</source>

It has an uninitialized value bug and two race condition bugs around the use of value.

# The first race condition is that rank==0 sets value=0 while at the same time doing a non-blocking receive into value (bug:16).
# The uninitialized value problem is that rank==1 starts a send of value to rank==0 without ever setting value (bug:22).
# The second race condition is that rank==1 sets value=1 while at the same time doing a non-blocking send of value (bug:23).

=== Basic Functionality ===

If you run this using the non-debug/valgrind openmpi valgrind

<source lang="bash">
module unload intel openmpi
module load intel/12.1.3 openmpi/intel/1.6.2
mpicc -Wall -g bug.c -o bug
mpirun -np 2 valgrind ./bug
</source>

you are presumably getting a report about the uninitialized value, but it is burried in 55517 other bogus error messages.

This solution to this is to link against the valgrind openmpi debug wrapper library too.

<source lang="bash">
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
mpirun -np 2 valgrind ./bug
</source>

This now only reports one bogus error (a free on exit) and picks up the sending of the uninitialized value

==27638== Uninitialised byte(s) found during client check request
==27638== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27638== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27638== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27638== by 0x402713: main (bug.c:22)
==27638== Address 0x7fefff734 is on thread 1's stack

=== Advanced Functionality ===

Now, if on top of this, you also bring in the valgrind enabled openmpi debug library, things get really sweet

<source lang="bash">
module unload intel openmpi
module load intel/12.1.3 openmpi/intel-debug/1.6.2
mpicc -Wall -g bug.c -o bug
LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug
</source>

You still only get one bogus error (a free on exit) and all the bugs in the code are detected and reported

==27774== Uninitialised byte(s) found during client check request
==27774== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27774== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27774== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27774== by 0x402713: main (bug.c:22)
==27774== Address 0x7fefff6f4 is on thread 1's stack

==27773== Invalid write of size 4
==27773== at 0x4026A0: main (bug.c:16)
==27773== Address 0x7fefff6f4 is on thread 1's stack

==27774== Invalid write of size 4
==27774== at 0x40271B: main (bug.c:23)
==27774== Address 0x7fefff6f4 is on thread 1's stack

=== Suppression File ===

There is also a valgrind suppression option <tt>--suppressions=/opt/sharcnet/openmpi/1.6.2/intel-debug/share/openmpi/openmpi-valgrind.supp</tt>, however we have not observed any cases where this makes a difference yet.

=References=

o Package website 
http://www.valgrind.org

[[Category:Software packages]]

VALGRIND

2015-09-08T18:02:44Z

Tyson: Add section about using client API

{{Software
|package_name=VALGRIND
|package_description=Memory debugger
|package_idnumber=132
}}
'''Valgrind''' is a powerful tool for analyzing programs, memory debugging, memory leak detection and profiling. It is freely available under GNU license. Version 3.5.0 is available on most SHARCNET systems.

'''NOTE''' To avoid spurious warnings it is important to not use too new of a version of GCC or OpenMPI. We recommend

* gcc/4.8.2
* openmpi/gcc-debug/1.8.1 modules

= Overview =

Valgrind is a dynamic binary instrumentation framework that dynamically translates executables to add instrumentation and track all memory and register usages by a program. The advantages of this approach are that

* it can be directly run on any executable, and
* dynamic translation allows ultimate instrumentation

while the disadvantages are

* 5-100 x slow down depending on tool,
* 12-18 x increase in size of translated code, and
* corner cases may exist between translated code and original.

Several tools have been built upon this framework. These include

* ''memcheck'' - memory error detector
* ''cachegrind'' - cache and branch-prediction profiler
* ''callgrind'' - call-graph generating cache and branch prediction profiler
* ''helgrind'' - thread error detector
* ''DRD'' - thread error detector
* ''Massif'' - heap profiler
* ''DHAT'' - dynamic heap analysis tool
* ''SGCheck'' - experimental stack and global array overrun detector
* ''BBV'' - experimental basic block vector generation tool

You are welcome to use any or all of these, but we have only used ''memcheck'' and ''cachegrind'' and only support ''memcheck''.

= Usage =

The primary used tool is ''memcheck''. This is the default tool and the only one we discuss here. Documentation for other tools can be found on the [http://www.valgrind.org valgrind website]. The memcheck tool detects several common memory errors

* overrunning and underrunning heap blocks,
* overrunning top of stack,
* continuing to access released memory,
* using uninitialized values,
* incorrectly using memory copying routines,
* incorrectly paired allocation/release calls,
* relasing unallocated memory, and
* not releasing memory.

We recommend running all new code under valgrind on small test cases (small due to the aforementioned ~10x slowdown). This can save hours and hours of debugging. Running the program under valgrind can be as simple as compiling with debugging information (adding the <tt>-g</tt> flag) and running as <tt>valgrind <program> <arguements></tt>.

== Serial code ==

Consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

int main() {
double array[10];

// Execution depends on uninitialized value
if (array[4] < 0.0)
printf("the results is negative\n");
else if (array[4] == 0.0)
printf("the results is zero\n");
else if (array[4] > 0.0)
printf("the results is positive\n");
else
printf("the results are special\n");

return 0;
}
</source>

It has an uninitialized value bug. Running this under valgrind

<source lang="bash">
cc -Wall -g bug.c -o bug
valgrind ./bug
</source>

reports this

==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400511: main (bug.c:7)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x40052C: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400536: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400551: main (bug.c:11)

Note that valgrind only reports uninitialized values usage once they lead to non-determinism (i.e., when the program encounters a branch whose choice depends on the result of an uninitiated value). This means you the first report you get is frequently not in the calculation done on the uninitialized values but rather the convergence test or print statement at the end of calculations (print statements contains a bunch of branches in order to handling printing of each different digit).

=== Tracking down uninitialized values ===

More typically a program will be composed of multiple routines that all work on the data. To this end, consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

void initialize_sequence(double* array, const int array_length) {
int i;

for (i=1; i<array_length+1; ++i)
array[i] = i;
}

double sum(const double* array, const int array_length) {
double array_sum;
int i;

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}

int main() {
double array[10];
const int array_length = sizeof(array)/sizeof(array[0]);

double array_sum;

initialize_sequence(array,array_length);
array_sum = sum(array,array_length);

printf("the sum of 0..%d is %f\n", array_length-1, array_sum);

return 0;
}
</source>

It has both indexing and uninitialized value bugs. Despite this, directly running the code

<source lang="bash">
cc -Wall -g bug.c -o bug
./bug
</source>

will mostly likely produce the correct answer on most machines most of the time

the sum of 0..9 is 45.000000

Running under valgrind

<source lang="bash">
valgrind ./bug
</source>

reliably gives many warnings of the following form

==15930== Conditional jump or move depends on uninitialised value(s)
==15930== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==15930== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==15930== by 0x5318189: printf (in /lib64/libc-2.12.so)
==15930== by 0x400722: main (bug.c:29)

This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is random as it is taking or does not taking a branch based on a result computed from something that was not set by the programmer).

We now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program. This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.

Fortunately Valgrind provides a ''--track-origins=yes'' flag to ease our search by telling us which variables are the source of the problem. Re-running with this flag

<source lang="bash">
valgrind --track-origins=yes ./bug
</source>

gives warning messages augmented to include the source of the uninitialized value that went into the computation of ''array_sum''

==17589== Conditional jump or move depends on uninitialised value(s)
==17589== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==17589== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==17589== by 0x5318189: printf (in /lib64/libc-2.12.so)
==17589== by 0x400722: main (bug.c:29)
==17589== Uninitialised value was created by a stack allocation
==17589== at 0x400632: sum (bug.c:10)

Now we know the source of the uninitialized value in the ''array_sum'' computation was a local variable in the ''sum'' routine. Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero. If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.

Correcting the ''sum'' routine

<source lang="bash">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

array_sum = 0.0;
for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation. Now (using the ''--track-origins=yes'') they are of the form

==18613== Conditional jump or move depends on uninitialised value(s)
==18613== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==18613== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==18613== by 0x5318189: printf (in /lib64/libc-2.12.so)
==18613== by 0x40072C: main (bug.c:30)
==18613== Uninitialised value was created by a stack allocation
==18613== at 0x400690: main (bug.c:21)

Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set. This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed ''sizeof(array)/sizeof(array[0])'' value.

Looking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''. Careful examination of this routine reveals the final error. The initialization loop was done using Fortran indices (1...''array_length'') instead of C (0...''array_length-1''). Correcting this

<source lang="bash">
void initialize_sequence(double* array, const int array_length) {
int i;

for (i=0; i<array_length; ++i)
array[i] = i;
}
</source>

finally produces code the runs under valgrind without any warnings.

'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.

=== Call Valgrind from your program ===

In some cases it can be helpful to call Valgrind directly from your program in order to check the status of various bits of memory. This is easily done by including the ''valgrind/memcheck.h'' header file

<source lang="C">
#include <valgrind/memcheck.h>
</source>

and then adding calls to the appropriate [http://valgrind.org/docs/manual/mc-manual.html#mc-manual.clientreqs client check routines]. For example, here is a modified ''sum'' routine for the above program that prints a warning if it is called with an ''array'' that is not fully initialized

<source lang="C">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

if (VALGRIND_CHECK_MEM_IS_DEFINED(array,sizeof(double)*array_length))
fprintf(stderr,"sum called with an array that is not fully defined...\n");

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

Running the ''bug.c'' code under Valgrind with this modification produces the following output

==29765== Uninitialised byte(s) found during client check request
==29765== at 0x400745: sum (bug.c:15)
==29765== by 0x400832: main (bug.c:31)
==29765== Address 0x7fefff8e8 is on thread 1's stack
==29765==
sum called with an array that is not fully defined...

== MPI code ==

Consider the following ''bug.c'' MPI code

<source lang="C">
#include <mpi.h>

int main(int argc,char *argv[]){
int rank, size;
int value;

MPI_Init(&argc, &argv);

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

if (rank == 0 && size > 1) {
MPI_Request request;

MPI_Irecv(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);
value = 0;
MPI_Wait (&request, MPI_STATUS_IGNORE);
}
else if (rank == 1) {
MPI_Request request;

MPI_Isend(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);
value = 1;
MPI_Wait(&request, MPI_STATUS_IGNORE);
}

MPI_Finalize();

return 0;
}
</source>

It has an uninitialized value bug and two race condition bugs around the use of value.

# The first race condition is that rank==0 sets value=0 while at the same time doing a non-blocking receive into value (bug:16).
# The uninitialized value problem is that rank==1 starts a send of value to rank==0 without ever setting value (bug:22).
# The second race condition is that rank==1 sets value=1 while at the same time doing a non-blocking send of value (bug:23).

=== Basic Functionality ===

If you run this using the non-debug/valgrind openmpi valgrind

<source lang="bash">
module unload intel openmpi
module load intel/12.1.3 openmpi/intel/1.6.2
mpicc -Wall -g bug.c -o bug
mpirun -np 2 valgrind ./bug
</source>

you are presumably getting a report about the uninitialized value, but it is burried in 55517 other bogus error messages.

This solution to this is to link against the valgrind openmpi debug wrapper library too.

<source lang="bash">
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
mpirun -np 2 valgrind ./bug
</source>

This now only reports one bogus error (a free on exit) and picks up the sending of the uninitialized value

==27638== Uninitialised byte(s) found during client check request
==27638== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27638== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27638== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27638== by 0x402713: main (bug.c:22)
==27638== Address 0x7fefff734 is on thread 1's stack

=== Advanced Functionality ===

Now, if on top of this, you also bring in the valgrind enabled openmpi debug library, things get really sweet

<source lang="bash">
module unload intel openmpi
module load intel/12.1.3 openmpi/intel-debug/1.6.2
mpicc -Wall -g bug.c -o bug
LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug
</source>

You still only get one bogus error (a free on exit) and all the bugs in the code are detected and reported

==27774== Uninitialised byte(s) found during client check request
==27774== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27774== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27774== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27774== by 0x402713: main (bug.c:22)
==27774== Address 0x7fefff6f4 is on thread 1's stack

==27773== Invalid write of size 4
==27773== at 0x4026A0: main (bug.c:16)
==27773== Address 0x7fefff6f4 is on thread 1's stack

==27774== Invalid write of size 4
==27774== at 0x40271B: main (bug.c:23)
==27774== Address 0x7fefff6f4 is on thread 1's stack

=== Suppression File ===

There is also a valgrind suppression option <tt>--suppressions=/opt/sharcnet/openmpi/1.6.2/intel-debug/share/openmpi/openmpi-valgrind.supp</tt>, however we have not observed any cases where this makes a difference yet.

=References=

o Package website 
http://www.valgrind.org

[[Category:Software packages]]

VALGRIND

2015-09-08T16:20:54Z

Tyson: Example of tracking down uninitialized value warnings

{{Software
|package_name=VALGRIND
|package_description=Memory debugger
|package_idnumber=132
}}
'''Valgrind''' is a powerful tool for analyzing programs, memory debugging, memory leak detection and profiling. It is freely available under GNU license. Version 3.5.0 is available on most SHARCNET systems.

'''NOTE''' To avoid spurious warnings it is important to not use too new of a version of GCC or OpenMPI. We recommend

* gcc/4.8.2
* openmpi/gcc-debug/1.8.1 modules

= Overview =

Valgrind is a dynamic binary instrumentation framework that dynamically translates executables to add instrumentation and track all memory and register usages by a program. The advantages of this approach are that

* it can be directly run on any executable, and
* dynamic translation allows ultimate instrumentation

while the disadvantages are

* 5-100 x slow down depending on tool,
* 12-18 x increase in size of translated code, and
* corner cases may exist between translated code and original.

Several tools have been built upon this framework. These include

* ''memcheck'' - memory error detector
* ''cachegrind'' - cache and branch-prediction profiler
* ''callgrind'' - call-graph generating cache and branch prediction profiler
* ''helgrind'' - thread error detector
* ''DRD'' - thread error detector
* ''Massif'' - heap profiler
* ''DHAT'' - dynamic heap analysis tool
* ''SGCheck'' - experimental stack and global array overrun detector
* ''BBV'' - experimental basic block vector generation tool

You are welcome to use any or all of these, but we have only used ''memcheck'' and ''cachegrind'' and only support ''memcheck''.

= Usage =

The primary used tool is ''memcheck''. This is the default tool and the only one we discuss here. Documentation for other tools can be found on the [http://www.valgrind.org valgrind website]. The memcheck tool detects several common memory errors

* overrunning and underrunning heap blocks,
* overrunning top of stack,
* continuing to access released memory,
* using uninitialized values,
* incorrectly using memory copying routines,
* incorrectly paired allocation/release calls,
* relasing unallocated memory, and
* not releasing memory.

We recommend running all new code under valgrind on small test cases (small due to the aforementioned ~10x slowdown). This can save hours and hours of debugging. Running the program under valgrind can be as simple as compiling with debugging information (adding the <tt>-g</tt> flag) and running as <tt>valgrind <program> <arguements></tt>.

== Serial code ==

Consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

int main() {
double array[10];

// Execution depends on uninitialized value
if (array[4] < 0.0)
printf("the results is negative\n");
else if (array[4] == 0.0)
printf("the results is zero\n");
else if (array[4] > 0.0)
printf("the results is positive\n");
else
printf("the results are special\n");

return 0;
}
</source>

It has an uninitialized value bug. Running this under valgrind

<source lang="bash">
cc -Wall -g bug.c -o bug
valgrind ./bug
</source>

reports this

==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400511: main (bug.c:7)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x40052C: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400536: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400551: main (bug.c:11)

Note that valgrind only reports uninitialized values usage once they lead to non-determinism (i.e., when the program encounters a branch whose choice depends on the result of an uninitiated value). This means you the first report you get is frequently not in the calculation done on the uninitialized values but rather the convergence test or print statement at the end of calculations (print statements contains a bunch of branches in order to handling printing of each different digit).

=== Tracking down uninitialized values ===

More typically a program will be composed of multiple routines that all work on the data. To this end, consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

void initialize_sequence(double* array, const int array_length) {
int i;

for (i=1; i<array_length+1; ++i)
array[i] = i;
}

double sum(const double* array, const int array_length) {
double array_sum;
int i;

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}

int main() {
double array[10];
const int array_length = sizeof(array)/sizeof(array[0]);

double array_sum;

initialize_sequence(array,array_length);
array_sum = sum(array,array_length);

printf("the sum of 0..%d is %f\n", array_length-1, array_sum);

return 0;
}
</source>

It has both indexing and uninitialized value bugs. Despite this, directly running the code

<source lang="bash">
cc -Wall -g bug.c -o bug
./bug
</source>

will mostly likely produce the correct answer on most machines most of the time

the sum of 0..9 is 45.000000

Running under valgrind

<source lang="bash">
valgrind ./bug
</source>

reliably gives many warnings of the following form

==15930== Conditional jump or move depends on uninitialised value(s)
==15930== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==15930== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==15930== by 0x5318189: printf (in /lib64/libc-2.12.so)
==15930== by 0x400722: main (bug.c:29)

This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is random as it is taking or does not taking a branch based on a result computed from something that was not set by the programmer).

We now know the problem is that something unset went into computing the value of ''array_sum'', so we should trace the ''array_sum'' calculation backwards through the program. This quickly gets difficult as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables, and so on.

Fortunately Valgrind provides a ''--track-origins=yes'' flag to ease our search by telling us which variables are the source of the problem. Re-running with this flag

<source lang="bash">
valgrind --track-origins=yes ./bug
</source>

gives warning messages augmented to include the source of the uninitialized value that went into the computation of ''array_sum''

==17589== Conditional jump or move depends on uninitialised value(s)
==17589== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==17589== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==17589== by 0x5318189: printf (in /lib64/libc-2.12.so)
==17589== by 0x400722: main (bug.c:29)
==17589== Uninitialised value was created by a stack allocation
==17589== at 0x400632: sum (bug.c:10)

Now we know the source of the uninitialized value in the ''array_sum'' computation was a local variable in the ''sum'' routine. Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero. If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.

Correcting the ''sum'' routine

<source lang="bash">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

array_sum = 0.0;
for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation. Now (using the ''--track-origins=yes'') they are of the form

==18613== Conditional jump or move depends on uninitialised value(s)
==18613== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==18613== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==18613== by 0x5318189: printf (in /lib64/libc-2.12.so)
==18613== by 0x40072C: main (bug.c:30)
==18613== Uninitialised value was created by a stack allocation
==18613== at 0x400690: main (bug.c:21)

Valgrind is now telling us that a local variable inside ''main'' went into the computation of ''array_sum'' despite never being set. This must be ''array'' itself as ''array_length'' is clearly set to the compiler computed ''sizeof(array)/sizeof(array[0])'' value.

Looking at ''main'' it is clear ''array'' should have been fully initialized by ''initialize_sequence''. Careful examination of this routine reveals the final error. The initialization loop was done using Fortran indices (1...''array_length'') instead of C (0...''array_length-1''). Correcting this

<source lang="bash">
void initialize_sequence(double* array, const int array_length) {
int i;

for (i=0; i<array_length; ++i)
array[i] = i;
}
</source>

finally produces code the runs under valgrind without any warnings.

'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.

== MPI code ==

Consider the following ''bug.c'' MPI code

<source lang="C">
#include <mpi.h>

int main(int argc,char *argv[]){
int rank, size;
int value;

MPI_Init(&argc, &argv);

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

if (rank == 0 && size > 1) {
MPI_Request request;

MPI_Irecv(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);
value = 0;
MPI_Wait (&request, MPI_STATUS_IGNORE);
}
else if (rank == 1) {
MPI_Request request;

MPI_Isend(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);
value = 1;
MPI_Wait(&request, MPI_STATUS_IGNORE);
}

MPI_Finalize();

return 0;
}
</source>

It has an uninitialized value bug and two race condition bugs around the use of value.

# The first race condition is that rank==0 sets value=0 while at the same time doing a non-blocking receive into value (bug:16).
# The uninitialized value problem is that rank==1 starts a send of value to rank==0 without ever setting value (bug:22).
# The second race condition is that rank==1 sets value=1 while at the same time doing a non-blocking send of value (bug:23).

=== Basic Functionality ===

If you run this using the non-debug/valgrind openmpi valgrind

<source lang="bash">
module unload intel openmpi
module load intel/12.1.3 openmpi/intel/1.6.2
mpicc -Wall -g bug.c -o bug
mpirun -np 2 valgrind ./bug
</source>

you are presumably getting a report about the uninitialized value, but it is burried in 55517 other bogus error messages.

This solution to this is to link against the valgrind openmpi debug wrapper library too.

<source lang="bash">
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
mpirun -np 2 valgrind ./bug
</source>

This now only reports one bogus error (a free on exit) and picks up the sending of the uninitialized value

==27638== Uninitialised byte(s) found during client check request
==27638== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27638== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27638== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27638== by 0x402713: main (bug.c:22)
==27638== Address 0x7fefff734 is on thread 1's stack

=== Advanced Functionality ===

Now, if on top of this, you also bring in the valgrind enabled openmpi debug library, things get really sweet

<source lang="bash">
module unload intel openmpi
module load intel/12.1.3 openmpi/intel-debug/1.6.2
mpicc -Wall -g bug.c -o bug
LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug
</source>

You still only get one bogus error (a free on exit) and all the bugs in the code are detected and reported

==27774== Uninitialised byte(s) found during client check request
==27774== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27774== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27774== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27774== by 0x402713: main (bug.c:22)
==27774== Address 0x7fefff6f4 is on thread 1's stack

==27773== Invalid write of size 4
==27773== at 0x4026A0: main (bug.c:16)
==27773== Address 0x7fefff6f4 is on thread 1's stack

==27774== Invalid write of size 4
==27774== at 0x40271B: main (bug.c:23)
==27774== Address 0x7fefff6f4 is on thread 1's stack

=== Suppression File ===

There is also a valgrind suppression option <tt>--suppressions=/opt/sharcnet/openmpi/1.6.2/intel-debug/share/openmpi/openmpi-valgrind.supp</tt>, however we have not observed any cases where this makes a difference yet.

=References=

o Package website 
http://www.valgrind.org

[[Category:Software packages]]

VALGRIND

2015-09-08T16:11:31Z

Tyson: Add directions on tracking down uninitialized values

{{Software
|package_name=VALGRIND
|package_description=Memory debugger
|package_idnumber=132
}}
'''Valgrind''' is a powerful tool for analyzing programs, memory debugging, memory leak detection and profiling. It is freely available under GNU license. Version 3.5.0 is available on most SHARCNET systems.

'''NOTE''' To avoid spurious warnings it is important to not use too new of a version of GCC or OpenMPI. We recommend

* gcc/4.8.2
* openmpi/gcc-debug/1.8.1 modules

= Overview =

Valgrind is a dynamic binary instrumentation framework that dynamically translates executables to add instrumentation and track all memory and register usages by a program. The advantages of this approach are that

* it can be directly run on any executable, and
* dynamic translation allows ultimate instrumentation

while the disadvantages are

* 5-100 x slow down depending on tool,
* 12-18 x increase in size of translated code, and
* corner cases may exist between translated code and original.

Several tools have been built upon this framework. These include

* ''memcheck'' - memory error detector
* ''cachegrind'' - cache and branch-prediction profiler
* ''callgrind'' - call-graph generating cache and branch prediction profiler
* ''helgrind'' - thread error detector
* ''DRD'' - thread error detector
* ''Massif'' - heap profiler
* ''DHAT'' - dynamic heap analysis tool
* ''SGCheck'' - experimental stack and global array overrun detector
* ''BBV'' - experimental basic block vector generation tool

You are welcome to use any or all of these, but we have only used ''memcheck'' and ''cachegrind'' and only support ''memcheck''.

= Usage =

The primary used tool is ''memcheck''. This is the default tool and the only one we discuss here. Documentation for other tools can be found on the [http://www.valgrind.org valgrind website]. The memcheck tool detects several common memory errors

* overrunning and underrunning heap blocks,
* overrunning top of stack,
* continuing to access released memory,
* using uninitialized values,
* incorrectly using memory copying routines,
* incorrectly paired allocation/release calls,
* relasing unallocated memory, and
* not releasing memory.

We recommend running all new code under valgrind on small test cases (small due to the aforementioned ~10x slowdown). This can save hours and hours of debugging. Running the program under valgrind can be as simple as compiling with debugging information (adding the <tt>-g</tt> flag) and running as <tt>valgrind <program> <arguements></tt>.

== Serial code ==

Consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

int main() {
double array[10];

// Execution depends on uninitialized value
if (array[4] < 0.0)
printf("the results is negative\n");
else if (array[4] == 0.0)
printf("the results is zero\n");
else if (array[4] > 0.0)
printf("the results is positive\n");
else
printf("the results are special\n");

return 0;
}
</source>

It has an uninitialized value bug. Running this under valgrind

<source lang="bash">
cc -Wall -g bug.c -o bug
valgrind ./bug
</source>

reports this

==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400511: main (bug.c:7)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x40052C: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400536: main (bug.c:9)
==5955==
==5955== Conditional jump or move depends on uninitialised value(s)
==5955== at 0x400551: main (bug.c:11)

Note that valgrind only reports uninitialized values usage once they lead to non-determinism (i.e., when the program encounters a branch whose choice depends on the result of an uninitiated value). This means you the first report you get is frequently not in the calculation done on the uninitialized values but rather the convergence test or print statement at the end of calculations (print statements contains a bunch of branches in order to handling printing of each different digit).

=== Tracking down uninitialized values ===

More typically a program will be composed of multiple routines that all work on the data. To this end, consider the following ''bug.c'' code

<source lang="C">
#include <stdio.h>

void initialize_sequence(double* array, const int array_length) {
int i;

for (i=1; i<array_length+1; ++i)
array[i] = i;
}

double sum(const double* array, const int array_length) {
double array_sum;
int i;

for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}

int main() {
double array[10];
const int array_length = sizeof(array)/sizeof(array[0]);

double array_sum;

initialize_sequence(array,array_length);
array_sum = sum(array,array_length);

printf("the sum of 0..%d is %f\n", array_length-1, array_sum);

return 0;
}
</source>

It has both indexing and uninitialized value bugs. Despite this, directly running the code

<source lang="bash">
cc -Wall -g bug.c -o bug
./bug
</source>

will mostly likely produce the correct answer on most machines most of the time

the sum of 0..9 is 45.000000

Running under valgrind

<source lang="bash">
valgrind ./bug
</source>

however, reliably gives many warnings of the following form

==15930== Conditional jump or move depends on uninitialised value(s)
==15930== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==15930== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==15930== by 0x5318189: printf (in /lib64/libc-2.12.so)
==15930== by 0x400722: main (bug.c:29)

This is a typical numeric code example were the warnings first occur at a print statement because this is the first time the uninitialized value leads to non-determinism (i.e., the program's behaviour is altered as it takes or does not take a branch based on a result computed from something that was not set by the programmer).

We now know the problem is that something unset went into computing the value of ''array_sum''. We can then trace each of the ''array_sum'' calculation backwards. This quickly gets difficult though as we then find ourselves also tracing back all variables that went into the ''array_sum'' calculation, and then all variables that went into those variables and so on.

Fortunately Valgrind provides a ''--track-origins=yes'' flag to ease our search by telling us which ones we can ignore. Re-running with this flag

<source lang="bash">
valgrind --track-origins=yes ./bug
</source>

our warnings messages are augmented to include information on the source of the uninitialized value that went into the computation of ''array_sum''

==17589== Conditional jump or move depends on uninitialised value(s)
==17589== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==17589== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==17589== by 0x5318189: printf (in /lib64/libc-2.12.so)
==17589== by 0x400722: main (bug.c:29)
==17589== Uninitialised value was created by a stack allocation
==17589== at 0x400632: sum (bug.c:10)

Now we know the source of our uninitialized value in the ''array_sum'' computation is a local variable in the ''sum'' routine. Looking into ''sum'' we hopefully figure out without too much difficulty that we forgot to initialize ''array_sum'' to zero. If our program is producing the correct answer, it is only because we are getting lucky and ''array_sum'' happens to be allocated from memory that is initially zero.

Correcting the ''sum'' routine

<source lang="bash">
double sum(const double* array, const int array_length) {
double array_sum;
int i;

array_sum = 0.0;
for (i=0; i<array_length; ++i)
array_sum += array[i];

return array_sum;
}
</source>

and re-running under valgrind reveals we are still getting uninitialized values warnings associated with the ''array_sum'' computation. Now (using the ''--track-origins=yes'') they are of the form

==18613== Conditional jump or move depends on uninitialised value(s)
==18613== at 0x5312CF0: __printf_fp (in /lib64/libc-2.12.so)
==18613== by 0x530E89F: vfprintf (in /lib64/libc-2.12.so)
==18613== by 0x5318189: printf (in /lib64/libc-2.12.so)
==18613== by 0x40072C: main (bug.c:30)
==18613== Uninitialised value was created by a stack allocation
==18613== at 0x400690: main (bug.c:21)

Valgrind is telling us that a local variable inside the ''main'' went into the computation of ''array_sum'' despite never being set. This means it must be ''array'' itself or ''array_length''. It cannot be ''array_length'' as the former is clearly set to the compiler computed ''sizeof(array)/sizeof(array[0])'' so it must be ''array'' itself.

It is clear the intent of the code is that ''array'' is initialized by the ''initialize_sequence'' routine. Careful examination of this routine reveals the final error. The initialization loop was done using Fortran indices (1...''array_length'') instead of C (0...''array_length-1''). Correcting this routine

<source lang="bash">
void initialize_sequence(double* array, const int array_length) {
int i;

for (i=0; i<array_length; ++i)
array[i] = i;
}
</source>

finally produces code the runs under valgrind without any warnings.

'''NOTE''' The Valgrind warnings were being generated because the ''array[0]'' value was not being initialized. The code was also incorrect in that it was initializing ''array[array_length]'' which is one past the end of ''array''. If you put this later error back into the program and run under Valgrind you will discover it does not produce any warnings. This shows that even codes that successfully under Valgrind can still contain errors.

== MPI code ==

Consider the following ''bug.c'' MPI code

<source lang="C">
#include <mpi.h>

int main(int argc,char *argv[]){
int rank, size;
int value;

MPI_Init(&argc, &argv);

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

if (rank == 0 && size > 1) {
MPI_Request request;

MPI_Irecv(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);
value = 0;
MPI_Wait (&request, MPI_STATUS_IGNORE);
}
else if (rank == 1) {
MPI_Request request;

MPI_Isend(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);
value = 1;
MPI_Wait(&request, MPI_STATUS_IGNORE);
}

MPI_Finalize();

return 0;
}
</source>

It has an uninitialized value bug and two race condition bugs around the use of value.

# The first race condition is that rank==0 sets value=0 while at the same time doing a non-blocking receive into value (bug:16).
# The uninitialized value problem is that rank==1 starts a send of value to rank==0 without ever setting value (bug:22).
# The second race condition is that rank==1 sets value=1 while at the same time doing a non-blocking send of value (bug:23).

=== Basic Functionality ===

If you run this using the non-debug/valgrind openmpi valgrind

<source lang="bash">
module unload intel openmpi
module load intel/12.1.3 openmpi/intel/1.6.2
mpicc -Wall -g bug.c -o bug
mpirun -np 2 valgrind ./bug
</source>

you are presumably getting a report about the uninitialized value, but it is burried in 55517 other bogus error messages.

This solution to this is to link against the valgrind openmpi debug wrapper library too.

<source lang="bash">
mpicc -Wall -g bug.c -L/usr/lib64/valgrind -lmpiwrap-amd64-linux -Xlinker -rpath=/usr/lib64/valgrind -o bug
mpirun -np 2 valgrind ./bug
</source>

This now only reports one bogus error (a free on exit) and picks up the sending of the uninitialized value

==27638== Uninitialised byte(s) found during client check request
==27638== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27638== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27638== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27638== by 0x402713: main (bug.c:22)
==27638== Address 0x7fefff734 is on thread 1's stack

=== Advanced Functionality ===

Now, if on top of this, you also bring in the valgrind enabled openmpi debug library, things get really sweet

<source lang="bash">
module unload intel openmpi
module load intel/12.1.3 openmpi/intel-debug/1.6.2
mpicc -Wall -g bug.c -o bug
LD_PRELOAD=/usr/lib64/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind ./bug
</source>

You still only get one bogus error (a free on exit) and all the bugs in the code are detected and reported

==27774== Uninitialised byte(s) found during client check request
==27774== at 0x4E3641D: check_mem_is_defined_untyped (libmpiwrap.c:952)
==27774== by 0x4E5BBC5: generic_Isend (libmpiwrap.c:908)
==27774== by 0x4E5BEE9: PMPI_Isend (libmpiwrap.c:1393)
==27774== by 0x402713: main (bug.c:22)
==27774== Address 0x7fefff6f4 is on thread 1's stack

==27773== Invalid write of size 4
==27773== at 0x4026A0: main (bug.c:16)
==27773== Address 0x7fefff6f4 is on thread 1's stack

==27774== Invalid write of size 4
==27774== at 0x40271B: main (bug.c:23)
==27774== Address 0x7fefff6f4 is on thread 1's stack

=== Suppression File ===

There is also a valgrind suppression option <tt>--suppressions=/opt/sharcnet/openmpi/1.6.2/intel-debug/share/openmpi/openmpi-valgrind.supp</tt>, however we have not observed any cases where this makes a difference yet.

=References=

o Package website 
http://www.valgrind.org

[[Category:Software packages]]

VALGRIND

2015-09-08T14:34:22Z

Tyson: Reformat version recommendations to make the more prominent

VALGRIND

2015-09-08T14:32:50Z

Tyson: Make a note of GCC and OpenMPI version recommendation

VALGRIND

2015-02-10T23:53:24Z

Tyson: Can't use LD_PRELOAD as required MPI library not in path (link against library instead)

MPIBLAST

2014-11-17T17:00:07Z

Tyson: Add bioinformatics category

{{Software
|package_name=MPIBLAST
|package_description=Parallel implementation of NCBI BLAST
|package_idnumber=55}}

=Introduction=

The mpiblast module must be manually loaded before submitting any mpiblast jobs. Two examples below
demonstrate how to setup and submit jobs to the mpi queue.

=Version Selection=

===All Clusters (except Guppy)===

module unload openmpi intel
module load intel/12.1.3 openmpi/intel/1.6.2 mpiblast/1.6.0

===Guppy Only===

module unload openmpi intel
module load intel/11.0.083 openmpi/intel/1.4.2 mpiblast/1.6.0

=Sample Problems=

==DROSOPH==

Copy sample problem files (fasta database and input) from the /opt/sharcnet examples directory a directory under work as shown here. The fasta database used in this example can be obtained as a guest from NCBI here http://www.ncbi.nlm.nih.gov/guide/all/#downloads_ then clicking "FTP: FASTA BLAST Databases".

<pre>
mkdir -p /work/$USER/samples/mpiblast/test1
rm /work/$USER/samples/mpiblast/test1/*
cd /work/$USER/samples/mpiblast/test1
cp /opt/sharcnet/mpiblast/1.6.0/examples/drosoph.in drosoph.in
gunzip -c /opt/sharcnet/mpiblast/1.6.0/examples/drosoph.nt.gz > drosoph.nt.$USER
</pre>

Create a hidden configuration file using a text editor (such as vi) to define a Shared storage location between nodes and a Local storage directory available on each compute node as follows:

<pre>
[roberpj@hnd20:/work/$USER/samples/mpiblast/test1] vi .ncbirc
[NCBI]
Data=/opt/sharcnet/mpiblast/1.6.0/ncbi/data
[BLAST]
BLASTDB=/scratch/YourSharcnetUsername/mpiblasttest1
BLASTMAT=/work/YourSharcnetUsername/samples/mpiblast/test1
[mpiBLAST]
Shared=/scratch/YourSharcnetUsername/mpiblasttest1
Local=/tmp
</pre>

Partition the database into 8 fragments into /scratch/$USER/mpiblasttest1 where they will be searched for when running a job in queue according to the .ncbirc file:

<pre>
module load mpiblast/1.6.0
mkdir /scratch/roberpj/mpiblasttest1
rm -f /scratch/$USER/mpiblasttest1/*
cd /work/$USER/samples/mpiblast/test1

[roberpj@hnd20:/work/$USER/samples/mpiblast/test1] mpiformatdb -N 8 -i drosoph.nt.$USER -o T -p F -n /scratch/$USER/mpiblasttest1
Reading input file
Done, read 1534943 lines
Breaking drosoph.nt into 8 fragments
Executing: formatdb -p F -i drosoph.nt -N 8 -n /scratch/roberpj/mpiblasttest1/drosoph.nt -o T
Created 8 fragments.
<<< Please make sure the formatted database fragments are placed in /scratch/roberpj/mpiblasttest1/ before executing mpiblast. >>>

[roberpj@hnd20:/scratch/roberpj/mpiblasttest1] ls
drosoph.nt.000.nhr drosoph.nt.002.nin drosoph.nt.004.nnd drosoph.nt.006.nni
drosoph.nt.000.nin drosoph.nt.002.nnd drosoph.nt.004.nni drosoph.nt.006.nsd
drosoph.nt.000.nnd drosoph.nt.002.nni drosoph.nt.004.nsd drosoph.nt.006.nsi
drosoph.nt.000.nni drosoph.nt.002.nsd drosoph.nt.004.nsi drosoph.nt.006.nsq
drosoph.nt.000.nsd drosoph.nt.002.nsi drosoph.nt.004.nsq drosoph.nt.007.nhr
drosoph.nt.000.nsi drosoph.nt.002.nsq drosoph.nt.005.nhr drosoph.nt.007.nin
drosoph.nt.000.nsq drosoph.nt.003.nhr drosoph.nt.005.nin drosoph.nt.007.nnd
drosoph.nt.001.nhr drosoph.nt.003.nin drosoph.nt.005.nnd drosoph.nt.007.nni
drosoph.nt.001.nin drosoph.nt.003.nnd drosoph.nt.005.nni drosoph.nt.007.nsd
drosoph.nt.001.nnd drosoph.nt.003.nni drosoph.nt.005.nsd drosoph.nt.007.nsi
drosoph.nt.001.nni drosoph.nt.003.nsd drosoph.nt.005.nsi drosoph.nt.007.nsq
drosoph.nt.001.nsd drosoph.nt.003.nsi drosoph.nt.005.nsq drosoph.nt.dbs
drosoph.nt.001.nsi drosoph.nt.003.nsq drosoph.nt.006.nhr drosoph.nt.mbf
drosoph.nt.001.nsq drosoph.nt.004.nhr drosoph.nt.006.nin drosoph.nt.nal
drosoph.nt.002.nhr drosoph.nt.004.nin drosoph.nt.006.nnd
</pre>

Submit a short job with a 15m time limit on n=16 cores calculate by taking N=8 fragments + 8. If all goes well output results will be written to drosoph.out and the execution time will appear in ofile%J where %J is the job number:

<pre>
[roberpj@hnd20:/work/$USER/samples/mpiblast/test1]
sqsub -r 15m -n 106 -q mpi --mpp=1G -o ofile%J mpiblast -d drosoph.nt.$USER -i drosoph.in
-p blastn -o drosoph.out --use-parallel-write --use-virtual-frags
submitted as jobid 6966896

[roberpj@hnd20:/work/roberpj/samples/mpiblast/test1] cat ofile6966896.hnd50
Total Execution Time: 1.80031
</pre>

When submitting a mpiblast job on a cluster such as goblin that doesnt have an inifiniband interconnect better performance (at least double speedup) will be achieved running the mpi job on one compute node. For regular users of non-contributed hardware typically specify "-n 8" to reflect the max number of cores on a single node:

<pre>
sqsub -r 15m -n 8 -N 1 -q mpi --mpp=4G -o ofile%J mpiblast -d drosoph.nt -i drosoph.in
-p blastn -o drosoph.out --use-parallel-write --use-virtual-frags
</pre>

Sample output results computed previously with BLASTN 2.2.15 [Oct-15-2006] are included in /opt/sharcnet/mpiblast/1.6.0/examples/ROSOPH.out to compare your newly generated drosoph.out file with.

==UNIGENE==

The main purpose of this example is to illustrate some additional options and switchs that maybe useful for debugging and for dealing with larger databases as described in official detail at http://www.mpiblast.org/Docs/Guide. The fasta database used in this example can also be downloaded from http://www.ncbi.nlm.nih.gov/guide/all/#downloads_ as a guest by clicking "FTP: UniGene" then entering the "Homo_sapiens" sub-directory. More information about UniGene alignments can be found at https://cgwb.nci.nih.gov/cgi-bin/hgTrackUi?hgsid=95443&c=chr1&g=uniGene_3 . As with Example1 above, for convenience all required files can simply be copied from the /opt/sharcnet examples subdirectory to work as shown here:

<pre>
mkdir /work/$USER/samples/mpiblast/test2
rm -f /work/$USER/samples/mpiblast/test2/*
cd /work/$USER/samples/mpiblast/test2
cp /opt/sharcnet/mpiblast/1.6.0/examples/il2ra.in il2ra.in
gunzip -c /opt/sharcnet/mpiblast/1.6.0/examples/Hs.seq.uniq.gz > Hs.seq.$USER

[roberpj@orc-login2:/work/roberpj/samples/mpiblast/test2] ls
Hs.seq.roberpj il2ra.in
</pre>

Create a hidden configuration file using a text editor (such as vi) to define a Shared storage location between nodes and a Local storage directory available on each compute node as followw. Note that the ncbi/data directory is not used in this example and hence can be omitted. If the Local and Shared directories are the same replace --copy-via=mpi with --copy-via=none as will be demonstrated in the below sqsub commands.

<pre>
[username@orc-login1:/work/$USER/samples/mpiblast/test2] vi .ncbirc
[NCBI]
Data=/opt/sharcnet/mpiblast/1.6.0/ncbi/data
[BLAST]
BLASTDB=/work/YourSharcnetUsername/mpiblasttest2
BLASTMAT=/work/YourSharcnetUsername/samples/mpiblast/test2
[mpiBLAST]
Shared=/work/YourSharcnetUsername/mpiblasttest2
Local=/tmp
</pre>

Again partition the database into 8 fragments into /work/$USER/mpiblasttest2 where they will be searched for when a job runs in the queue according to the .ncbirc file, first removing previous partitioning if any such files exist. For this example assume $USER=roberpj however be sure to replace with your username!

<pre>
module load mpiblast/1.6.0
mkdir -p /work/$USER/mpiblasttest2
rm -f /work/$USER/mpiblasttest2/*
cd /work/$USER/samples/mpiblast/test2

[roberpj@orc-login1:/work/roberpj/samples/mpiblast/test2] mpiformatdb -N 8 -i Hs.seq.$USER -o T -p F -n /work/roberpj/mpiblasttest2
Reading input file
Done, read 2348651 lines
Breaking Hs.seq.roberpj into 8 fragments
Executing: formatdb -p F -i Hs.seq.roberpj -N 8 -n /work/roberpj/mpiblasttest2/Hs.seq.roberpj -o T
Created 8 fragments.
<<< Please make sure the formatted database fragments are placed in /work/roberpj/mpiblasttest2/ before executing mpiblast. >>>

[roberpj@orc-login1:/work/roberpj/samples/mpiblast/test1] ls
formatdb.log Hs.seq.roberpj il2ra.in

[roberpj@orc-login1:/work/roberpj/mpiblasttest2] ls
Hs.seq.roberpj.000.nhr Hs.seq.roberpj.002.nin Hs.seq.roberpj.004.nnd Hs.seq.roberpj.006.nni
Hs.seq.roberpj.000.nin Hs.seq.roberpj.002.nnd Hs.seq.roberpj.004.nni Hs.seq.roberpj.006.nsd
Hs.seq.roberpj.000.nnd Hs.seq.roberpj.002.nni Hs.seq.roberpj.004.nsd Hs.seq.roberpj.006.nsi
Hs.seq.roberpj.000.nni Hs.seq.roberpj.002.nsd Hs.seq.roberpj.004.nsi Hs.seq.roberpj.006.nsq
Hs.seq.roberpj.000.nsd Hs.seq.roberpj.002.nsi Hs.seq.roberpj.004.nsq Hs.seq.roberpj.007.nhr
Hs.seq.roberpj.000.nsi Hs.seq.roberpj.002.nsq Hs.seq.roberpj.005.nhr Hs.seq.roberpj.007.nin
Hs.seq.roberpj.000.nsq Hs.seq.roberpj.003.nhr Hs.seq.roberpj.005.nin Hs.seq.roberpj.007.nnd
Hs.seq.roberpj.001.nhr Hs.seq.roberpj.003.nin Hs.seq.roberpj.005.nnd Hs.seq.roberpj.007.nni
Hs.seq.roberpj.001.nin Hs.seq.roberpj.003.nnd Hs.seq.roberpj.005.nni Hs.seq.roberpj.007.nsd
Hs.seq.roberpj.001.nnd Hs.seq.roberpj.003.nni Hs.seq.roberpj.005.nsd Hs.seq.roberpj.007.nsi
Hs.seq.roberpj.001.nni Hs.seq.roberpj.003.nsd Hs.seq.roberpj.005.nsi Hs.seq.roberpj.007.nsq
Hs.seq.roberpj.001.nsd Hs.seq.roberpj.003.nsi Hs.seq.roberpj.005.nsq Hs.seq.roberpj.dbs
Hs.seq.roberpj.001.nsi Hs.seq.roberpj.003.nsq Hs.seq.roberpj.006.nhr Hs.seq.roberpj.mbf
Hs.seq.roberpj.001.nsq Hs.seq.roberpj.004.nhr Hs.seq.roberpj.006.nin Hs.seq.roberpj.nal
Hs.seq.roberpj.002.nhr Hs.seq.roberpj.004.nin Hs.seq.roberpj.006.nnd
</pre>

Submit a couple of short jobs 15m time limit. If all goes well output results will be written to biobrewA.out and biobrewB.out and the execution time appear in corresponding ofile%J's where %J is the job number as per usual. In these examples we will submit the job from a saw specific directory which can be copied from test2 as follows:

<pre>
[roberpj@saw-login1:~] module load mpiblast/1.6.0
[roberpj@saw-login1:~] cd /work/roberpj/samples/mpiblast
[roberpj@saw-login1:/work/roberpj/samples/mpiblast] cp -a test2 test2.saw

[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] vi .ncbirc
[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] cat .ncbirc | grep BLASTMAT
BLASTMAT=/work/roberpj/samples/mpiblast/test2.saw
</pre>

A) Submit job with profile time option choosing n=24 (8 fragments + 8 or greater):

<pre>
[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] rm -f oTime*;
sqsub -r 15m -n 24 -q mpi -o ofile%J mpiblast --use-parallel-write --copy-via=mpi
-d Hs.seq.roberpj -i il2ra.in -p blastn -o biobrew.out --time-profile=oTime
</pre>

B) Usage of the debug option again choosing n=16 (8 fragments + 8 or greater):

<pre>
[roberpj@saw-login1:/work/roberpj/samples/mpiblast/test2.saw] rm -f oLog*;
sqsub -r 15m -n 16 -q mpi -o ofile%J mpiblast --use-parallel-write --copy-via=none
-d Hs.seq.$USER -i il2ra.in -p blastn -o biobrew.out --debug=oLog
</pre>

Finally compare /opt/sharcnet/mpiblast/1.6.0/examples/BIOBREW.out computed previously with BLASTN 2.2.15 [Oct-15-2006] with your newly generated biobrew.out output file to verify the results and submit a ticket if there are any problems!

SUPPORTED PROGRAMS IN MPIBLAST

As described in http://www.mpiblast.org/Docs/FAQ mpiblast supports the standard blast programs http://www.ncbi.nlm.nih.gov/BLAST/blast_program.shtml which are reproduced here for reference:

<pre>
blastp: Compares an amino acid query sequence against a protein sequence database.
blastn: Compares a nucleotide query sequence against a nucleotide sequence database.
blastx: Compares a nucleotide query sequence translated in all reading frames against
a protein sequence database.
tblastn: Compares a protein query sequence against a nucleotide sequence database
dynamically translated in all reading frames.
tblastx: Compares the six-frame translations of a nucleotide query sequence against
the six-frame translations of a nucleotide sequence database.
</pre>

MPIBLAST BINARIES OPTIONS

<pre>
[roberpj@orc-login1:/opt/sharcnet/mpiblast/1.6.0/bin] ./mpiblast -help
mpiBLAST requires the following options: -d [database] -i [query file] -p [blast program name]
</pre>

<pre>
[roberpj@orc-login1:/opt/sharcnet/mpiblast/1.6.0/bin] ./mpiformatdb --help
Executing: formatdb -

formatdb 2.2.20 arguments:

-t Title for database file [String] Optional
-i Input file(s) for formatting [File In] Optional
-l Logfile name: [File Out] Optional
default = formatdb.log
-p Type of file
T - protein
F - nucleotide [T/F] Optional
default = T
-o Parse options
T - True: Parse SeqId and create indexes.
F - False: Do not parse SeqId. Do not create indexes.
[T/F] Optional
default = F
-a Input file is database in ASN.1 format (otherwise FASTA is expected)
T - True,
F - False.
[T/F] Optional
default = F
-b ASN.1 database in binary mode
T - binary,
F - text mode.
[T/F] Optional
default = F
-e Input is a Seq-entry [T/F] Optional
default = F
-n Base name for BLAST files [String] Optional
-v Database volume size in millions of letters [Integer] Optional
default = 4000
-s Create indexes limited only to accessions - sparse [T/F] Optional
default = F
-V Verbose: check for non-unique string ids in the database [T/F] Optional
default = F
-L Create an alias file with this name
use the gifile arg (below) if set to calculate db size
use the BLAST db specified with -i (above) [File Out] Optional
-F Gifile (file containing list of gi's) [File In] Optional
-B Binary Gifile produced from the Gifile specified above [File Out] Optional
-T Taxid file to set the taxonomy ids in ASN.1 deflines [File In] Optional
-N Number of database volumes [Integer] Optional
default = 0
</pre>

=General Notes=

==Issue1: Jobs Fail to Start==

If the following error message occurs it maybe necessary to stop all mpiblast jobs you are running on the cluster and clear
out all the files from /tmp then submit your job again. In the case of Example2 above the following steps work to resolve the problem, where the nodes from the last run job are used red[7-14. Should there be any questions please open a problem ticket.

<pre>
Purging Job Node Subset (redfin)
pdsh -w red[7-14] ls /tmp/Hs* (confirm existence of file from previous runs)
pdsh -w red[7-14] rm -f /tmp/Hs*
pdsh -w red[7-14] ls /tmp/Hs* (confirm all files from previous runs removed)

Purging Full Cluster (hound)
pdsh -w hnd[1-18] -f 4 rm -f /tmp/Hs*
pdsh -w hnd[1-18] -f 4 ls /tmp/Hs*
</pre>

<pre>
[roberpj@red-admin:/work/roberpj/samples/mpiblast/test2.red4] cat ofile489214.red-admin.redfin.sharcnet
8 0.577904 Bailing out with signal 11
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 8 in communicator MPI_COMM_WORLD
with errorcode 0.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
20 0.577719 Bailing out with signal 11
0 0.591128 Bailing out with signal 15
[red14:09194] [[36424,0],0]-[[36424,1],0] mca_oob_tcp_msg_recv: readv failed: Connection reset by peer (104)
1 0.591327 Bailing out with signal 15
etc
</pre>

==Issue2: Jobs Run for a While then Die==

The solution here is to filter the input sequence file. For reasons yet understood the presence of repeat sections results in many thousands of WARNING and ERROR messages rapidly written to the "sqsub -o ofile" output file presumably as mpiblast ignores sequences before eventually diing after several hours, or possibly days.

<pre>
# cat ofile1635556.saw-admin.saw.sharcnet | grep "WARNING\|ERROR" | wc -l
10560
</pre>

<pre>
# cat ofile1635556.saw-admin.saw.sharcnet
Selenocysteine (U) at position 60 replaced by X
Selenocysteine (U) at position 42 replaced by X
[blastall] WARNING: [000.000] NODE_84_length_162_cov_46.259258_1_192_-: SetUpBlastSearch failed.
[blastall] ERROR: [000.000] NODE_84_length_162_cov_46.259258_1_192_-: BLASTSetUpSearch: Unable to calculate Karlin-Altschul params, check query sequence
<<<< snipped out ~10000 similar WARNING and ERROR messages from this example >>>>
[blastall] WARNING: [000.000] NODE_65409_length_87_cov_2.367816_1_77_+: SetUpBlastSearch failed.
[blastall] ERROR: [000.000] NODE_65409_length_87_cov_2.367816_1_77_+: BLASTSetUpSearch: Unable to calculate Karlin-Altschul params, check query sequence
Selenocysteine (U) at position 61 replaced by X
Selenocysteine (U) at position 62 replaced by X
Selenocysteine (U) at position 34 replaced by X
Selenocysteine (U) at position 1058 replaced by X
--------------------------------------------------------------------------
mpirun noticed that process rank 52 with PID 30067 on node saw214 exited on signal 9 (Killed).
--------------------------------------------------------------------------
1618 17 651 62909.3 Bailing out with signal 15
14 3 62909.3 Bailing out with signal 15
19 15 62909.3 Bailing out with signal 1536 5 7 62909.312 62909.310 62909.3 Bailing out with signal 1547 21 62909.3 Bailing out with signal 159 62909.3 Bailing out with signal 15
45 62909.3 Bailing out with signal 1525 62909.3 Bailing out with signal 158 62909.3 Bailing out with signal 15
50 62909.3 Bailing out with signal 1522 62909.3 Bailing out with signal 15
11 62909.3 Bailing out with signal 15
48 62909.323 62909.3 Bailing out with signal 15
Bailing out with signal 15
46 62909.3 Bailing out with signal 15
etc
</pre>

=References=

o MPIBLAST Homepage 
http://www.mpiblast.org/

o MPIBLAST Version History 
http://www.mpiblast.org/Downloads/Version-History

[[Category:Bioinformatics]]

File:Seminar-2014-04-16-slides.pdf

2014-04-16T19:13:09Z

Tyson:

VALGRIND

2013-10-07T15:25:20Z

Tyson: Add instructions and examples for serial and MPI code

MediaWiki:MenuSidebar

2010-11-05T15:24:55Z

Tyson: Update links now that MediaWiki code is supported

* Common Destinations
** [[Knowledge Base]]
** [http://www.sharcnet.ca/my SHARCNET Web Portal]
* Clusters Facilities
** [[Main_Page#SHARCNET|Working with the Clusters]]
** [[Main_Page#Programming|Programming the Clusters]]
* Other Facilities
** [[Main_Page#Visualization|Visualization]]
** [[Main_Page#AccessGrid|Video Conferencing]]
* Domain Specific Portals
** [[Astrophysics]]
** [[Bioinformatics]]
** [[Digital Humanities]]
** [[Chemistry, Biochemistry and Biophysics]]
* SEARCH
* TOOLBOX
* LANGUAGES

MediaWiki:Sidebar

2010-08-06T17:57:16Z

Tyson: Put back to the same in order to avoid toolbars floating to the top

* Common Destinations
** Knowledge_Base|Knowledge Base
** http://www.sharcnet.ca/my|SHARCNET Web Portal
* Clusters Facilities
** Main_Page#SHARCNET|Working with the Clusters
** Main_Page#Programming|Programming the Clusters
* Other Facilities
** Main_Page#Visualization|Visualization
** Main_Page#AccessGrid|Video Conferencing
* Domain Specific Portals
** Astrophysics|Astrophysics
** Bioinformatics|Bioinformatics
** Digital_Humanities|Digital Humanities
** Chemistry,_Biochemistry_and_Biophysics|Chemistry, Biochemistry and Biophysics
* SEARCH
* TOOLBOX
* LANGUAGES

MediaWiki:Sidebar

2010-08-06T17:50:11Z

Tyson: Screws up if entirely blank, try generic ones at bottom

* SEARCH
* TOOLBOX
* LANGUAGES

MediaWiki:Sidebar

2010-08-06T17:48:54Z

Tyson: We are using the MenuSideBar extension instead

MediaWiki:MenuSidebar

2010-08-06T17:44:52Z

Tyson: Duplicate new SideBar menu

MediaWiki:MenuSidebar

2010-08-06T17:32:52Z

Tyson: Don't use for now (just use Sidebar instead)

MediaWiki:Sidebar

2010-08-06T17:12:29Z

Tyson: A possible alternative sidebar

File:Winscp folders.png

2010-07-05T19:25:55Z

Tyson: The WinSCP main interface (two drag-and-drop panes)

The WinSCP main interface (two drag-and-drop panes)

File:Winscp newkey.png

2010-07-05T19:25:00Z

Tyson: The WinSCP new key window

The WinSCP new key window

File:Winscp initial.png

2010-07-05T19:11:52Z

Tyson: uploaded a new version of "File:Winscp initial.png": Initial window re-uploaded (original had corruption)

Initial WinSCP window

File:Winscp initial.png

2010-07-05T19:10:22Z

Tyson: Initial WinSCP window

Initial WinSCP window

File:Putty terminal.png

2010-07-05T19:05:39Z

Tyson:

File:Putty newkey.png

2010-07-05T18:46:33Z

Tyson: The PuTTY new key window

The PuTTY new key window

File:Putty initial.png

2010-07-05T18:35:35Z

Tyson: Initial screen upon running PuTTY

Initial screen upon running PuTTY