Documentation - User contributions [en]

LSDYNA

2019-02-08T20:16:55Z

Roberpj: /* Command Line Use */

{{Software
|package_name=LSDYNA
|package_description=Suite of programs for transient dynamic finite element program
|package_idnumber=104
}}

=Introduction=

Before a research group can use LSDYNA on sharcnet, a license must be purchased directly from LSTC for the sharcnet license server. Alternately, if a research group resides at a institution that has sharcnet computing hardware (mac, uwo, guelph, waterloo) it maybe possible to use a pre-existing site license hosted on an accessible institutional license server. To access and use this software you must open a ticket and request to be added to the sharcnet lsdyna group.

=Version Selection=

==Graham Cluster==

===License===

Create and customize this file for your license server, where XXXXX should be replaced with your port number and Y with your server number:

[roberpj@gra1:~] cat .licenses/ls-dyna.lic
#LICENSE_TYPE: network
#LICENSE_SERVER: XXXXX@licenseY.uwo.sharcnet

===Module===

The modules are loaded automatically in the testomp.sh and testmpi.sh script shown below in the example section:

For single node smp jobs the available versions can be found by doing:

[roberpj@gra-login2:~] module spider ls-dyna
ls-dyna/7.1.2
ls-dyna/7.1.3
ls-dyna/8.1
ls-dyna/9.1
ls-dyna/9.2
ls-dyna/10.0

To load any version do for example:

module load ls-dyna/9.1

For multi node mpp jobs the available versions can be found by doing:

[roberpj@gra-login2:~] module spider ls-dyna-mpi
ls-dyna-mpi/7.1.2
ls-dyna-mpi/7.1.3
ls-dyna-mpi/8.1
ls-dyna-mpi/9.1
ls-dyna-mpi/9.2
ls-dyna-mpi/10.0

To load the 7.1.X modules do the following:
module load openmpi/1.6.5 ls-dyna-mpi/7.1.3

To load the 8.X or 9.X modules do:
module load openmpi/1.8.8 ls-dyna-mpi/9.2

To load the 10.X modules do:
module load openmpi/1.10.7 ls-dyna-mpi/10.0

==Legacy Clusters==

===License===

Once a license configuration is established, the research group will be given a 5 digit port number. The value should then be inserted into the appropriate departmental export statement before loading the module file as follows:

o UofT Mechanical Engineering Dept
export LSTC_LICENSE_SERVER=Port@license1.uwo.sharcnet

o McGill Mechanical Engineering Department
export LSTC_LICENSE_SERVER=Port@license2.uwo.sharcnet

o UW Mechanical and Mechatronics Engineering Dept
export LSTC_LICENSE_SERVER=Port@license3.uwo.sharcnet

o Laurentian University Bharti School of Engineering
export LSTC_LICENSE_SERVER=Port@license4.uwo.sharcnet

o Fictitious Example:

export LSTC_LICENSE_SERVER=12345@license1.uwo.sharcnet

===Module===

The next step is load the desired sharcnet lsdyna module version. Check which modules are available by running the module avail command then load one the modules as shown:

<source lang="bash">
[roberpj@orc-login2:~] module avail lsdyna
------------------------------------ /opt/sharcnet/modules ------------------------------------
lsdyna/hyb/r711.88920 lsdyna/mpp/r711.88920 lsdyna/smp/r611.79036
lsdyna/mpp/ls971.r85718 lsdyna/mpp/r712.95028 lsdyna/smp/r711.88920
lsdyna/mpp/ls980B1.011113 lsdyna/mpp/r800.95359 lsdyna/smp/r712.95028
lsdyna/mpp/r611.79036 lsdyna/mpp/r901.109912 lsdyna/smp/r800.95359
lsdyna/mpp/r611.80542 lsdyna/smp/ls980B1.78258 lsdyna/smp/r901.109912
</source>

====r9.0.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r901.109912</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna mkl
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r901.109912
</pre>

====r8.0.0 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r800.95359</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna mkl
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r800.95359
</pre>

====r7.1.2 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r712.95028</pre>

o For mpi jobs:
<pre>module unload intel openmpi lsdyna
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
</pre>

====r7.1.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r711.88920</pre>

o For mpi jobs:
<pre>module load lsdyna/mpp/r711.88920</pre>

====r6.1.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r611.79036</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna

module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036
or
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.80542
</pre>

where r611.80542 provides lsdyna_s only. 

====ls980 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/ls980B1.78258</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna

module load intel/12.1.3 openmpi/intel/1.4.5 lsdyna/mpp/ls980B1.011113
or
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/smp/ls980B1.78258
</pre>

Note1) restart capability for ls980smpB1 and ls980mppB1 is not supported
Note2) the module using legacy openmpi/intel/1.4.5 will run extremely slow

=Job Submission=

==Graham Cluster==

The submission scripts myompjob.sh and mympijob.sh for the airbag problem are shown in the Example Job section below, for both graham and orca. Please note that you should specify your own username if submitting to a def account (not roberpj). Alternatively you could specify your resource allocation account:

Sample threaded job submit script:

sbatch myompjob.sh

Sample mpi job submit script:

sbatch mympijob.sh

==Legacy Clusters==

When loading a lsdyna sharcnet legacy module on the new orca or a sharcnet legacy system, the single or double precision solvers are specified with lsdyna_s or lsdyna_d respectively, as shown in the following sqsub commands:

===1cpu SERIAL Job===
<pre>sqsub -r 1h -q serial -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=1</pre>

===4cpu SMP Job===
<pre>sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=1G lsdyna_s i=airbag.deploy.k ncpu=4</pre>

If using an explicit solver, one can specify a conservative initial memory setting on the command line as follows:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=4 memory=754414</pre>
where memory is the minimum number of 8 byte words shared by all processors in double precision, or, 4 byte words in single precision.

The initial value can be determined by starting a simulation interactively on the command line, and finding the output resembling: Memory required to begin solution : 754414. The number of words can be specified as memory=260M instead of memory=260000000, for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/.

===8cpu MPI Job===

<pre>sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k</pre>

The initial memory can also be specified for mpi jobs on the sqsub command line with "memory=" for the first master processor to decompose the problem and "memory2=" used on all processors including the master to solves the decomposed problem, where the values are specified as 4 bytes per word in single precision and 8 bytes per word in double precision. The number of words can be specified as memory=260M instead of memory=260000000 OR memory2=260M instead of memory2=260000000 for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/. The initial values can be found by running simulation interactively on a orca compute node and checking the output for a line such as: Memory required to begin solution (memory= 464898 memory2= 158794 ) which could then be implemented for a job run in the queue by doing:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898 memory2=158794
</pre>

The specification of memory2 on sharcnet or compute canada clusters is not beneficial since the queue reserves the same memory per core for all nodes, such that the decomposition master node process cannot be allocated to have larger system memory. Therefore its sufficient to specify a single memory parameter by doing:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898
</pre>

where the LSTC_MEMORY variable will only allow the memory to grow for explicit simulations. The following slurm examples demonstrate how to prescribe the memory parameters exactly.

=Example Job=

Copy the airbag example to your account:

rsync -av orca.computecanada.ca:/opt/sharcnet/lsdyna/r901.109912/examples r901.109912_examples
cd r901.109912_examples/examples/misc/airbag
gunzip airbag.deploy.k.gz

==Graham (default modules)==

Please note that graham does not have the sharcnet legacy modules installed on it.

===Threaded Job ===

Sample submission script mysmpjob1.sh for 4 core single precision smp job using compute canada default cvmfs modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --nodes=1 # use one node
#SBATCH --cpus-per-task=4 # number threads
#SBATCH --mem=4000M # total memory
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=testsmp1 # %x = jobname
#SBATCH --output=slurm-%x-%N-%j.out
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# ----------------------------------------------------------
# IMPORTANT: to set license server do either:
# 1) export LSTC_LICENSE_SERVER=port@some.license.server
# 2) create ~/.licenses/ls-dyna.lic as shown above
# ----------------------------------------------------------
module load ls-dyna/9.1
ls-dyna_s i=airbag.deploy.k ncpu=4 memory=1000M
</pre>

where memory = 4000M / 4 (bytes/word) = 1000M words

===Mpi Job ===

Sample submission script mympijob2.sh for 4 core double precision mpp job using compute canada default cvmfs modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --ntasks=4 # number of ranks
#SBATCH --mem-per-cpu=4000M # memory per rank
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=testmpi2 # %x = jobname
#SBATCH --output=slurm-%x-%N-%j.out
# ----------------------------------------------------------
# IMPORTANT: to set license server do either:
# 1) export LSTC_LICENSE_SERVER=port@some.license.server
# 2) create ~/.licenses/ls-dyna.lic as shown above
# ----------------------------------------------------------
module load openmpi/1.8.8 ls-dyna-mpi/9.1
srun ls-dyna_d i=airbag.deploy.k memory=500M memory2=500M
</pre>

where memory = 4000M / 8 (bytes/word) = 500M words

==New Orca (legacy modules)==

Please note, while new orca has most compute canada cvmfs modules available by default it does not have the graham the ls-dyna or ls-dyna-mpi modules installed. Therefore currently the only way to run lsdyna is by using the legacy sharcnet lsdyna modules as shown in the following two smp and mpi examples.

===Threaded Job ===

Submission script mysmpjob3.sh to run 16 core single precision single node smp job with sharcnet legacy modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --nodes=1 # use one node
#SBATCH --cpus-per-task=16 # number threads
#SBATCH --mem=4000M # total memory
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=mysmpjob3-1node-16core # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# ----------------------------------------------------------------
# IMPORTANT: to set license server for legacy modules:
export LSTC_LICENSE_SERVER=31047@license3.sharcnet.ca
# since ~/.licenses/ls-dyna.lic does NOT work with legacy modules
# ----------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load lsdyna/smp/r712.95028
lsdyna_s i=airbag.deploy.k ncpu=$SLURM_CPUS_PER_TASK memory=500M
</pre>

where memory = 4000M / 8 (bytes/word) = 500M words

===MPI Job ===

To run the sharcnet legacy module lsdyna/mpp/r712.95028 efficiently over several compute nodes with many cores requires using orca's xeon nodes as shown in mympijob4.sh below. To run lsdyna on orca's opteron nodes with module lsdyna/mpp/r712.95028 requires using only one opteron node as shown in mympijob5.sh below. The other legacy sharcnet lsdyna mpp modules, as shown in the Availability Table lsdyna/mpp/ls971.r85718, lsdyna/mpp/ls980B1.011113, lsdyna/mpp/r611.79036, lsdyna/mpp/r611.80542, lsdyna/mpp/r711.88920, lsdyna/mpp/r800.95359 and lsdyna/mpp/r901.109912 have yet to be tested. If you want to use one but cannot get it working please open a ticket. Note that lines containing a space between "# and SBATCH" are comments. To activate such line(s) required removal of the space ie) "#SBATCH". Its recommended to use module ls-dyna-mpi on graham instead since the machine is far newer, faster and reliable.

====Multi Node Slurm Script====

<pre>
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob4.sh
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --constraint=xeon
#SBATCH --ntasks=48
#SBATCH --mem-per-cpu=4000M
#SBATCH --time=00:30:00
#SBATCH --job-name=mympijob4-Xnodes-48cores-xeon # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
# -------------------------------------------------------------
# To set license server when using legacy sharcnet modules:
export LSTC_LICENSE_SERVER=port@some.license.server
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.
# -------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`
mpirun --host $MACHINEFILE --mca plm_slurm_args '--cpu_bind=none' lsdyna_s i=airbag.deploy.k memory=250M memory2=250M
</pre>

====Single Node Slurm Script====

<pre>
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob5.sh
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --constraint=opteron
#SBATCH --nodes=1
# SBATCH --ntasks=24
# SBATCH --mem=0
#SBATCH --ntasks=16
#SBATCH --mem-per-cpu=1000M
#SBATCH --time=00:30:00
#SBATCH --job-name=mympijob5-1node-16cores-opteron # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
# -------------------------------------------------------------
# To set license server when using legacy sharcnet modules:
export LSTC_LICENSE_SERVER=port@some.license.server
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.
# -------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`
mpirun --host $MACHINEFILE lsdyna_s i=airbag.deploy.k memory=250M memory2=250M
</pre>

where memory = 1000M / 4 (bytes/word) = 250M words

==Legacy Clusters==

STEP1) The following shows sqsub submission of the [http://www.dynaexamples.com/examples-manual/misc/airbag airbag] example to the mpi queue. Its recommended to first edit airbag.deploy.k and change endtim to 3.000E-00 so the job runs long enough to perform the restart in steps 2 and 3 below:
<pre>
cp -a /opt/sharcnet/lsdyna/r611.79036/examples /scratch/$USER/test-lsdyna
cd /scratch/$USER/test-lsdyna/misc/airbag
gunzip airbag.deploy.k.gz
cp airbag.deploy.k airbag.deploy.restart.k
nano deploy.restart.k (reduce plot file creation frequency)
*DATABASE_BINARY_D3PLOT
$ dt lcdtdeploy.restart.k
1.000E-01 <--- change from 5.000E-04
export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet
module unload intel openmpi lsdyna
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036
sqsub -r 10m -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.k
</pre>

STEP2) With the job still running, use the echo command as follows to create a file called "D3KIL" that will trigger generation of restart files at which point the file D3KIL itself will be erased. Do this once a day if data loss is critical to you OR once or twice just before the sqsub -r time limit is reached. Further information can be found here http://www.dynasupport.com/tutorial/ls-dyna-users-guide/sense-switch-control:

<pre>
echo "sw3" > D3KIL
sqkill job#
</pre>

STEP3) Before the job can be restarted, the following two lines must be added to the airbag.deploy.restart.k file:

<pre>
*CHANGE_CURVE_DEFINITION
1
</pre>

STEP4) Now resubmit the job as follows using "r=" to specify the restart file:

<pre>
sqsub -r 1h -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.restart.k r=d3dump01
</pre>

=General Notes=

==Graham Cluster==

There are no special notes at the present time in regards to using lsdyna on graham.

==Command Line Use==

Following the upgrade of orca to the compute canada module stack, the orca development nodes were discontinued. 
Therefore this section is no longer relevent and remain for historical purposes only.

Parallel lsdyna jobs can be run interactively on the command line (outside the queue) for short run testing purposes as follows:

o Orca Development Node MPP (mpi) Example

ssh orca.sharcnet.ca
ssh orc-dev1 (or dev2,3,4)
module unload intel openmpi lsdyna
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/mpp/r711.88920
mpirun -np 8 lsdyna_d i=airbag.deploy.k

o Orca Development Node SMP (threaded) Example

ssh orca.sharcnet.ca
ssh orc-dev1 (or dev2,3,4)
module unload intel openmpi lsdyna
module load lsdyna/smp/r711.88920
lsdyna_d i=airbag.deploy.k ncpu=8

==Memory Issues==

A minumum of mpp=2G is recommended although the memory requirement of a job may suggest much less is required. For instance setting mpp=1G for the airbag test job above will result in the following error when running a job in the queue:

<pre>
--------------------------------------------------------------------------
An attempt to set processor affinity has failed - please check to
ensure that your system supports such functionality. If so, then
this is probably something that should be reported to the OMPI developers.
--------------------------------------------------------------------------
</pre>

Another error message which can occur after a job runs for a while if mpp is chosen to small is:

<pre>
>>>>> Process 45 <<<<<
>>>>> Signal 11 : Segmentation Violation <<<<<
</pre>

To get an idea of the amount of memory a job used run grep on the output file:

<pre>
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep Memory
| Distributed Memory Parallel |
Memory size from command line: 500000, 500000
Memory for the head node
Memory installed (MB) : 32237
Memory free (MB) : 11259
Memory required (MB) : 0
Memory required to process keyword : 458120
Memory required for decomposition : 458120
Memory required to begin solution (memory= 458120 memory2= 230721)
Max. Memory reqd for implicit sol: max used 0
Max. Memory reqd for implicit sol: incore 0
Max. Memory reqd for implicit sol: oocore 0
Memory required to complete solution (memory= 458120 memory2= 230721)
</pre>

==Version and Revision==

Again run grep on the output file to extract the major and minor revision:

<pre>
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep 'Revision\|Version'
| Version : mpp s R6.1.1 Date: 01/02/2013 |
| Revision: 78769 Time: 07:43:30 |
| SVN Version: 80542 |
</pre>

==Check License Status==

===Running and Queued Programs===

To get a summary of running and/or queued jobs use the lstc_qrun command as follows, where queued means the job has started running according to the sqsub command but its actually sitting waiting for license resources to come available. Once these are acquired the job will start running on the cluster and appear as a Running Program according to lstc_qrun ie)

<source lang="bash">
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet

[roberpj@hnd19:~] lstc_qrun
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
Running Programs
User Host Program Started # procs
-----------------------------------------------------------------------------
dgierczy 20277@saw61.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1
dgierczy 8570@saw32.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1
dscronin 25486@hnd6 MPPDYNA_971 Tue Apr 9 20:19 6
dscronin 14897@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 14971@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 15046@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 31237@hnd16 MPPDYNA_971 Tue Apr 9 21:53 6
dscronin 31313@hnd16 MPPDYNA_971 Tue Apr 9 21:54 6
dscronin 6396@hnd15 MPPDYNA_971 Tue Apr 9 21:54 6
csharson 28890@saw175.saw.sharc MPPDYNA_971 Wed Apr 10 16:48 6
No programs queued
</source>

To show license file expiration details, append "-R" to the command:

<source lang="bash">
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet

[roberpj@hnd19:~] lstc_qrun -R
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
**** LICENSE INFORMATION ****
PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE
---------------- ---------- ----- ------ ------ | -----
LS-DYNA_971 12/31/2013 - 966 1024 | 0
dgierczy 20277@saw61.saw.sharcnet 1
dgierczy 8570@saw32.saw.sharcnet 1
MPPDYNA_971 12/31/2013 - 966 1024 | 0
dscronin 25486@hnd6 6
dscronin 14897@hnd18 6
dscronin 14971@hnd18 6
dscronin 15046@hnd18 6
dscronin 31237@hnd16 6
dscronin 31313@hnd16 6
dscronin 6396@hnd15 6
csharson 28890@saw175.saw.sharcnet 6
LICENSE GROUP 58 966 1024 | 0

PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE
---------------- ---------- ----- ------ ------ | -----
LS-OPT 12/31/2013 0 1024 1024 | 0
LICENSE GROUP 0 1024 1024 | 0
</source>

===Killling A Program===

The queue normally kills lsdyna jobs cleanly. However its possible that licences for a job (which is no longer running in the queue and therefore no longer has any processes running on the cluster) will continue to be tied up according to the lstc_qrun command. To kill such a program determine the pid@hostname from the lstc_qrun command then run the following kill command. The following example demonstrates the procedure for username roberpj:

<pre>
[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj
roberpj 45312@orc329.orca.shar MPPDYNA Wed Dec 5 15:50 12

[roberpj@orc-login2:~] lstc_qkill 45312@orc329.orca.sharcnet.ca
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
Program queued for termination

[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj
[roberpj@orc-login2:~]
</pre>

==Legacy Instructions==

The following binaries remain available on saw and orca for backward compatibility testing:

<pre>
[roberpj@orc-login2:~] cd /opt/sharcnet/local/lsdyna
[roberpj@orc129:/opt/sharcnet/local/lsdyna] ls ls971*
ls971_d_R3_1 ls971_d_R4_2_1 ls971_s_R3_1 ls971_s_R4_2_1 ls971_s_R5_1_1
ls971_d_R4_2_0 ls971_d_R5_0 ls971_s_R4_2_0 ls971_s_R5_0
</pre>

There are currently no sharcnet modules for these versions, hence jobs should be submitted as follows:

<pre>
module load lsdyna
export PATH=/opt/sharcnet/local/lsdyna:$PATH
export LSTC_LICENSE_SERVER=XXXXX@license3.uwo.sharcnet
cp /opt/sharcnet/local/lsdyna/examples/airbag.deploy.k airbag.deploy.k
SERIAL JOB: sqsub -q serial -r 1d -o ofile.%J ls971_d_R4_2_1 i=airbag.deploy.k
THREADED JOB: sqsub -q threaded -n 4 -r 1d -o ofile.%J ls971_d_R4_2_1 ncpu=4 para=2 i=airbag.deploy.k
</pre>

Note! the ls971_s_R3_1 and ls971_d_R3_1 binaries do not work, a fix is being looked for.

=References=

o LSTC LS-DYNA Homepage 
http://www.lstc.com/products/ls-dyna

o LSTC LS-DYNA Support (Tutorials, HowTos, Faq, Manuals, Release Notes, News, Links) 
http://www.dynasupport.com/

o LS-DYNA Release Notes 
http://www.dynasupport.com/release-notes

o LS-PrePost Online Documentation FAQ 
http://www.lstc.com/lspp/content/faq.shtml

o LSTC Download/Install Overview Page 
Provides links to binaries for LS-DYNA SMP/MPP, LS-OPT, LS-PREPOST, LS-TASC. 
http://www.lstc.com/download

==Memory Links==

o Implicit: Memory notes 
http://www.dynasupport.com/howtos/implicit/implicit-memory-notes

o LS-DYNA Support Environment variables 
http://www.dynasupport.com/howtos/general/environment-variables

o LS-DYNA and d3VIEW Blog (a few words on memory settings) 
http://blog2.d3view.com/a-few-words-on-memory-settings-in-ls-dyna/

o Convert Words to GB ie) memory=500MW (3.73GB) 
http://deviceanalytics.com/memcalc.php

LSDYNA

2019-02-08T20:14:03Z

Roberpj: /* Graham Cluster */

{{Software
|package_name=LSDYNA
|package_description=Suite of programs for transient dynamic finite element program
|package_idnumber=104
}}

=Introduction=

Before a research group can use LSDYNA on sharcnet, a license must be purchased directly from LSTC for the sharcnet license server. Alternately, if a research group resides at a institution that has sharcnet computing hardware (mac, uwo, guelph, waterloo) it maybe possible to use a pre-existing site license hosted on an accessible institutional license server. To access and use this software you must open a ticket and request to be added to the sharcnet lsdyna group.

=Version Selection=

==Graham Cluster==

===License===

Create and customize this file for your license server, where XXXXX should be replaced with your port number and Y with your server number:

[roberpj@gra1:~] cat .licenses/ls-dyna.lic
#LICENSE_TYPE: network
#LICENSE_SERVER: XXXXX@licenseY.uwo.sharcnet

===Module===

The modules are loaded automatically in the testomp.sh and testmpi.sh script shown below in the example section:

For single node smp jobs the available versions can be found by doing:

[roberpj@gra-login2:~] module spider ls-dyna
ls-dyna/7.1.2
ls-dyna/7.1.3
ls-dyna/8.1
ls-dyna/9.1
ls-dyna/9.2
ls-dyna/10.0

To load any version do for example:

module load ls-dyna/9.1

For multi node mpp jobs the available versions can be found by doing:

[roberpj@gra-login2:~] module spider ls-dyna-mpi
ls-dyna-mpi/7.1.2
ls-dyna-mpi/7.1.3
ls-dyna-mpi/8.1
ls-dyna-mpi/9.1
ls-dyna-mpi/9.2
ls-dyna-mpi/10.0

To load the 7.1.X modules do the following:
module load openmpi/1.6.5 ls-dyna-mpi/7.1.3

To load the 8.X or 9.X modules do:
module load openmpi/1.8.8 ls-dyna-mpi/9.2

To load the 10.X modules do:
module load openmpi/1.10.7 ls-dyna-mpi/10.0

==Legacy Clusters==

===License===

Once a license configuration is established, the research group will be given a 5 digit port number. The value should then be inserted into the appropriate departmental export statement before loading the module file as follows:

o UofT Mechanical Engineering Dept
export LSTC_LICENSE_SERVER=Port@license1.uwo.sharcnet

o McGill Mechanical Engineering Department
export LSTC_LICENSE_SERVER=Port@license2.uwo.sharcnet

o UW Mechanical and Mechatronics Engineering Dept
export LSTC_LICENSE_SERVER=Port@license3.uwo.sharcnet

o Laurentian University Bharti School of Engineering
export LSTC_LICENSE_SERVER=Port@license4.uwo.sharcnet

o Fictitious Example:

export LSTC_LICENSE_SERVER=12345@license1.uwo.sharcnet

===Module===

The next step is load the desired sharcnet lsdyna module version. Check which modules are available by running the module avail command then load one the modules as shown:

<source lang="bash">
[roberpj@orc-login2:~] module avail lsdyna
------------------------------------ /opt/sharcnet/modules ------------------------------------
lsdyna/hyb/r711.88920 lsdyna/mpp/r711.88920 lsdyna/smp/r611.79036
lsdyna/mpp/ls971.r85718 lsdyna/mpp/r712.95028 lsdyna/smp/r711.88920
lsdyna/mpp/ls980B1.011113 lsdyna/mpp/r800.95359 lsdyna/smp/r712.95028
lsdyna/mpp/r611.79036 lsdyna/mpp/r901.109912 lsdyna/smp/r800.95359
lsdyna/mpp/r611.80542 lsdyna/smp/ls980B1.78258 lsdyna/smp/r901.109912
</source>

====r9.0.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r901.109912</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna mkl
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r901.109912
</pre>

====r8.0.0 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r800.95359</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna mkl
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r800.95359
</pre>

====r7.1.2 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r712.95028</pre>

o For mpi jobs:
<pre>module unload intel openmpi lsdyna
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
</pre>

====r7.1.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r711.88920</pre>

o For mpi jobs:
<pre>module load lsdyna/mpp/r711.88920</pre>

====r6.1.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r611.79036</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna

module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036
or
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.80542
</pre>

where r611.80542 provides lsdyna_s only. 

====ls980 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/ls980B1.78258</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna

module load intel/12.1.3 openmpi/intel/1.4.5 lsdyna/mpp/ls980B1.011113
or
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/smp/ls980B1.78258
</pre>

Note1) restart capability for ls980smpB1 and ls980mppB1 is not supported
Note2) the module using legacy openmpi/intel/1.4.5 will run extremely slow

=Job Submission=

==Graham Cluster==

The submission scripts myompjob.sh and mympijob.sh for the airbag problem are shown in the Example Job section below, for both graham and orca. Please note that you should specify your own username if submitting to a def account (not roberpj). Alternatively you could specify your resource allocation account:

Sample threaded job submit script:

sbatch myompjob.sh

Sample mpi job submit script:

sbatch mympijob.sh

==Legacy Clusters==

When loading a lsdyna sharcnet legacy module on the new orca or a sharcnet legacy system, the single or double precision solvers are specified with lsdyna_s or lsdyna_d respectively, as shown in the following sqsub commands:

===1cpu SERIAL Job===
<pre>sqsub -r 1h -q serial -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=1</pre>

===4cpu SMP Job===
<pre>sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=1G lsdyna_s i=airbag.deploy.k ncpu=4</pre>

If using an explicit solver, one can specify a conservative initial memory setting on the command line as follows:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=4 memory=754414</pre>
where memory is the minimum number of 8 byte words shared by all processors in double precision, or, 4 byte words in single precision.

The initial value can be determined by starting a simulation interactively on the command line, and finding the output resembling: Memory required to begin solution : 754414. The number of words can be specified as memory=260M instead of memory=260000000, for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/.

===8cpu MPI Job===

<pre>sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k</pre>

The initial memory can also be specified for mpi jobs on the sqsub command line with "memory=" for the first master processor to decompose the problem and "memory2=" used on all processors including the master to solves the decomposed problem, where the values are specified as 4 bytes per word in single precision and 8 bytes per word in double precision. The number of words can be specified as memory=260M instead of memory=260000000 OR memory2=260M instead of memory2=260000000 for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/. The initial values can be found by running simulation interactively on a orca compute node and checking the output for a line such as: Memory required to begin solution (memory= 464898 memory2= 158794 ) which could then be implemented for a job run in the queue by doing:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898 memory2=158794
</pre>

The specification of memory2 on sharcnet or compute canada clusters is not beneficial since the queue reserves the same memory per core for all nodes, such that the decomposition master node process cannot be allocated to have larger system memory. Therefore its sufficient to specify a single memory parameter by doing:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898
</pre>

where the LSTC_MEMORY variable will only allow the memory to grow for explicit simulations. The following slurm examples demonstrate how to prescribe the memory parameters exactly.

=Example Job=

Copy the airbag example to your account:

rsync -av orca.computecanada.ca:/opt/sharcnet/lsdyna/r901.109912/examples r901.109912_examples
cd r901.109912_examples/examples/misc/airbag
gunzip airbag.deploy.k.gz

==Graham (default modules)==

Please note that graham does not have the sharcnet legacy modules installed on it.

===Threaded Job ===

Sample submission script mysmpjob1.sh for 4 core single precision smp job using compute canada default cvmfs modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --nodes=1 # use one node
#SBATCH --cpus-per-task=4 # number threads
#SBATCH --mem=4000M # total memory
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=testsmp1 # %x = jobname
#SBATCH --output=slurm-%x-%N-%j.out
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# ----------------------------------------------------------
# IMPORTANT: to set license server do either:
# 1) export LSTC_LICENSE_SERVER=port@some.license.server
# 2) create ~/.licenses/ls-dyna.lic as shown above
# ----------------------------------------------------------
module load ls-dyna/9.1
ls-dyna_s i=airbag.deploy.k ncpu=4 memory=1000M
</pre>

where memory = 4000M / 4 (bytes/word) = 1000M words

===Mpi Job ===

Sample submission script mympijob2.sh for 4 core double precision mpp job using compute canada default cvmfs modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --ntasks=4 # number of ranks
#SBATCH --mem-per-cpu=4000M # memory per rank
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=testmpi2 # %x = jobname
#SBATCH --output=slurm-%x-%N-%j.out
# ----------------------------------------------------------
# IMPORTANT: to set license server do either:
# 1) export LSTC_LICENSE_SERVER=port@some.license.server
# 2) create ~/.licenses/ls-dyna.lic as shown above
# ----------------------------------------------------------
module load openmpi/1.8.8 ls-dyna-mpi/9.1
srun ls-dyna_d i=airbag.deploy.k memory=500M memory2=500M
</pre>

where memory = 4000M / 8 (bytes/word) = 500M words

==New Orca (legacy modules)==

Please note, while new orca has most compute canada cvmfs modules available by default it does not have the graham the ls-dyna or ls-dyna-mpi modules installed. Therefore currently the only way to run lsdyna is by using the legacy sharcnet lsdyna modules as shown in the following two smp and mpi examples.

===Threaded Job ===

Submission script mysmpjob3.sh to run 16 core single precision single node smp job with sharcnet legacy modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --nodes=1 # use one node
#SBATCH --cpus-per-task=16 # number threads
#SBATCH --mem=4000M # total memory
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=mysmpjob3-1node-16core # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# ----------------------------------------------------------------
# IMPORTANT: to set license server for legacy modules:
export LSTC_LICENSE_SERVER=31047@license3.sharcnet.ca
# since ~/.licenses/ls-dyna.lic does NOT work with legacy modules
# ----------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load lsdyna/smp/r712.95028
lsdyna_s i=airbag.deploy.k ncpu=$SLURM_CPUS_PER_TASK memory=500M
</pre>

where memory = 4000M / 8 (bytes/word) = 500M words

===MPI Job ===

To run the sharcnet legacy module lsdyna/mpp/r712.95028 efficiently over several compute nodes with many cores requires using orca's xeon nodes as shown in mympijob4.sh below. To run lsdyna on orca's opteron nodes with module lsdyna/mpp/r712.95028 requires using only one opteron node as shown in mympijob5.sh below. The other legacy sharcnet lsdyna mpp modules, as shown in the Availability Table lsdyna/mpp/ls971.r85718, lsdyna/mpp/ls980B1.011113, lsdyna/mpp/r611.79036, lsdyna/mpp/r611.80542, lsdyna/mpp/r711.88920, lsdyna/mpp/r800.95359 and lsdyna/mpp/r901.109912 have yet to be tested. If you want to use one but cannot get it working please open a ticket. Note that lines containing a space between "# and SBATCH" are comments. To activate such line(s) required removal of the space ie) "#SBATCH". Its recommended to use module ls-dyna-mpi on graham instead since the machine is far newer, faster and reliable.

====Multi Node Slurm Script====

<pre>
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob4.sh
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --constraint=xeon
#SBATCH --ntasks=48
#SBATCH --mem-per-cpu=4000M
#SBATCH --time=00:30:00
#SBATCH --job-name=mympijob4-Xnodes-48cores-xeon # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
# -------------------------------------------------------------
# To set license server when using legacy sharcnet modules:
export LSTC_LICENSE_SERVER=port@some.license.server
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.
# -------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`
mpirun --host $MACHINEFILE --mca plm_slurm_args '--cpu_bind=none' lsdyna_s i=airbag.deploy.k memory=250M memory2=250M
</pre>

====Single Node Slurm Script====

<pre>
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob5.sh
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --constraint=opteron
#SBATCH --nodes=1
# SBATCH --ntasks=24
# SBATCH --mem=0
#SBATCH --ntasks=16
#SBATCH --mem-per-cpu=1000M
#SBATCH --time=00:30:00
#SBATCH --job-name=mympijob5-1node-16cores-opteron # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
# -------------------------------------------------------------
# To set license server when using legacy sharcnet modules:
export LSTC_LICENSE_SERVER=port@some.license.server
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.
# -------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`
mpirun --host $MACHINEFILE lsdyna_s i=airbag.deploy.k memory=250M memory2=250M
</pre>

where memory = 1000M / 4 (bytes/word) = 250M words

==Legacy Clusters==

STEP1) The following shows sqsub submission of the [http://www.dynaexamples.com/examples-manual/misc/airbag airbag] example to the mpi queue. Its recommended to first edit airbag.deploy.k and change endtim to 3.000E-00 so the job runs long enough to perform the restart in steps 2 and 3 below:
<pre>
cp -a /opt/sharcnet/lsdyna/r611.79036/examples /scratch/$USER/test-lsdyna
cd /scratch/$USER/test-lsdyna/misc/airbag
gunzip airbag.deploy.k.gz
cp airbag.deploy.k airbag.deploy.restart.k
nano deploy.restart.k (reduce plot file creation frequency)
*DATABASE_BINARY_D3PLOT
$ dt lcdtdeploy.restart.k
1.000E-01 <--- change from 5.000E-04
export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet
module unload intel openmpi lsdyna
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036
sqsub -r 10m -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.k
</pre>

STEP2) With the job still running, use the echo command as follows to create a file called "D3KIL" that will trigger generation of restart files at which point the file D3KIL itself will be erased. Do this once a day if data loss is critical to you OR once or twice just before the sqsub -r time limit is reached. Further information can be found here http://www.dynasupport.com/tutorial/ls-dyna-users-guide/sense-switch-control:

<pre>
echo "sw3" > D3KIL
sqkill job#
</pre>

STEP3) Before the job can be restarted, the following two lines must be added to the airbag.deploy.restart.k file:

<pre>
*CHANGE_CURVE_DEFINITION
1
</pre>

STEP4) Now resubmit the job as follows using "r=" to specify the restart file:

<pre>
sqsub -r 1h -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.restart.k r=d3dump01
</pre>

=General Notes=

==Graham Cluster==

There are no special notes at the present time in regards to using lsdyna on graham.

==Command Line Use==

Parallel lsdyna jobs can be run interactively on the command line (outside the queue) for short run testing purposes as follows:

o Orca Development Node MPP (mpi) Example

ssh orca.sharcnet.ca
ssh orc-dev1 (or dev2,3,4)
module unload intel openmpi lsdyna
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/mpp/r711.88920
mpirun -np 8 lsdyna_d i=airbag.deploy.k

o Orca Development Node SMP (threaded) Example

ssh orca.sharcnet.ca
ssh orc-dev1 (or dev2,3,4)
module unload intel openmpi lsdyna
module load lsdyna/smp/r711.88920
lsdyna_d i=airbag.deploy.k ncpu=8

==Memory Issues==

A minumum of mpp=2G is recommended although the memory requirement of a job may suggest much less is required. For instance setting mpp=1G for the airbag test job above will result in the following error when running a job in the queue:

<pre>
--------------------------------------------------------------------------
An attempt to set processor affinity has failed - please check to
ensure that your system supports such functionality. If so, then
this is probably something that should be reported to the OMPI developers.
--------------------------------------------------------------------------
</pre>

Another error message which can occur after a job runs for a while if mpp is chosen to small is:

<pre>
>>>>> Process 45 <<<<<
>>>>> Signal 11 : Segmentation Violation <<<<<
</pre>

To get an idea of the amount of memory a job used run grep on the output file:

<pre>
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep Memory
| Distributed Memory Parallel |
Memory size from command line: 500000, 500000
Memory for the head node
Memory installed (MB) : 32237
Memory free (MB) : 11259
Memory required (MB) : 0
Memory required to process keyword : 458120
Memory required for decomposition : 458120
Memory required to begin solution (memory= 458120 memory2= 230721)
Max. Memory reqd for implicit sol: max used 0
Max. Memory reqd for implicit sol: incore 0
Max. Memory reqd for implicit sol: oocore 0
Memory required to complete solution (memory= 458120 memory2= 230721)
</pre>

==Version and Revision==

Again run grep on the output file to extract the major and minor revision:

<pre>
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep 'Revision\|Version'
| Version : mpp s R6.1.1 Date: 01/02/2013 |
| Revision: 78769 Time: 07:43:30 |
| SVN Version: 80542 |
</pre>

==Check License Status==

===Running and Queued Programs===

To get a summary of running and/or queued jobs use the lstc_qrun command as follows, where queued means the job has started running according to the sqsub command but its actually sitting waiting for license resources to come available. Once these are acquired the job will start running on the cluster and appear as a Running Program according to lstc_qrun ie)

<source lang="bash">
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet

[roberpj@hnd19:~] lstc_qrun
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
Running Programs
User Host Program Started # procs
-----------------------------------------------------------------------------
dgierczy 20277@saw61.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1
dgierczy 8570@saw32.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1
dscronin 25486@hnd6 MPPDYNA_971 Tue Apr 9 20:19 6
dscronin 14897@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 14971@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 15046@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 31237@hnd16 MPPDYNA_971 Tue Apr 9 21:53 6
dscronin 31313@hnd16 MPPDYNA_971 Tue Apr 9 21:54 6
dscronin 6396@hnd15 MPPDYNA_971 Tue Apr 9 21:54 6
csharson 28890@saw175.saw.sharc MPPDYNA_971 Wed Apr 10 16:48 6
No programs queued
</source>

To show license file expiration details, append "-R" to the command:

<source lang="bash">
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet

[roberpj@hnd19:~] lstc_qrun -R
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
**** LICENSE INFORMATION ****
PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE
---------------- ---------- ----- ------ ------ | -----
LS-DYNA_971 12/31/2013 - 966 1024 | 0
dgierczy 20277@saw61.saw.sharcnet 1
dgierczy 8570@saw32.saw.sharcnet 1
MPPDYNA_971 12/31/2013 - 966 1024 | 0
dscronin 25486@hnd6 6
dscronin 14897@hnd18 6
dscronin 14971@hnd18 6
dscronin 15046@hnd18 6
dscronin 31237@hnd16 6
dscronin 31313@hnd16 6
dscronin 6396@hnd15 6
csharson 28890@saw175.saw.sharcnet 6
LICENSE GROUP 58 966 1024 | 0

PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE
---------------- ---------- ----- ------ ------ | -----
LS-OPT 12/31/2013 0 1024 1024 | 0
LICENSE GROUP 0 1024 1024 | 0
</source>

===Killling A Program===

The queue normally kills lsdyna jobs cleanly. However its possible that licences for a job (which is no longer running in the queue and therefore no longer has any processes running on the cluster) will continue to be tied up according to the lstc_qrun command. To kill such a program determine the pid@hostname from the lstc_qrun command then run the following kill command. The following example demonstrates the procedure for username roberpj:

<pre>
[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj
roberpj 45312@orc329.orca.shar MPPDYNA Wed Dec 5 15:50 12

[roberpj@orc-login2:~] lstc_qkill 45312@orc329.orca.sharcnet.ca
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
Program queued for termination

[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj
[roberpj@orc-login2:~]
</pre>

==Legacy Instructions==

The following binaries remain available on saw and orca for backward compatibility testing:

<pre>
[roberpj@orc-login2:~] cd /opt/sharcnet/local/lsdyna
[roberpj@orc129:/opt/sharcnet/local/lsdyna] ls ls971*
ls971_d_R3_1 ls971_d_R4_2_1 ls971_s_R3_1 ls971_s_R4_2_1 ls971_s_R5_1_1
ls971_d_R4_2_0 ls971_d_R5_0 ls971_s_R4_2_0 ls971_s_R5_0
</pre>

There are currently no sharcnet modules for these versions, hence jobs should be submitted as follows:

<pre>
module load lsdyna
export PATH=/opt/sharcnet/local/lsdyna:$PATH
export LSTC_LICENSE_SERVER=XXXXX@license3.uwo.sharcnet
cp /opt/sharcnet/local/lsdyna/examples/airbag.deploy.k airbag.deploy.k
SERIAL JOB: sqsub -q serial -r 1d -o ofile.%J ls971_d_R4_2_1 i=airbag.deploy.k
THREADED JOB: sqsub -q threaded -n 4 -r 1d -o ofile.%J ls971_d_R4_2_1 ncpu=4 para=2 i=airbag.deploy.k
</pre>

Note! the ls971_s_R3_1 and ls971_d_R3_1 binaries do not work, a fix is being looked for.

=References=

o LSTC LS-DYNA Homepage 
http://www.lstc.com/products/ls-dyna

o LSTC LS-DYNA Support (Tutorials, HowTos, Faq, Manuals, Release Notes, News, Links) 
http://www.dynasupport.com/

o LS-DYNA Release Notes 
http://www.dynasupport.com/release-notes

o LS-PrePost Online Documentation FAQ 
http://www.lstc.com/lspp/content/faq.shtml

o LSTC Download/Install Overview Page 
Provides links to binaries for LS-DYNA SMP/MPP, LS-OPT, LS-PREPOST, LS-TASC. 
http://www.lstc.com/download

==Memory Links==

o Implicit: Memory notes 
http://www.dynasupport.com/howtos/implicit/implicit-memory-notes

o LS-DYNA Support Environment variables 
http://www.dynasupport.com/howtos/general/environment-variables

o LS-DYNA and d3VIEW Blog (a few words on memory settings) 
http://blog2.d3view.com/a-few-words-on-memory-settings-in-ls-dyna/

o Convert Words to GB ie) memory=500MW (3.73GB) 
http://deviceanalytics.com/memcalc.php

LSDYNA

2019-02-08T20:13:27Z

Roberpj: /* Graham Cluster */

{{Software
|package_name=LSDYNA
|package_description=Suite of programs for transient dynamic finite element program
|package_idnumber=104
}}

=Introduction=

Before a research group can use LSDYNA on sharcnet, a license must be purchased directly from LSTC for the sharcnet license server. Alternately, if a research group resides at a institution that has sharcnet computing hardware (mac, uwo, guelph, waterloo) it maybe possible to use a pre-existing site license hosted on an accessible institutional license server. To access and use this software you must open a ticket and request to be added to the sharcnet lsdyna group.

=Version Selection=

==Graham Cluster==

===License===

Create and customize this file for your license server, where XXXXX should be replaced with your port number and Y with your server number:

[roberpj@gra1:~] cat .licenses/ls-dyna.lic
#LICENSE_TYPE: network
#LICENSE_SERVER: XXXXX@licenseY.uwo.sharcnet

===Module===

The modules are loaded automatically in the testomp.sh and testmpi.sh script shown below in the example section:

For single node smp jobs the available versions can be found by doing:

[roberpj@gra-login2:~] module spider ls-dyna
ls-dyna/7.1.2
ls-dyna/7.1.3
ls-dyna/8.1
ls-dyna/9.1
ls-dyna/9.2
ls-dyna/10.0

To load any version do for example:

module load ls-dyna/9.1

For multi node mpp jobs the available versions can be found by doing:

[roberpj@gra-login2:~] module spider ls-dyna-mpi
ls-dyna-mpi/7.1.2
ls-dyna-mpi/7.1.3
ls-dyna-mpi/8.1
ls-dyna-mpi/9.1
ls-dyna-mpi/9.2
ls-dyna-mpi/10.0

To load the 7.1.X modules do the following:
module load openmpi/1.6.5 ls-dyna-mpi/7.1.3

To load the 8.X or 9.X modules do:
module load openmpi/1.8.8 ls-dyna-mpi/9.2

To load the 10.X modules do:
module load openmpi/1.10.7 ls-dyna-mpi/10.0

==Legacy Clusters==

===License===

Once a license configuration is established, the research group will be given a 5 digit port number. The value should then be inserted into the appropriate departmental export statement before loading the module file as follows:

o UofT Mechanical Engineering Dept
export LSTC_LICENSE_SERVER=Port@license1.uwo.sharcnet

o McGill Mechanical Engineering Department
export LSTC_LICENSE_SERVER=Port@license2.uwo.sharcnet

o UW Mechanical and Mechatronics Engineering Dept
export LSTC_LICENSE_SERVER=Port@license3.uwo.sharcnet

o Laurentian University Bharti School of Engineering
export LSTC_LICENSE_SERVER=Port@license4.uwo.sharcnet

o Fictitious Example:

export LSTC_LICENSE_SERVER=12345@license1.uwo.sharcnet

===Module===

The next step is load the desired sharcnet lsdyna module version. Check which modules are available by running the module avail command then load one the modules as shown:

<source lang="bash">
[roberpj@orc-login2:~] module avail lsdyna
------------------------------------ /opt/sharcnet/modules ------------------------------------
lsdyna/hyb/r711.88920 lsdyna/mpp/r711.88920 lsdyna/smp/r611.79036
lsdyna/mpp/ls971.r85718 lsdyna/mpp/r712.95028 lsdyna/smp/r711.88920
lsdyna/mpp/ls980B1.011113 lsdyna/mpp/r800.95359 lsdyna/smp/r712.95028
lsdyna/mpp/r611.79036 lsdyna/mpp/r901.109912 lsdyna/smp/r800.95359
lsdyna/mpp/r611.80542 lsdyna/smp/ls980B1.78258 lsdyna/smp/r901.109912
</source>

====r9.0.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r901.109912</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna mkl
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r901.109912
</pre>

====r8.0.0 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r800.95359</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna mkl
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r800.95359
</pre>

====r7.1.2 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r712.95028</pre>

o For mpi jobs:
<pre>module unload intel openmpi lsdyna
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
</pre>

====r7.1.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r711.88920</pre>

o For mpi jobs:
<pre>module load lsdyna/mpp/r711.88920</pre>

====r6.1.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r611.79036</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna

module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036
or
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.80542
</pre>

where r611.80542 provides lsdyna_s only. 

====ls980 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/ls980B1.78258</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna

module load intel/12.1.3 openmpi/intel/1.4.5 lsdyna/mpp/ls980B1.011113
or
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/smp/ls980B1.78258
</pre>

Note1) restart capability for ls980smpB1 and ls980mppB1 is not supported
Note2) the module using legacy openmpi/intel/1.4.5 will run extremely slow

=Job Submission=

==Graham Cluster==

The submission scripts myompjob.sh and mympijob.sh for the airbag problem are shown in the Example Job section below, for both graham and orca. Please note that you should specify your own username if submitting to a def account (not roberpj). Alternatively you could specify your resource allocation account:

Sample threaded job submit script:

sbatch myompjob.sh

Sample mpi job submit script:

sbatch mympijob.sh

==Legacy Clusters==

When loading a lsdyna sharcnet legacy module on the new orca or a sharcnet legacy system, the single or double precision solvers are specified with lsdyna_s or lsdyna_d respectively, as shown in the following sqsub commands:

===1cpu SERIAL Job===
<pre>sqsub -r 1h -q serial -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=1</pre>

===4cpu SMP Job===
<pre>sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=1G lsdyna_s i=airbag.deploy.k ncpu=4</pre>

If using an explicit solver, one can specify a conservative initial memory setting on the command line as follows:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=4 memory=754414</pre>
where memory is the minimum number of 8 byte words shared by all processors in double precision, or, 4 byte words in single precision.

The initial value can be determined by starting a simulation interactively on the command line, and finding the output resembling: Memory required to begin solution : 754414. The number of words can be specified as memory=260M instead of memory=260000000, for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/.

===8cpu MPI Job===

<pre>sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k</pre>

The initial memory can also be specified for mpi jobs on the sqsub command line with "memory=" for the first master processor to decompose the problem and "memory2=" used on all processors including the master to solves the decomposed problem, where the values are specified as 4 bytes per word in single precision and 8 bytes per word in double precision. The number of words can be specified as memory=260M instead of memory=260000000 OR memory2=260M instead of memory2=260000000 for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/. The initial values can be found by running simulation interactively on a orca compute node and checking the output for a line such as: Memory required to begin solution (memory= 464898 memory2= 158794 ) which could then be implemented for a job run in the queue by doing:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898 memory2=158794
</pre>

The specification of memory2 on sharcnet or compute canada clusters is not beneficial since the queue reserves the same memory per core for all nodes, such that the decomposition master node process cannot be allocated to have larger system memory. Therefore its sufficient to specify a single memory parameter by doing:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898
</pre>

where the LSTC_MEMORY variable will only allow the memory to grow for explicit simulations. The following slurm examples demonstrate how to prescribe the memory parameters exactly.

=Example Job=

Copy the airbag example to your account:

rsync -av orca.computecanada.ca:/opt/sharcnet/lsdyna/r901.109912/examples r901.109912_examples
cd r901.109912_examples/examples/misc/airbag
gunzip airbag.deploy.k.gz

==Graham (default modules)==

Please note that graham does not have the sharcnet legacy modules installed on it.

===Threaded Job ===

Sample submission script mysmpjob1.sh for 4 core single precision smp job using compute canada default cvmfs modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --nodes=1 # use one node
#SBATCH --cpus-per-task=4 # number threads
#SBATCH --mem=4000M # total memory
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=testsmp1 # %x = jobname
#SBATCH --output=slurm-%x-%N-%j.out
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# ----------------------------------------------------------
# IMPORTANT: to set license server do either:
# 1) export LSTC_LICENSE_SERVER=port@some.license.server
# 2) create ~/.licenses/ls-dyna.lic as shown above
# ----------------------------------------------------------
module load ls-dyna/9.1
ls-dyna_s i=airbag.deploy.k ncpu=4 memory=1000M
</pre>

where memory = 4000M / 4 (bytes/word) = 1000M words

===Mpi Job ===

Sample submission script mympijob2.sh for 4 core double precision mpp job using compute canada default cvmfs modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --ntasks=4 # number of ranks
#SBATCH --mem-per-cpu=4000M # memory per rank
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=testmpi2 # %x = jobname
#SBATCH --output=slurm-%x-%N-%j.out
# ----------------------------------------------------------
# IMPORTANT: to set license server do either:
# 1) export LSTC_LICENSE_SERVER=port@some.license.server
# 2) create ~/.licenses/ls-dyna.lic as shown above
# ----------------------------------------------------------
module load openmpi/1.8.8 ls-dyna-mpi/9.1
srun ls-dyna_d i=airbag.deploy.k memory=500M memory2=500M
</pre>

where memory = 4000M / 8 (bytes/word) = 500M words

==New Orca (legacy modules)==

Please note, while new orca has most compute canada cvmfs modules available by default it does not have the graham the ls-dyna or ls-dyna-mpi modules installed. Therefore currently the only way to run lsdyna is by using the legacy sharcnet lsdyna modules as shown in the following two smp and mpi examples.

===Threaded Job ===

Submission script mysmpjob3.sh to run 16 core single precision single node smp job with sharcnet legacy modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --nodes=1 # use one node
#SBATCH --cpus-per-task=16 # number threads
#SBATCH --mem=4000M # total memory
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=mysmpjob3-1node-16core # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# ----------------------------------------------------------------
# IMPORTANT: to set license server for legacy modules:
export LSTC_LICENSE_SERVER=31047@license3.sharcnet.ca
# since ~/.licenses/ls-dyna.lic does NOT work with legacy modules
# ----------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load lsdyna/smp/r712.95028
lsdyna_s i=airbag.deploy.k ncpu=$SLURM_CPUS_PER_TASK memory=500M
</pre>

where memory = 4000M / 8 (bytes/word) = 500M words

===MPI Job ===

To run the sharcnet legacy module lsdyna/mpp/r712.95028 efficiently over several compute nodes with many cores requires using orca's xeon nodes as shown in mympijob4.sh below. To run lsdyna on orca's opteron nodes with module lsdyna/mpp/r712.95028 requires using only one opteron node as shown in mympijob5.sh below. The other legacy sharcnet lsdyna mpp modules, as shown in the Availability Table lsdyna/mpp/ls971.r85718, lsdyna/mpp/ls980B1.011113, lsdyna/mpp/r611.79036, lsdyna/mpp/r611.80542, lsdyna/mpp/r711.88920, lsdyna/mpp/r800.95359 and lsdyna/mpp/r901.109912 have yet to be tested. If you want to use one but cannot get it working please open a ticket. Note that lines containing a space between "# and SBATCH" are comments. To activate such line(s) required removal of the space ie) "#SBATCH". Its recommended to use module ls-dyna-mpi on graham instead since the machine is far newer, faster and reliable.

====Multi Node Slurm Script====

<pre>
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob4.sh
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --constraint=xeon
#SBATCH --ntasks=48
#SBATCH --mem-per-cpu=4000M
#SBATCH --time=00:30:00
#SBATCH --job-name=mympijob4-Xnodes-48cores-xeon # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
# -------------------------------------------------------------
# To set license server when using legacy sharcnet modules:
export LSTC_LICENSE_SERVER=port@some.license.server
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.
# -------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`
mpirun --host $MACHINEFILE --mca plm_slurm_args '--cpu_bind=none' lsdyna_s i=airbag.deploy.k memory=250M memory2=250M
</pre>

====Single Node Slurm Script====

<pre>
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob5.sh
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --constraint=opteron
#SBATCH --nodes=1
# SBATCH --ntasks=24
# SBATCH --mem=0
#SBATCH --ntasks=16
#SBATCH --mem-per-cpu=1000M
#SBATCH --time=00:30:00
#SBATCH --job-name=mympijob5-1node-16cores-opteron # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
# -------------------------------------------------------------
# To set license server when using legacy sharcnet modules:
export LSTC_LICENSE_SERVER=port@some.license.server
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.
# -------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`
mpirun --host $MACHINEFILE lsdyna_s i=airbag.deploy.k memory=250M memory2=250M
</pre>

where memory = 1000M / 4 (bytes/word) = 250M words

==Legacy Clusters==

STEP1) The following shows sqsub submission of the [http://www.dynaexamples.com/examples-manual/misc/airbag airbag] example to the mpi queue. Its recommended to first edit airbag.deploy.k and change endtim to 3.000E-00 so the job runs long enough to perform the restart in steps 2 and 3 below:
<pre>
cp -a /opt/sharcnet/lsdyna/r611.79036/examples /scratch/$USER/test-lsdyna
cd /scratch/$USER/test-lsdyna/misc/airbag
gunzip airbag.deploy.k.gz
cp airbag.deploy.k airbag.deploy.restart.k
nano deploy.restart.k (reduce plot file creation frequency)
*DATABASE_BINARY_D3PLOT
$ dt lcdtdeploy.restart.k
1.000E-01 <--- change from 5.000E-04
export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet
module unload intel openmpi lsdyna
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036
sqsub -r 10m -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.k
</pre>

STEP2) With the job still running, use the echo command as follows to create a file called "D3KIL" that will trigger generation of restart files at which point the file D3KIL itself will be erased. Do this once a day if data loss is critical to you OR once or twice just before the sqsub -r time limit is reached. Further information can be found here http://www.dynasupport.com/tutorial/ls-dyna-users-guide/sense-switch-control:

<pre>
echo "sw3" > D3KIL
sqkill job#
</pre>

STEP3) Before the job can be restarted, the following two lines must be added to the airbag.deploy.restart.k file:

<pre>
*CHANGE_CURVE_DEFINITION
1
</pre>

STEP4) Now resubmit the job as follows using "r=" to specify the restart file:

<pre>
sqsub -r 1h -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.restart.k r=d3dump01
</pre>

=General Notes=

==Graham Cluster==

There are no special notes at present regarding the usage of lsdyna on graham,

==Command Line Use==

Parallel lsdyna jobs can be run interactively on the command line (outside the queue) for short run testing purposes as follows:

o Orca Development Node MPP (mpi) Example

ssh orca.sharcnet.ca
ssh orc-dev1 (or dev2,3,4)
module unload intel openmpi lsdyna
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/mpp/r711.88920
mpirun -np 8 lsdyna_d i=airbag.deploy.k

o Orca Development Node SMP (threaded) Example

ssh orca.sharcnet.ca
ssh orc-dev1 (or dev2,3,4)
module unload intel openmpi lsdyna
module load lsdyna/smp/r711.88920
lsdyna_d i=airbag.deploy.k ncpu=8

==Memory Issues==

A minumum of mpp=2G is recommended although the memory requirement of a job may suggest much less is required. For instance setting mpp=1G for the airbag test job above will result in the following error when running a job in the queue:

<pre>
--------------------------------------------------------------------------
An attempt to set processor affinity has failed - please check to
ensure that your system supports such functionality. If so, then
this is probably something that should be reported to the OMPI developers.
--------------------------------------------------------------------------
</pre>

Another error message which can occur after a job runs for a while if mpp is chosen to small is:

<pre>
>>>>> Process 45 <<<<<
>>>>> Signal 11 : Segmentation Violation <<<<<
</pre>

To get an idea of the amount of memory a job used run grep on the output file:

<pre>
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep Memory
| Distributed Memory Parallel |
Memory size from command line: 500000, 500000
Memory for the head node
Memory installed (MB) : 32237
Memory free (MB) : 11259
Memory required (MB) : 0
Memory required to process keyword : 458120
Memory required for decomposition : 458120
Memory required to begin solution (memory= 458120 memory2= 230721)
Max. Memory reqd for implicit sol: max used 0
Max. Memory reqd for implicit sol: incore 0
Max. Memory reqd for implicit sol: oocore 0
Memory required to complete solution (memory= 458120 memory2= 230721)
</pre>

==Version and Revision==

Again run grep on the output file to extract the major and minor revision:

<pre>
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep 'Revision\|Version'
| Version : mpp s R6.1.1 Date: 01/02/2013 |
| Revision: 78769 Time: 07:43:30 |
| SVN Version: 80542 |
</pre>

==Check License Status==

===Running and Queued Programs===

To get a summary of running and/or queued jobs use the lstc_qrun command as follows, where queued means the job has started running according to the sqsub command but its actually sitting waiting for license resources to come available. Once these are acquired the job will start running on the cluster and appear as a Running Program according to lstc_qrun ie)

<source lang="bash">
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet

[roberpj@hnd19:~] lstc_qrun
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
Running Programs
User Host Program Started # procs
-----------------------------------------------------------------------------
dgierczy 20277@saw61.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1
dgierczy 8570@saw32.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1
dscronin 25486@hnd6 MPPDYNA_971 Tue Apr 9 20:19 6
dscronin 14897@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 14971@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 15046@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 31237@hnd16 MPPDYNA_971 Tue Apr 9 21:53 6
dscronin 31313@hnd16 MPPDYNA_971 Tue Apr 9 21:54 6
dscronin 6396@hnd15 MPPDYNA_971 Tue Apr 9 21:54 6
csharson 28890@saw175.saw.sharc MPPDYNA_971 Wed Apr 10 16:48 6
No programs queued
</source>

To show license file expiration details, append "-R" to the command:

<source lang="bash">
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet

[roberpj@hnd19:~] lstc_qrun -R
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
**** LICENSE INFORMATION ****
PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE
---------------- ---------- ----- ------ ------ | -----
LS-DYNA_971 12/31/2013 - 966 1024 | 0
dgierczy 20277@saw61.saw.sharcnet 1
dgierczy 8570@saw32.saw.sharcnet 1
MPPDYNA_971 12/31/2013 - 966 1024 | 0
dscronin 25486@hnd6 6
dscronin 14897@hnd18 6
dscronin 14971@hnd18 6
dscronin 15046@hnd18 6
dscronin 31237@hnd16 6
dscronin 31313@hnd16 6
dscronin 6396@hnd15 6
csharson 28890@saw175.saw.sharcnet 6
LICENSE GROUP 58 966 1024 | 0

PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE
---------------- ---------- ----- ------ ------ | -----
LS-OPT 12/31/2013 0 1024 1024 | 0
LICENSE GROUP 0 1024 1024 | 0
</source>

===Killling A Program===

The queue normally kills lsdyna jobs cleanly. However its possible that licences for a job (which is no longer running in the queue and therefore no longer has any processes running on the cluster) will continue to be tied up according to the lstc_qrun command. To kill such a program determine the pid@hostname from the lstc_qrun command then run the following kill command. The following example demonstrates the procedure for username roberpj:

<pre>
[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj
roberpj 45312@orc329.orca.shar MPPDYNA Wed Dec 5 15:50 12

[roberpj@orc-login2:~] lstc_qkill 45312@orc329.orca.sharcnet.ca
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
Program queued for termination

[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj
[roberpj@orc-login2:~]
</pre>

==Legacy Instructions==

The following binaries remain available on saw and orca for backward compatibility testing:

<pre>
[roberpj@orc-login2:~] cd /opt/sharcnet/local/lsdyna
[roberpj@orc129:/opt/sharcnet/local/lsdyna] ls ls971*
ls971_d_R3_1 ls971_d_R4_2_1 ls971_s_R3_1 ls971_s_R4_2_1 ls971_s_R5_1_1
ls971_d_R4_2_0 ls971_d_R5_0 ls971_s_R4_2_0 ls971_s_R5_0
</pre>

There are currently no sharcnet modules for these versions, hence jobs should be submitted as follows:

<pre>
module load lsdyna
export PATH=/opt/sharcnet/local/lsdyna:$PATH
export LSTC_LICENSE_SERVER=XXXXX@license3.uwo.sharcnet
cp /opt/sharcnet/local/lsdyna/examples/airbag.deploy.k airbag.deploy.k
SERIAL JOB: sqsub -q serial -r 1d -o ofile.%J ls971_d_R4_2_1 i=airbag.deploy.k
THREADED JOB: sqsub -q threaded -n 4 -r 1d -o ofile.%J ls971_d_R4_2_1 ncpu=4 para=2 i=airbag.deploy.k
</pre>

Note! the ls971_s_R3_1 and ls971_d_R3_1 binaries do not work, a fix is being looked for.

=References=

o LSTC LS-DYNA Homepage 
http://www.lstc.com/products/ls-dyna

o LSTC LS-DYNA Support (Tutorials, HowTos, Faq, Manuals, Release Notes, News, Links) 
http://www.dynasupport.com/

o LS-DYNA Release Notes 
http://www.dynasupport.com/release-notes

o LS-PrePost Online Documentation FAQ 
http://www.lstc.com/lspp/content/faq.shtml

o LSTC Download/Install Overview Page 
Provides links to binaries for LS-DYNA SMP/MPP, LS-OPT, LS-PREPOST, LS-TASC. 
http://www.lstc.com/download

==Memory Links==

o Implicit: Memory notes 
http://www.dynasupport.com/howtos/implicit/implicit-memory-notes

o LS-DYNA Support Environment variables 
http://www.dynasupport.com/howtos/general/environment-variables

o LS-DYNA and d3VIEW Blog (a few words on memory settings) 
http://blog2.d3view.com/a-few-words-on-memory-settings-in-ls-dyna/

o Convert Words to GB ie) memory=500MW (3.73GB) 
http://deviceanalytics.com/memcalc.php

NIX

2018-12-13T01:12:21Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

== With sqsub ==

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>

= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) For future logins, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

The above steps can also be followed to start over if a previously installed nix package is not showing up in your path as expected.

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:11:59Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

== With sqsub ==

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>

= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) For future logins, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

The above steps can also be followed to start over if a previously installed nix package is not showing up in your path as expected, after loading the nix module.

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:11:07Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

== With sqsub ==

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>

= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) For future logins, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

The above example can also be followed to start over if a previously installed nix package is not showing up in your path as expected.

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:10:39Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

== With sqsub ==

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>

= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) For future logins, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

TIP: The above example can also be followed to reset your environment if a previously installed nix package is not showing up in your path as expected.

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:10:01Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

== With sqsub ==

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>

= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

This example can also be followed to reset your environment if a previously installed nix package is not showing up in your path as expected.

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) For future logins, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:08:06Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

== With sqsub ==

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>

= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

This example can also be followed to reset your environment if a previously installed nix package is not showing up in your path as expected.

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:07:53Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

== With sqsub ==

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>

= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

This example can also be follows if a package to reset your environment if a previously installed nix package is not showing up in your path as expected.

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:06:55Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

== With sqsub ==

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>

= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

This example can also be follows if a package is not showing up in your path as expected ...

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:06:28Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

== With sqsub ==

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>

= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

New users can jump directly to step 2. This example is also worth following if a package is not showing up in your path as expected.

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:05:12Z

Roberpj: /* With sqsub = */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

== With sqsub ==

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>

= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:05:04Z

Roberpj: /* Submitting Jobs */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

== With sqsub ===

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>

= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:04:31Z

Roberpj: /* Examples */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T01:04:03Z

Roberpj:

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:21:49Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:21:11Z

Roberpj: /* Example(s) */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:21:00Z

Roberpj: /* Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Example(s) =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:20:33Z

Roberpj: /* Examples */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Example =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:16:48Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users :
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:16:09Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:13:32Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) In the future, after you login simply load the nix module and atop will be in your path ;)
[roberpj@laptop:~] ssh graham.sharcnet.ca
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:12:26Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a (((ONE TIME))) required step for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) Next time after logging in, simply load the nix module and atop will be in your path ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:11:42Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a ONE TIME required step for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) Next time after logging in, simply load the nix module and atop will be in your path ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:10:51Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a ONE TIME required step for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) Next time you login, simply load the nix module and atop will be in your path ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:10:13Z

Roberpj: /* Installing a package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a ONE TIME required step for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Next install package "atop" or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) Next time you login, simply load the nix module to use atop ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:09:30Z

Roberpj: /* Installing a Package as a new user */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a ONE TIME required step for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Finally install atop, or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) Next time you login, simply load the nix module to use atop ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:09:21Z

Roberpj: /* Installing A Package As A New User */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing a Package as a new user ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a ONE TIME required step for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Finally install atop, or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) Next time you login, simply load the nix module to use atop ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:06:39Z

Roberpj: /* Installing A Package As A New User */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Examples =

== Installing A Package As A New User ==

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a ONE TIME required step for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Finally install atop, or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) Next time you login, simply load the nix module to use atop ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:05:37Z

Roberpj: /* Installing A Package As A New User Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Installing A Package As A New User =

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a ONE TIME required step for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Finally install atop, or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) Next time you login, simply load the nix module to use atop ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:05:03Z

Roberpj: /* Installing A Package As A New User Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Installing A Package As A New User Example =

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'

2) Now install nixpkgs, this is a ONE TIME required step for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Finally install atop, or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) Next time you login, simply load the nix module to use atop ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:04:03Z

Roberpj: /* Installing A Package As A New User Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Installing A Package As A New User Example =

<pre>
1) Lets uninstall atop AND remove nixpkgs to become just like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Now install nixpkgs, which is a ONE TIME required operation for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Finally install atop, or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) Next time you login, simply load the nix module to use atop ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:03:30Z

Roberpj: /* Installing A Package As A New User Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Installing A Package As A New User Example =

<pre>
1) Lets uninstall atop and remove nixpkgs to be like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Now install nixpkgs, which is a ONE TIME required operation for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Finally install atop, or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

4) Next time you login, simply load the nix module to use atop ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:03:11Z

Roberpj: /* Installing A Package As A New User Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Installing A Package As A New User Example =

<pre>
1) Lets uninstall atop and remove nixpkgs to be like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Now install nixpkgs, which is a ONE TIME required operation for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Finally install atop, or any other nix package for that matter !!!
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

4) Next time you login, simply load the nix module to use atop ;)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:01:58Z

Roberpj: /* SIMPLE EXAMPLE: Installing A Package As A New User */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= Installing A Package As A New User Example =

<pre>
1) Lets uninstall atop and remove nixpkgs to be like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Now install nixpkgs, which is a ONE TIME required operation for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Now install atop, or any other nix package for that matter ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

4) Next time you login, simply load the nix module to use atop !
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-13T00:01:07Z

Roberpj: /* Using channels to obtain Nix expressions */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>

== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>

= SIMPLE EXAMPLE: Installing A Package As A New User =

<pre>
1) Lets uninstall atop and remove nixpkgs to be like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Now install nixpkgs, which is a ONE TIME required operation for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Now install atop, or any other nix package for that matter ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

4) Next time you login, simply load the nix module to use atop !
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-12T23:59:31Z

Roberpj: /* Reinstalling atop Package AND nixpkgs Channel Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== Installing A Package As A New User ==

<pre>
1) Lets uninstall atop and remove nixpkgs to be like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Now install nixpkgs, which is a ONE TIME required operation for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Now install atop, or any other nix package for that matter ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

4) Next time you login, simply load the nix module to use atop !
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-12T23:51:04Z

Roberpj: /* Reinstalling atop Package AND nix Channel Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== Reinstalling atop Package AND nixpkgs Channel Example ==

<pre>
1) Lets uninstall atop and remove nixpkgs to be like a new user ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Now install nixpkgs, which is a ONE TIME required operation for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Now install atop, or any other nix package for that matter ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

4) Next time you login, simply load the nix module to use atop !
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-12T23:48:58Z

Roberpj: /* Remove/Read Package & Channel Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== Reinstalling atop Package AND nix Channel Example ==

<pre>
1) Lets clear the slate and uninstall atop and nixpkgs for this example ...
If you have never used nix before on the system this step is unnecessary then goto step 2)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Lets now install nixpkgs, which is a ONE TIME required operation for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Now install atop, or any other nix package for that matter ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

4) Next time we login, to use atop simply load the nix module !
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-12T23:47:45Z

Roberpj: /* Remove/Read Package & Channel Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== Remove/Read Package & Channel Example==

<pre>
1) Lets clear the slate and uninstall atop and nixpkgs for this example ...
If you have never used nix before on the system this step is unnecessary then goto step 2)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Lets now install nixpkgs, which is a ONE TIME required operation for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...

3) Now install atop, or any other nix package for that matter ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

4) Next time we login, to use atop simply load the nix module !
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-12T23:47:07Z

Roberpj: /* Remove/Read Package & Channel Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== Remove/Read Package & Channel Example==

<pre>
1) Lets clear the slate and uninstall atop and nixpkgs for this example ...
If you have never used nix before on the system this step is unnecessary then goto step 2)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Lets now install nixpkgs, which is a ONE TIME required operation for new nix users ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...
[roberpj@gra-login1:~] module unload nix

3) Now install atop, or any other nix package for that matter ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

4) Next time we login, to use atop simply load the nix module !
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-12T23:45:36Z

Roberpj: /* Remove/Read Package & Channel Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== Remove/Read Package & Channel Example==

<pre>
1) Lets clear the slate and uninstall atop and nixpkgs for this example. 
If you have never used nix before on the system this step is unnecessary then goto step 2)
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Lets now install nixpkgs, which is a ONE TIME required operation ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...
[roberpj@gra-login1:~] module unload nix

3) And lets install atop as an example, or any other nix package ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

4) Next time we login atop will be available in our path once the nix module is loaded ....
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-12T23:42:52Z

Roberpj: /* Remove/Read Package & Channel Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== Remove/Read Package & Channel Example==

1) Lets clear the slate and uninstall atop and nixpkgs for this example. 
If you have never used nix before on the system this step is unnecessary then goto step 2)
<pre>
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --query
atop-2.3.0
nix-index-0.1.0
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-env --query
nix-index-0.1.0

2) Lets now install nixpkgs, which is a ONE TIME required operation ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...
[roberpj@gra-login1:~] module unload nix

3) And lets install atop as an example, or any other nix package ...
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop
</pre>

4) Next time we login atop will be available in our path once the nix module is loaded ....
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] which atop
~/.nix-profile/bin/atop

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-12T23:30:05Z

Roberpj: /* Remove/Read Package & Channel Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== Remove/Read Package & Channel Example==

<pre>
[roberpj@gra-login1:~] module unload nix
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...
[roberpj@gra-login1:~] module unload nix

Logout and login again ....

[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
</pre>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-12T23:28:36Z

Roberpj: /* Remove/Read Package & Channel Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== Remove/Read Package & Channel Example==

<pre>
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] module unload nix
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
unpacking channels...
[roberpj@gra-login1:~] module unload nix
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] nix-env --install atop
</pre>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-12T23:27:55Z

Roberpj: /* Remove/Read Package & Channel Example */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== Remove/Read Package & Channel Example==

<pre>
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] module unload nix
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
failed to lock thread to CPU 8
unpacking channels...
[roberpj@gra-login1:~] module unload nix
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
[roberpj@gra-login1:~] nix-env --install atop
</pre>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

NIX

2018-12-12T23:25:56Z

Roberpj: /* Installing packages from a channel */

{{Software
|package_name=NIX
|package_description=User Level Purely Functional Package Manager
|package_idnumber=171
}}

= Introduction =

Nix is a package manager system that allows users to manage their own software environment. This environment is persistent and is shared across all clusters.

* Users can build, install, upgrade, downgrade, and remove packages from their environment without root privileges and without affecting other users.
* Operations either succeed and create a new environment or fail leaving the previous environment in place (operations are atomic).
* Previous environments can be switched back to at any point.
* Users can add their own packages and share them with other users.

The default Nix package set includes a huge selection (over 10,000) of recent versions of many packages.
== Enabling and disabling the Nix environment ==

The user's current Nix environment is enabled by loading the nix module. This creates some ''.nix*'' files and sets some environment variables.

<pre class="sh">module load nix
ls -ld .nix*
env | fgrep -i nix</pre>
It is disabled by unloading the nix module. This unsets the environment variables but leaves the ''.nix*'' files alone.

<pre class="sh">module unload nix
ls -ld .nix*
env | fgrep -i nix
module load nix</pre>
== Completely reseting the Nix environment ==

Most operations can be undone with the <code>--rollback</code> option (i.e., <code>nix-env --rollback</code> or <code>nix-channel --rollback</code>). Sometimes it is useful to entirely reset nix though. This is done by unloading the module, erasing all user related nix files, and then reloading the module file.

<pre class="sh">module unload nix
rm -fr ~/.nix-profile ~/.nix-defexpr ~/.nix-channels ~/.config/nixpkgs
rm -fr /nix/var/nix/profiles/per-user/$USER /nix/var/nix/gcroots/per-user/$USER
module load nix</pre>

= Installing and remove packages =

The <code>nix-env</code> command is used to setup your Nix environment.
== What do I have installed and what can I install ==

Lets first see what we currently have installed.

<nowiki>nix-env --query</nowiki>
Now let's see what is available. We request the attribute paths (unambiguous way of specifying a package) and the descriptions too (cursor to the right to see them). This takes a bit of time as it visits a lot of small files. Especially over NFS it can be a good idea to pipe it to a file and then grep that in the future.

<nowiki>nix-env --query --available --attr-path --description</nowiki>
== Installing packages ==

Let's say that we need a newer version of git than provided by default on our OS (e.g., the CentOS 6 one has issues with fetching over https). First lets check what our OS comes with.

<nowiki>git --version
which git</nowiki>
Let's tell Nix to install its version in our environment.

<nowiki>nix-env --install --attr nixpkgs.git
nix-env --query</nowiki>
Let's checkout what we have now (it may be necessary to tell bash to to forget remembered executable locations with <code>hash -r</code> so it notices the new one).

<nowiki>git --version
which git</nowiki>
== Removing packages ==

For completeness, lets add in the other usual version-control suspects.

<nowiki>nix-env --install --attr nixpkgs.subversion nixpkgs.mercurial
nix-env --query</nowiki>
Actually, we probably don't really want subversion any more. Let's remove that.

<nowiki>nix-env --uninstall subversion
nix-env --query</nowiki>
= Environments =

Nix keeps referring to user environments. Each time we install or remove packages we create a new environment based off of the previous environment.
== Switching between previous environments ==

This means the previous environments still exist and we can switch back to them at any point. Let's say we changed our mind and want subversion back. It's trivial to restore the previous environment.

<nowiki>nix-env --rollback
nix-env --query</nowiki>
Of course we may want to do more than just move to the previous environment. We can get a list of all our environments so far and then jump directly to whatever one we want. Let's undo the rollback.

<nowiki>nix-env --list-generations
nix-env --switch-generation 4
nix-env --query</nowiki>
== Operations are atomic ==

Due to the atomic property of Nix environments, we can't be left halfway through installing/updating packages. They either succeed and create us a new environment or leave us with the previous one intact.

Let's go back to the start when we just had Nix itself and install the one true GNU distributed version control system tla. Don't let it complete though. Hit it with <code>CTRL+c</code> partway through.

<nowiki>nix-env --switch-generation 1
nix-env --install --attr nixpkgs.tla</nowiki>
<blockquote>CTRL+c
</blockquote>
Nothing bad happens. The operation didn't complete so it has no effect on the environment whatsoever.

<nowiki>nix-env --query
nix-env --list-generations</nowiki>
== Nix only does things once ==

The install and remove commands take the current environment and create a new environment with the changes. This works regardless of which environment we are currently in. Let's create a new environment from our original environment by just adding git and mercurial.

<nowiki>nix-env --list-generations
nix-env --install nixpkgs.git nixpkgs.mercurial
nix-env --list-generations</nowiki>
Notice how much much faster it was to install git and mercurial the second time? That is because the software already existed in the local Nix store from the previous installs so Nix just reused it.
== Garbage collection ==

Nix periodically goes through and removes any software not accessible from any existing environments. This means we have to explicitly delete environments we don't want anymore so Nix is able to reclaim the space. We can delete specific environments or any sufficiently old.

<nowiki>nix-env --delete-generations 30d</nowiki>
= Using channels to obtain Nix expressions =

Nix packages are Nix expressions that specify how to build something. The default source for of these expression for <code>nix-env</code> is ''~/.nix-defexp'' and ''~/.nix-defexp/channels''. Nix channels provide a way to populate this later directory with existing expressions from repositories on the internet.
== Subscribing to a channel ==

By default we are subscribed to the SHARCNET version of the latest NixOS release under the name nixpkg. Let us add the older NixOS 15.09 release too under the name nixpkgs_old and then download the latest versions of all NixOS 15.09 the expressions.

<nowiki>nix-channel --list
nix-channel --add file:///nix/channels/sharcnet-15.09 nixpkgs_old
nix-channel --list
nix-channel --update nixpkgs_old</nowiki>
== Installing packages from a channel ==

Now searching our available packages we see that we can install software using either the nixpkgs or nixpkgs_old expressions.

<nowiki>nix-env --query --available --attr-path --description git</nowiki>
Let's replace our git built and installed from the unstable (nixpkgs) expression with one built and installed from the stable expression (note that git is the default version of git which is gitMinimal).

<nowiki>nix-env --install --attr nixpkgs_old.git
nix-env --query
git --version</nowiki>

== Remove/Read Package & Channel Example==

<pre>
[roberpj@gra-login1:~] nix-env --uninstall atop
uninstalling 'atop-2.3.0'
[roberpj@gra-login1:~] module unload nix
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-channel --remove nixpkgs
uninstalling 'nixpkgs'
[roberpj@gra-login1:~] nix-channel --add file:///nix/channels/ccpkgs/latest nixpkgs
[roberpj@gra-login1:~] nix-channel --update
failed to lock thread to CPU 8
unpacking channels...
[roberpj@gra-login1:~] module unload nix
[roberpj@gra-login1:~] module load nix
[roberpj@gra-login1:~] nix-env --install --attr nixpkgs.atop
replacing old 'atop-2.3.0'
installing 'atop-2.3.0'
</pre>

== What about dependency conflicts ==

There are no issues with having a mix of packages installed from different sources even if they have conflicting dependencies. Nix puts each package in a separate directory under ''/nix/store'' versioned by the hash of the its build instructions. Binaries are linked with rpaths to ensure they always find the versions they need.

<nowiki>ldd $(readlink ($which git))</nowiki>
Switching between environments continue to work as before.

<nowiki>nix-env --rollback
git --version
nix-env --list-generations
nix-env --switch-generation 5
git --version</nowiki>
== Switching between previous updates ==

The channel updates (updating the list of expressions we can build and install from) are also atomic and versioned. This ensures we never find ourselves stuck due to accidentally updating to something broken.

<nowiki>nix-channel --rollback
nix-env --query --available --attr-path git
nix-channel --rollback 2</nowiki>
= Development with Nix (e.g., Python, R, etc.) =

Several languages have their own repository of packages and associated infrastructure (e.g., PyPI and pip). Nix builds on these to automatically handle the external dependencies (e.g., C libraries) and makes it easy to work simultaneously with different projects requiring different versions of internal packages.
== Wrapper scripts for setting paths ==

Nix has expressions for many of these that generate wrapper scripts that set path environment variables to bring a specific set of the internal packages into scope and then run the approriate program.

The following Nix expression defines top-level myPython, myR, and myHaskell attributes that use the appropriate package-specific expressions to create wrappers for a selection of internal packages (for details about the Nix langauge see the [https://nixos.org/nix/manual/#ch-expression-language Nix Expression Language].

<pre class="nix">with import <nixpkgs> { };

rec {
myPython = python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
};

myR = rWrapper.override rec {
packages = with rPackages; [ rgeos rgdal ];
};

myHaskell = haskellPackages.ghcWithPackages (ghcpkgs:
with ghcpkgs; [ pipes lens cabal-install ]
);
}</pre>
Saving it as ''~/.nix-defexpr/mypkgs.nix'' makes these available for installation with <code>nix-env</code>.

<nowiki>nix-env --install --attr mypkgs.myPython
cat $(which python3)
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
== Development environments ==

A Nix expression evaluates to a specification for setting up a (build) environment and then doing a build in it. Frequently we want to do something very similar: setup a (development) environment and then do development in it.

The <code>nix-shell</code> command bridges the gap between these two. It uses Nix to sets up a non-rooted (build) environment with the dependencies we specify and then laucnhes a (development) shell in it instead of a build process.

<nowiki>mkdir devel
cd devel</nowiki>
Say we need Clang and Python with the NumPy and SciPy packages. Instead of installing these into our global environment we can create a Nix expression that specifies these as build (development) inputs

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
( python3.buildEnv.override rec {
extraLibs = with python3Packages; [ numpy scipy ];
} )
clang
];
}</pre>
save it in ''default.nix'' and then run <code>nix-shell</code>.

<nowiki>nix-shell</nowiki>
Now we are in a build (development) environment with all the dependencies we specified in the appropriate PATHs.

<nowiki>cat $(which python3)
clang --version
python3</nowiki>
<nowiki>import scipy</nowiki>
<blockquote>CTRL+d
</blockquote>
To return to our standard environment we just exit the shell. This is extreamily nice if we work on many projects with conflicting dependencies.

<nowiki>exit
cd ..</nowiki>
== One-off environments ==

Quite frequently we need a one-off development environment with a few packages. Say CLang and git. Rather than have to write the following boilerplate ''default.nix'' file

<pre class="nix">with import <nixpkgs> { };

stdenv.mkDerivation {
name = "my-env";
buildInputs = [
clang
git
];
}</pre>
we can just get <code>nix-shell</code> to do it for us.

<pre class="sh">nix-shell --packages clang git</pre>
<blockquote>CTRL+d
</blockquote>

= Submitting Jobs =

Loading the Nix module before submitting a job makes the Nix environment available to the job. Note that these jobs will see the current Nix environment including any changes made after submission. Compiled binaries should not require the Nix module to be loaded to run.

<pre class="sh">module load nix
sqsub ...</pre>
= Internals =

This section details some of the internals of how Nix works and how to create your own packages. It is more advanced material not required for the basic usage.
== The Nix store ==

A Nix environment is just a collection of symlinks to all the packages that exist in that environment.

<pre class="sh">readlink $(which git)
readlink -f ~/.nix-profile</pre>
Everything in Nix exists as a path in the Nix store. Each path is distinguished with a hash of either its contents or everything that went into creating it to keep it separate from everything else. Paths are immutable once created. This means they can be freely shared.

Paths are created by the Nix build server when it realizes a Nix derivation. Nix derivations specify all the inputs to a jailed process that creates the contents of the store path.
== Nix expressions and instantiation ==

Nix derivations are instantiated from Nix expressions, which are written in the Nix language. The <code>nix-repl</code> program can be used to interactively experiment with the Nix language.

<pre class="sh">nix-env --install --attr nix-repl
nix-repl</pre>
The following ''cabextract.nix'' Nix expression evaluates to a derivation fully specifying how to build the <code>cabextract</code> utility.

<pre class="nix">with import ~/.nix-defexpr/nixpkgs { };
stdenv.mkDerivation rec {
name = "cabextract-1.6"

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
}</pre>
It is written using the <code>stdenv.mkDerivation</code> function from <code>nixpkgs</code>. This function creates a derivation that executes the standard unpack, patch, configure, build, check, install, fixup, check, and distribute steps on the package. It provides a series of hooks that can be used at each step to customize the process for non-standard packages. For full details see the [https://nixos.org/nixpkgs/manual nixpkgs manual].

The Nix expression is instantiate to a derivation using <code>nix-instantiate</code>. The leading ''./'' is required to distinguish that the arguments is file containing a Nix expression to be instantiated and not the name of a package in the default Nix expression to instantiate.

<pre class="sh">cabdrv=$(nix-instantiate ./cabextract.nix)
echo $cabdrv</pre>

== Nix derivations and realization ==

The Nix derivation instantiated from the above Nix expression can be pretty-printed using the <code>pp-aterm</code> program.

<pre class="sh">nix-env --install --attr nixpkgs.strategoPackages.strategoxt
pp-aterm -i $cabdrv</pre>
<pre>Derive(
[("out", "/home/nixbld/store/...-cabextract-1.6", "", "")]
, [ ("/home/nixbld/store/...-stdenv.drv", ["out"])
, ("/home/nixbld/store/...-cabextract-1.6.tar.gz.drv", ["out"])
, ("/home/nixbld/store/...-bash-4.3-p42.drv", ["out"])
]
, ["/home/nixbld/store/...-default-builder.sh"]
, "x86_64-linux"
, "/home/nixbld/store/...-bash-4.3-p42/bin/bash"
, ["-e", "...-default-builder.sh"]
, [ ("buildInputs", "")
, ("builder", "/home/nixbld/store/...-bash-4.3-p42/bin/bash")
, ("name", "cabextract-1.6")
, ("nativeBuildInputs", "")
, ("out", "/home/nixbld/store/...-cabextract-1.6")
, ("propagatedBuildInputs", "")
, ("propagatedNativeBuildInputs", "")
, ("src", "/home/nixbld/store/...-cabextract-1.6.tar.gz")
, ("stdenv", "/home/nixbld/store/...-stdenv")
, ("system", "x86_64-linux")
]
)</pre>
The Nix build server realizes derivations by building the packages in an isolated environment. Each derivation specifies what comes out of the environment, what goes into it, the required machine architecture, what build program to execute in the environment, and what arguments and environment to pass to the program to be executed.

The <code>nix-store</code> program <code>--realize</code> option is used to signal the build server to realize the derivation.

<pre class="sh">cabpath=$(nix-store --realize $cabdrv)
echo $cabpath</pre>
The <code>nix-env</code> program will add the realized derivation to the users environment with the <code>--install</code> (<code>-i</code>) option. Technically it creates a new realization of the <code>user-environment</code> package that symlinks in the entire contents of the path path and switches to it.

<pre class="sh">nix-env --install $cabpath</pre>
The build log associated with a realization can be viewed with the <code>--read-log</code> (<code>-l</code>) option.

<pre class="sh">nix-store --read-log $cabdrv
nix-store --read-log $cabpath
nix-store --read-log $(which git)</pre>
The <code>nix-store</code> program also supports numerous other store related items such as computing the transitive closure of a package with the <code>--query</code> (<code>-q</code>) <code>--requisites</code> (<code>-R</code>) and exporting them via the <code>--export</code> option to a Nix archive for import into another store.

<pre class="sh">nix-store --query --requisites $cabpath
nix-store --export $(nix-store --query --requisites $(which git)) | gzip > git.nar.gz</pre>
== Nix default expression ==

Nix has a default expression created from the contents of ''~/.nix-defexpr''. The <code>nix-env</code> <code>--install</code> operation normally works by selecting an attribute in this expression (fast) or matching against the <code>name</code> attribute in all the attributes in this expression (slow) to determine an expression to instantiate, realize, and then symlink into a new environment.

The <code>nix-channel</code> command works by building new Nix expression packages (containing all the Nix expressions for the subscribed channels) and symlinking ''~/.nix-defexprs/channels'' to then. Adding additional expressions to ''~/.nix-defexpr'' makes them available for use with <code>nix-env</code> as well. For example, the following ''~/.nix-defexpr/mypkgs'' expression packages up the above ''cabextract.nix'' example.

<pre class="nix">args@{ ... }: with import ./nixpkgs args;
rec {
cabextract = stdenv.mkDerivation rec {
name = "cabextract-1.6";

src = fetchurl {
url = "http://www.cabextract.org.uk/${name}.tar.gz";
sha256 = "1ysmmz25fjghq7mxb2anyyvr1ljxqxzi4piwjhk0sdamcnsn3rnf";
};

meta = with stdenv.lib; {
homepage = http://www.cabextract.org.uk/;
description = "Free Software for extracting Microsoft cabinet files";
platforms = platforms.all;
license = licenses.gpl3;
maintainers = with maintainers; [ pSub ];
};
};
}</pre>
This makes it possible to install packages from <code>mypkgs</code> as easily as those from the official <code>nixpkgs</code>.

<pre class="sh">nix-env --install --attr mypkgs.cabextract</pre>
=References=

o Nix Package Manager 
http://nixos.org/nix/

o Official Nix/Nixpkgs/NixOS 
https://github.com/NixOS

o Nix on SHARCNET (slides by Tyson Whitehead) 
https://www.sharcnet.ca/help/images/5/5f/Tyson_nix_2015.pdf

o Exploring a new approach to package management (youtube video by Tyson Whitehead) 
https://www.youtube.com/watch?v=pQE9WTLAPHQ

LSDYNA

2018-12-06T21:24:32Z

Roberpj: /* MPI Job */

{{Software
|package_name=LSDYNA
|package_description=Suite of programs for transient dynamic finite element program
|package_idnumber=104
}}

=Introduction=

Before a research group can use LSDYNA on sharcnet, a license must be purchased directly from LSTC for the sharcnet license server. Alternately, if a research group resides at a institution that has sharcnet computing hardware (mac, uwo, guelph, waterloo) it maybe possible to use a pre-existing site license hosted on an accessible institutional license server. To access and use this software you must open a ticket and request to be added to the sharcnet lsdyna group.

=Version Selection=

==Graham Cluster==

===License===

Create and customize this file for your license server, where XXXXX should be replaced with your port number and Y with your server number:

[roberpj@gra1:~] cat .licenses/ls-dyna.lic
#LICENSE_TYPE: network
#LICENSE_SERVER: XXXXX@licenseY.uwo.sharcnet

===Module===

The modules are loaded automatically in the testomp.sh and testmpi.sh script shown below in the example section:

For single node smp jobs the available versions can be found by doing:

[roberpj@gra-login2:~] module spider ls-dyna
ls-dyna/7.1.2
ls-dyna/7.1.3
ls-dyna/8.1
ls-dyna/9.1
ls-dyna/9.2
ls-dyna/10.0

To load any version do for example:

module load ls-dyna/9.1

For multi node mpp jobs the available versions can be found by doing:

[roberpj@gra-login2:~] module spider ls-dyna-mpi
ls-dyna-mpi/7.1.2
ls-dyna-mpi/7.1.3
ls-dyna-mpi/8.1
ls-dyna-mpi/9.1
ls-dyna-mpi/9.2
ls-dyna-mpi/10.0

To load the 7.1.X modules do the following:
module load openmpi/1.6.5 ls-dyna-mpi/7.1.3

To load the 8.X or 9.X modules do:
module load openmpi/1.8.8 ls-dyna-mpi/9.2

To load the 10.X modules do:
module load openmpi/1.10.7 ls-dyna-mpi/10.0

==Legacy Clusters==

===License===

Once a license configuration is established, the research group will be given a 5 digit port number. The value should then be inserted into the appropriate departmental export statement before loading the module file as follows:

o UofT Mechanical Engineering Dept
export LSTC_LICENSE_SERVER=Port@license1.uwo.sharcnet

o McGill Mechanical Engineering Department
export LSTC_LICENSE_SERVER=Port@license2.uwo.sharcnet

o UW Mechanical and Mechatronics Engineering Dept
export LSTC_LICENSE_SERVER=Port@license3.uwo.sharcnet

o Laurentian University Bharti School of Engineering
export LSTC_LICENSE_SERVER=Port@license4.uwo.sharcnet

o Fictitious Example:

export LSTC_LICENSE_SERVER=12345@license1.uwo.sharcnet

===Module===

The next step is load the desired sharcnet lsdyna module version. Check which modules are available by running the module avail command then load one the modules as shown:

<source lang="bash">
[roberpj@orc-login2:~] module avail lsdyna
------------------------------------ /opt/sharcnet/modules ------------------------------------
lsdyna/hyb/r711.88920 lsdyna/mpp/r711.88920 lsdyna/smp/r611.79036
lsdyna/mpp/ls971.r85718 lsdyna/mpp/r712.95028 lsdyna/smp/r711.88920
lsdyna/mpp/ls980B1.011113 lsdyna/mpp/r800.95359 lsdyna/smp/r712.95028
lsdyna/mpp/r611.79036 lsdyna/mpp/r901.109912 lsdyna/smp/r800.95359
lsdyna/mpp/r611.80542 lsdyna/smp/ls980B1.78258 lsdyna/smp/r901.109912
</source>

====r9.0.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r901.109912</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna mkl
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r901.109912
</pre>

====r8.0.0 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r800.95359</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna mkl
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r800.95359
</pre>

====r7.1.2 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r712.95028</pre>

o For mpi jobs:
<pre>module unload intel openmpi lsdyna
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
</pre>

====r7.1.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r711.88920</pre>

o For mpi jobs:
<pre>module load lsdyna/mpp/r711.88920</pre>

====r6.1.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r611.79036</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna

module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036
or
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.80542
</pre>

where r611.80542 provides lsdyna_s only. 

====ls980 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/ls980B1.78258</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna

module load intel/12.1.3 openmpi/intel/1.4.5 lsdyna/mpp/ls980B1.011113
or
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/smp/ls980B1.78258
</pre>

Note1) restart capability for ls980smpB1 and ls980mppB1 is not supported
Note2) the module using legacy openmpi/intel/1.4.5 will run extremely slow

=Job Submission=

==Graham Cluster==

The submission scripts myompjob.sh and mympijob.sh for the airbag problem are shown in the Example Job section below, for both graham and orca. Please note that you should specify your own username if submitting to a def account (not roberpj). Alternatively you could specify your resource allocation account:

Sample threaded job submit script:

sbatch myompjob.sh

Sample mpi job submit script:

sbatch mympijob.sh

==Legacy Clusters==

When loading a lsdyna sharcnet legacy module on the new orca or a sharcnet legacy system, the single or double precision solvers are specified with lsdyna_s or lsdyna_d respectively, as shown in the following sqsub commands:

===1cpu SERIAL Job===
<pre>sqsub -r 1h -q serial -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=1</pre>

===4cpu SMP Job===
<pre>sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=1G lsdyna_s i=airbag.deploy.k ncpu=4</pre>

If using an explicit solver, one can specify a conservative initial memory setting on the command line as follows:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=4 memory=754414</pre>
where memory is the minimum number of 8 byte words shared by all processors in double precision, or, 4 byte words in single precision.

The initial value can be determined by starting a simulation interactively on the command line, and finding the output resembling: Memory required to begin solution : 754414. The number of words can be specified as memory=260M instead of memory=260000000, for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/.

===8cpu MPI Job===

<pre>sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k</pre>

The initial memory can also be specified for mpi jobs on the sqsub command line with "memory=" for the first master processor to decompose the problem and "memory2=" used on all processors including the master to solves the decomposed problem, where the values are specified as 4 bytes per word in single precision and 8 bytes per word in double precision. The number of words can be specified as memory=260M instead of memory=260000000 OR memory2=260M instead of memory2=260000000 for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/. The initial values can be found by running simulation interactively on a orca compute node and checking the output for a line such as: Memory required to begin solution (memory= 464898 memory2= 158794 ) which could then be implemented for a job run in the queue by doing:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898 memory2=158794
</pre>

The specification of memory2 on sharcnet or compute canada clusters is not beneficial since the queue reserves the same memory per core for all nodes, such that the decomposition master node process cannot be allocated to have larger system memory. Therefore its sufficient to specify a single memory parameter by doing:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898
</pre>

where the LSTC_MEMORY variable will only allow the memory to grow for explicit simulations. The following slurm examples demonstrate how to prescribe the memory parameters exactly.

=Example Job=

Copy the airbag example to your account:

rsync -av orca.computecanada.ca:/opt/sharcnet/lsdyna/r901.109912/examples r901.109912_examples
cd r901.109912_examples/examples/misc/airbag
gunzip airbag.deploy.k.gz

==Graham (default modules)==

Please note that graham does not have the sharcnet legacy modules installed on it.

===Threaded Job ===

Sample submission script mysmpjob1.sh for 4 core single precision smp job using compute canada default cvmfs modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --nodes=1 # use one node
#SBATCH --cpus-per-task=4 # number threads
#SBATCH --mem=4000M # total memory
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=testsmp1 # %x = jobname
#SBATCH --output=slurm-%x-%N-%j.out
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# ----------------------------------------------------------
# IMPORTANT: to set license server do either:
# 1) export LSTC_LICENSE_SERVER=port@some.license.server
# 2) create ~/.licenses/ls-dyna.lic as shown above
# ----------------------------------------------------------
module load ls-dyna/9.1
ls-dyna_s i=airbag.deploy.k ncpu=4 memory=1000M
</pre>

where memory = 4000M / 4 (bytes/word) = 1000M words

===Mpi Job ===

Sample submission script mympijob2.sh for 4 core double precision mpp job using compute canada default cvmfs modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --ntasks=4 # number of ranks
#SBATCH --mem-per-cpu=4000M # memory per rank
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=testmpi2 # %x = jobname
#SBATCH --output=slurm-%x-%N-%j.out
# ----------------------------------------------------------
# IMPORTANT: to set license server do either:
# 1) export LSTC_LICENSE_SERVER=port@some.license.server
# 2) create ~/.licenses/ls-dyna.lic as shown above
# ----------------------------------------------------------
module load openmpi/1.8.8 ls-dyna-mpi/9.1
srun ls-dyna_d i=airbag.deploy.k memory=500M memory2=500M
</pre>

where memory = 4000M / 8 (bytes/word) = 500M words

==New Orca (legacy modules)==

Please note, while new orca has most compute canada cvmfs modules available by default it does not have the graham the ls-dyna or ls-dyna-mpi modules installed. Therefore currently the only way to run lsdyna is by using the legacy sharcnet lsdyna modules as shown in the following two smp and mpi examples.

===Threaded Job ===

Submission script mysmpjob3.sh to run 16 core single precision single node smp job with sharcnet legacy modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --nodes=1 # use one node
#SBATCH --cpus-per-task=16 # number threads
#SBATCH --mem=4000M # total memory
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=mysmpjob3-1node-16core # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# ----------------------------------------------------------------
# IMPORTANT: to set license server for legacy modules:
export LSTC_LICENSE_SERVER=31047@license3.sharcnet.ca
# since ~/.licenses/ls-dyna.lic does NOT work with legacy modules
# ----------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load lsdyna/smp/r712.95028
lsdyna_s i=airbag.deploy.k ncpu=$SLURM_CPUS_PER_TASK memory=500M
</pre>

where memory = 4000M / 8 (bytes/word) = 500M words

===MPI Job ===

To run the sharcnet legacy module lsdyna/mpp/r712.95028 efficiently over several compute nodes with many cores requires using orca's xeon nodes as shown in mympijob4.sh below. To run lsdyna on orca's opteron nodes with module lsdyna/mpp/r712.95028 requires using only one opteron node as shown in mympijob5.sh below. The other legacy sharcnet lsdyna mpp modules, as shown in the Availability Table lsdyna/mpp/ls971.r85718, lsdyna/mpp/ls980B1.011113, lsdyna/mpp/r611.79036, lsdyna/mpp/r611.80542, lsdyna/mpp/r711.88920, lsdyna/mpp/r800.95359 and lsdyna/mpp/r901.109912 have yet to be tested. If you want to use one but cannot get it working please open a ticket. Note that lines containing a space between "# and SBATCH" are comments. To activate such line(s) required removal of the space ie) "#SBATCH". Its recommended to use module ls-dyna-mpi on graham instead since the machine is far newer, faster and reliable.

====Multi Node Slurm Script====

<pre>
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob4.sh
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --constraint=xeon
#SBATCH --ntasks=48
#SBATCH --mem-per-cpu=4000M
#SBATCH --time=00:30:00
#SBATCH --job-name=mympijob4-Xnodes-48cores-xeon # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
# -------------------------------------------------------------
# To set license server when using legacy sharcnet modules:
export LSTC_LICENSE_SERVER=port@some.license.server
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.
# -------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`
mpirun --host $MACHINEFILE --mca plm_slurm_args '--cpu_bind=none' lsdyna_s i=airbag.deploy.k memory=250M memory2=250M
</pre>

====Single Node Slurm Script====

<pre>
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob5.sh
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --constraint=opteron
#SBATCH --nodes=1
# SBATCH --ntasks=24
# SBATCH --mem=0
#SBATCH --ntasks=16
#SBATCH --mem-per-cpu=1000M
#SBATCH --time=00:30:00
#SBATCH --job-name=mympijob5-1node-16cores-opteron # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
# -------------------------------------------------------------
# To set license server when using legacy sharcnet modules:
export LSTC_LICENSE_SERVER=port@some.license.server
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.
# -------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`
mpirun --host $MACHINEFILE lsdyna_s i=airbag.deploy.k memory=250M memory2=250M
</pre>

where memory = 1000M / 4 (bytes/word) = 250M words

==Legacy Clusters==

STEP1) The following shows sqsub submission of the [http://www.dynaexamples.com/examples-manual/misc/airbag airbag] example to the mpi queue. Its recommended to first edit airbag.deploy.k and change endtim to 3.000E-00 so the job runs long enough to perform the restart in steps 2 and 3 below:
<pre>
cp -a /opt/sharcnet/lsdyna/r611.79036/examples /scratch/$USER/test-lsdyna
cd /scratch/$USER/test-lsdyna/misc/airbag
gunzip airbag.deploy.k.gz
cp airbag.deploy.k airbag.deploy.restart.k
nano deploy.restart.k (reduce plot file creation frequency)
*DATABASE_BINARY_D3PLOT
$ dt lcdtdeploy.restart.k
1.000E-01 <--- change from 5.000E-04
export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet
module unload intel openmpi lsdyna
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036
sqsub -r 10m -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.k
</pre>

STEP2) With the job still running, use the echo command as follows to create a file called "D3KIL" that will trigger generation of restart files at which point the file D3KIL itself will be erased. Do this once a day if data loss is critical to you OR once or twice just before the sqsub -r time limit is reached. Further information can be found here http://www.dynasupport.com/tutorial/ls-dyna-users-guide/sense-switch-control:

<pre>
echo "sw3" > D3KIL
sqkill job#
</pre>

STEP3) Before the job can be restarted, the following two lines must be added to the airbag.deploy.restart.k file:

<pre>
*CHANGE_CURVE_DEFINITION
1
</pre>

STEP4) Now resubmit the job as follows using "r=" to specify the restart file:

<pre>
sqsub -r 1h -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.restart.k r=d3dump01
</pre>

=General Notes=

==Graham Cluster==

==Command Line Use==

Parallel lsdyna jobs can be run interactively on the command line (outside the queue) for short run testing purposes as follows:

o Orca Development Node MPP (mpi) Example

ssh orca.sharcnet.ca
ssh orc-dev1 (or dev2,3,4)
module unload intel openmpi lsdyna
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/mpp/r711.88920
mpirun -np 8 lsdyna_d i=airbag.deploy.k

o Orca Development Node SMP (threaded) Example

ssh orca.sharcnet.ca
ssh orc-dev1 (or dev2,3,4)
module unload intel openmpi lsdyna
module load lsdyna/smp/r711.88920
lsdyna_d i=airbag.deploy.k ncpu=8

==Memory Issues==

A minumum of mpp=2G is recommended although the memory requirement of a job may suggest much less is required. For instance setting mpp=1G for the airbag test job above will result in the following error when running a job in the queue:

<pre>
--------------------------------------------------------------------------
An attempt to set processor affinity has failed - please check to
ensure that your system supports such functionality. If so, then
this is probably something that should be reported to the OMPI developers.
--------------------------------------------------------------------------
</pre>

Another error message which can occur after a job runs for a while if mpp is chosen to small is:

<pre>
>>>>> Process 45 <<<<<
>>>>> Signal 11 : Segmentation Violation <<<<<
</pre>

To get an idea of the amount of memory a job used run grep on the output file:

<pre>
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep Memory
| Distributed Memory Parallel |
Memory size from command line: 500000, 500000
Memory for the head node
Memory installed (MB) : 32237
Memory free (MB) : 11259
Memory required (MB) : 0
Memory required to process keyword : 458120
Memory required for decomposition : 458120
Memory required to begin solution (memory= 458120 memory2= 230721)
Max. Memory reqd for implicit sol: max used 0
Max. Memory reqd for implicit sol: incore 0
Max. Memory reqd for implicit sol: oocore 0
Memory required to complete solution (memory= 458120 memory2= 230721)
</pre>

==Version and Revision==

Again run grep on the output file to extract the major and minor revision:

<pre>
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep 'Revision\|Version'
| Version : mpp s R6.1.1 Date: 01/02/2013 |
| Revision: 78769 Time: 07:43:30 |
| SVN Version: 80542 |
</pre>

==Check License Status==

===Running and Queued Programs===

To get a summary of running and/or queued jobs use the lstc_qrun command as follows, where queued means the job has started running according to the sqsub command but its actually sitting waiting for license resources to come available. Once these are acquired the job will start running on the cluster and appear as a Running Program according to lstc_qrun ie)

<source lang="bash">
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet

[roberpj@hnd19:~] lstc_qrun
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
Running Programs
User Host Program Started # procs
-----------------------------------------------------------------------------
dgierczy 20277@saw61.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1
dgierczy 8570@saw32.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1
dscronin 25486@hnd6 MPPDYNA_971 Tue Apr 9 20:19 6
dscronin 14897@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 14971@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 15046@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 31237@hnd16 MPPDYNA_971 Tue Apr 9 21:53 6
dscronin 31313@hnd16 MPPDYNA_971 Tue Apr 9 21:54 6
dscronin 6396@hnd15 MPPDYNA_971 Tue Apr 9 21:54 6
csharson 28890@saw175.saw.sharc MPPDYNA_971 Wed Apr 10 16:48 6
No programs queued
</source>

To show license file expiration details, append "-R" to the command:

<source lang="bash">
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet

[roberpj@hnd19:~] lstc_qrun -R
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
**** LICENSE INFORMATION ****
PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE
---------------- ---------- ----- ------ ------ | -----
LS-DYNA_971 12/31/2013 - 966 1024 | 0
dgierczy 20277@saw61.saw.sharcnet 1
dgierczy 8570@saw32.saw.sharcnet 1
MPPDYNA_971 12/31/2013 - 966 1024 | 0
dscronin 25486@hnd6 6
dscronin 14897@hnd18 6
dscronin 14971@hnd18 6
dscronin 15046@hnd18 6
dscronin 31237@hnd16 6
dscronin 31313@hnd16 6
dscronin 6396@hnd15 6
csharson 28890@saw175.saw.sharcnet 6
LICENSE GROUP 58 966 1024 | 0

PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE
---------------- ---------- ----- ------ ------ | -----
LS-OPT 12/31/2013 0 1024 1024 | 0
LICENSE GROUP 0 1024 1024 | 0
</source>

===Killling A Program===

The queue normally kills lsdyna jobs cleanly. However its possible that licences for a job (which is no longer running in the queue and therefore no longer has any processes running on the cluster) will continue to be tied up according to the lstc_qrun command. To kill such a program determine the pid@hostname from the lstc_qrun command then run the following kill command. The following example demonstrates the procedure for username roberpj:

<pre>
[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj
roberpj 45312@orc329.orca.shar MPPDYNA Wed Dec 5 15:50 12

[roberpj@orc-login2:~] lstc_qkill 45312@orc329.orca.sharcnet.ca
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
Program queued for termination

[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj
[roberpj@orc-login2:~]
</pre>

==Legacy Instructions==

The following binaries remain available on saw and orca for backward compatibility testing:

<pre>
[roberpj@orc-login2:~] cd /opt/sharcnet/local/lsdyna
[roberpj@orc129:/opt/sharcnet/local/lsdyna] ls ls971*
ls971_d_R3_1 ls971_d_R4_2_1 ls971_s_R3_1 ls971_s_R4_2_1 ls971_s_R5_1_1
ls971_d_R4_2_0 ls971_d_R5_0 ls971_s_R4_2_0 ls971_s_R5_0
</pre>

There are currently no sharcnet modules for these versions, hence jobs should be submitted as follows:

<pre>
module load lsdyna
export PATH=/opt/sharcnet/local/lsdyna:$PATH
export LSTC_LICENSE_SERVER=XXXXX@license3.uwo.sharcnet
cp /opt/sharcnet/local/lsdyna/examples/airbag.deploy.k airbag.deploy.k
SERIAL JOB: sqsub -q serial -r 1d -o ofile.%J ls971_d_R4_2_1 i=airbag.deploy.k
THREADED JOB: sqsub -q threaded -n 4 -r 1d -o ofile.%J ls971_d_R4_2_1 ncpu=4 para=2 i=airbag.deploy.k
</pre>

Note! the ls971_s_R3_1 and ls971_d_R3_1 binaries do not work, a fix is being looked for.

=References=

o LSTC LS-DYNA Homepage 
http://www.lstc.com/products/ls-dyna

o LSTC LS-DYNA Support (Tutorials, HowTos, Faq, Manuals, Release Notes, News, Links) 
http://www.dynasupport.com/

o LS-DYNA Release Notes 
http://www.dynasupport.com/release-notes

o LS-PrePost Online Documentation FAQ 
http://www.lstc.com/lspp/content/faq.shtml

o LSTC Download/Install Overview Page 
Provides links to binaries for LS-DYNA SMP/MPP, LS-OPT, LS-PREPOST, LS-TASC. 
http://www.lstc.com/download

==Memory Links==

o Implicit: Memory notes 
http://www.dynasupport.com/howtos/implicit/implicit-memory-notes

o LS-DYNA Support Environment variables 
http://www.dynasupport.com/howtos/general/environment-variables

o LS-DYNA and d3VIEW Blog (a few words on memory settings) 
http://blog2.d3view.com/a-few-words-on-memory-settings-in-ls-dyna/

o Convert Words to GB ie) memory=500MW (3.73GB) 
http://deviceanalytics.com/memcalc.php

LSDYNA

2018-12-06T21:21:00Z

Roberpj: /* MPI Job */

{{Software
|package_name=LSDYNA
|package_description=Suite of programs for transient dynamic finite element program
|package_idnumber=104
}}

=Introduction=

Before a research group can use LSDYNA on sharcnet, a license must be purchased directly from LSTC for the sharcnet license server. Alternately, if a research group resides at a institution that has sharcnet computing hardware (mac, uwo, guelph, waterloo) it maybe possible to use a pre-existing site license hosted on an accessible institutional license server. To access and use this software you must open a ticket and request to be added to the sharcnet lsdyna group.

=Version Selection=

==Graham Cluster==

===License===

Create and customize this file for your license server, where XXXXX should be replaced with your port number and Y with your server number:

[roberpj@gra1:~] cat .licenses/ls-dyna.lic
#LICENSE_TYPE: network
#LICENSE_SERVER: XXXXX@licenseY.uwo.sharcnet

===Module===

The modules are loaded automatically in the testomp.sh and testmpi.sh script shown below in the example section:

For single node smp jobs the available versions can be found by doing:

[roberpj@gra-login2:~] module spider ls-dyna
ls-dyna/7.1.2
ls-dyna/7.1.3
ls-dyna/8.1
ls-dyna/9.1
ls-dyna/9.2
ls-dyna/10.0

To load any version do for example:

module load ls-dyna/9.1

For multi node mpp jobs the available versions can be found by doing:

[roberpj@gra-login2:~] module spider ls-dyna-mpi
ls-dyna-mpi/7.1.2
ls-dyna-mpi/7.1.3
ls-dyna-mpi/8.1
ls-dyna-mpi/9.1
ls-dyna-mpi/9.2
ls-dyna-mpi/10.0

To load the 7.1.X modules do the following:
module load openmpi/1.6.5 ls-dyna-mpi/7.1.3

To load the 8.X or 9.X modules do:
module load openmpi/1.8.8 ls-dyna-mpi/9.2

To load the 10.X modules do:
module load openmpi/1.10.7 ls-dyna-mpi/10.0

==Legacy Clusters==

===License===

Once a license configuration is established, the research group will be given a 5 digit port number. The value should then be inserted into the appropriate departmental export statement before loading the module file as follows:

o UofT Mechanical Engineering Dept
export LSTC_LICENSE_SERVER=Port@license1.uwo.sharcnet

o McGill Mechanical Engineering Department
export LSTC_LICENSE_SERVER=Port@license2.uwo.sharcnet

o UW Mechanical and Mechatronics Engineering Dept
export LSTC_LICENSE_SERVER=Port@license3.uwo.sharcnet

o Laurentian University Bharti School of Engineering
export LSTC_LICENSE_SERVER=Port@license4.uwo.sharcnet

o Fictitious Example:

export LSTC_LICENSE_SERVER=12345@license1.uwo.sharcnet

===Module===

The next step is load the desired sharcnet lsdyna module version. Check which modules are available by running the module avail command then load one the modules as shown:

<source lang="bash">
[roberpj@orc-login2:~] module avail lsdyna
------------------------------------ /opt/sharcnet/modules ------------------------------------
lsdyna/hyb/r711.88920 lsdyna/mpp/r711.88920 lsdyna/smp/r611.79036
lsdyna/mpp/ls971.r85718 lsdyna/mpp/r712.95028 lsdyna/smp/r711.88920
lsdyna/mpp/ls980B1.011113 lsdyna/mpp/r800.95359 lsdyna/smp/r712.95028
lsdyna/mpp/r611.79036 lsdyna/mpp/r901.109912 lsdyna/smp/r800.95359
lsdyna/mpp/r611.80542 lsdyna/smp/ls980B1.78258 lsdyna/smp/r901.109912
</source>

====r9.0.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r901.109912</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna mkl
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r901.109912
</pre>

====r8.0.0 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r800.95359</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna mkl
module load intel/15.0.3 openmpi/intel1503-std/1.8.7 lsdyna/mpp/r800.95359
</pre>

====r7.1.2 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r712.95028</pre>

o For mpi jobs:
<pre>module unload intel openmpi lsdyna
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
</pre>

====r7.1.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r711.88920</pre>

o For mpi jobs:
<pre>module load lsdyna/mpp/r711.88920</pre>

====r6.1.1 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/r611.79036</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna

module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036
or
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.80542
</pre>

where r611.80542 provides lsdyna_s only. 

====ls980 versions====

o For serial or threaded jobs:
<pre>module load lsdyna/smp/ls980B1.78258</pre>

o For mpi jobs:
<pre>
module unload intel openmpi lsdyna

module load intel/12.1.3 openmpi/intel/1.4.5 lsdyna/mpp/ls980B1.011113
or
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/smp/ls980B1.78258
</pre>

Note1) restart capability for ls980smpB1 and ls980mppB1 is not supported
Note2) the module using legacy openmpi/intel/1.4.5 will run extremely slow

=Job Submission=

==Graham Cluster==

The submission scripts myompjob.sh and mympijob.sh for the airbag problem are shown in the Example Job section below, for both graham and orca. Please note that you should specify your own username if submitting to a def account (not roberpj). Alternatively you could specify your resource allocation account:

Sample threaded job submit script:

sbatch myompjob.sh

Sample mpi job submit script:

sbatch mympijob.sh

==Legacy Clusters==

When loading a lsdyna sharcnet legacy module on the new orca or a sharcnet legacy system, the single or double precision solvers are specified with lsdyna_s or lsdyna_d respectively, as shown in the following sqsub commands:

===1cpu SERIAL Job===
<pre>sqsub -r 1h -q serial -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=1</pre>

===4cpu SMP Job===
<pre>sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=1G lsdyna_s i=airbag.deploy.k ncpu=4</pre>

If using an explicit solver, one can specify a conservative initial memory setting on the command line as follows:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q threaded -n 4 -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=4 memory=754414</pre>
where memory is the minimum number of 8 byte words shared by all processors in double precision, or, 4 byte words in single precision.

The initial value can be determined by starting a simulation interactively on the command line, and finding the output resembling: Memory required to begin solution : 754414. The number of words can be specified as memory=260M instead of memory=260000000, for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/.

===8cpu MPI Job===

<pre>sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k</pre>

The initial memory can also be specified for mpi jobs on the sqsub command line with "memory=" for the first master processor to decompose the problem and "memory2=" used on all processors including the master to solves the decomposed problem, where the values are specified as 4 bytes per word in single precision and 8 bytes per word in double precision. The number of words can be specified as memory=260M instead of memory=260000000 OR memory2=260M instead of memory2=260000000 for further details see https://www.d3view.com/2006/10/a-few-words-on-memory-settings-in-ls-dyna/. The initial values can be found by running simulation interactively on a orca compute node and checking the output for a line such as: Memory required to begin solution (memory= 464898 memory2= 158794 ) which could then be implemented for a job run in the queue by doing:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898 memory2=158794
</pre>

The specification of memory2 on sharcnet or compute canada clusters is not beneficial since the queue reserves the same memory per core for all nodes, such that the decomposition master node process cannot be allocated to have larger system memory. Therefore its sufficient to specify a single memory parameter by doing:

<pre>
export LSTC_MEMORY=auto
sqsub -r 1h -q mpi -o ofile.%J -n 8 --mpp=2G lsdyna_d i=airbag.deploy.k memory=464898
</pre>

where the LSTC_MEMORY variable will only allow the memory to grow for explicit simulations. The following slurm examples demonstrate how to prescribe the memory parameters exactly.

=Example Job=

Copy the airbag example to your account:

rsync -av orca.computecanada.ca:/opt/sharcnet/lsdyna/r901.109912/examples r901.109912_examples
cd r901.109912_examples/examples/misc/airbag
gunzip airbag.deploy.k.gz

==Graham (default modules)==

Please note that graham does not have the sharcnet legacy modules installed on it.

===Threaded Job ===

Sample submission script mysmpjob1.sh for 4 core single precision smp job using compute canada default cvmfs modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --nodes=1 # use one node
#SBATCH --cpus-per-task=4 # number threads
#SBATCH --mem=4000M # total memory
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=testsmp1 # %x = jobname
#SBATCH --output=slurm-%x-%N-%j.out
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# ----------------------------------------------------------
# IMPORTANT: to set license server do either:
# 1) export LSTC_LICENSE_SERVER=port@some.license.server
# 2) create ~/.licenses/ls-dyna.lic as shown above
# ----------------------------------------------------------
module load ls-dyna/9.1
ls-dyna_s i=airbag.deploy.k ncpu=4 memory=1000M
</pre>

where memory = 4000M / 4 (bytes/word) = 1000M words

===Mpi Job ===

Sample submission script mympijob2.sh for 4 core double precision mpp job using compute canada default cvmfs modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --ntasks=4 # number of ranks
#SBATCH --mem-per-cpu=4000M # memory per rank
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=testmpi2 # %x = jobname
#SBATCH --output=slurm-%x-%N-%j.out
# ----------------------------------------------------------
# IMPORTANT: to set license server do either:
# 1) export LSTC_LICENSE_SERVER=port@some.license.server
# 2) create ~/.licenses/ls-dyna.lic as shown above
# ----------------------------------------------------------
module load openmpi/1.8.8 ls-dyna-mpi/9.1
srun ls-dyna_d i=airbag.deploy.k memory=500M memory2=500M
</pre>

where memory = 4000M / 8 (bytes/word) = 500M words

==New Orca (legacy modules)==

Please note, while new orca has most compute canada cvmfs modules available by default it does not have the graham the ls-dyna or ls-dyna-mpi modules installed. Therefore currently the only way to run lsdyna is by using the legacy sharcnet lsdyna modules as shown in the following two smp and mpi examples.

===Threaded Job ===

Submission script mysmpjob3.sh to run 16 core single precision single node smp job with sharcnet legacy modules:

<pre>
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --nodes=1 # use one node
#SBATCH --cpus-per-task=16 # number threads
#SBATCH --mem=4000M # total memory
#SBATCH --time=00:30:00 # hrs:min:sec
#SBATCH --job-name=mysmpjob3-1node-16core # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
echo "SLURM_CPUS_PER_TASK= "$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
# ----------------------------------------------------------------
# IMPORTANT: to set license server for legacy modules:
export LSTC_LICENSE_SERVER=31047@license3.sharcnet.ca
# since ~/.licenses/ls-dyna.lic does NOT work with legacy modules
# ----------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load lsdyna/smp/r712.95028
lsdyna_s i=airbag.deploy.k ncpu=$SLURM_CPUS_PER_TASK memory=500M
</pre>

where memory = 4000M / 8 (bytes/word) = 500M words

===MPI Job ===

To run the sharcnet legacy module lsdyna/mpp/r712.95028 efficiently over several compute nodes with many cores requires using orca's xeon nodes as shown in mympijob4.sh below. To run lsdyna on orca's opteron nodes with module lsdyna/mpp/r712.95028 requires using only one opteron node as shown in mympijob5.sh below. The other legacy sharcnet lsdyna mpp modules, as shown in the Availability Table lsdyna/mpp/ls971.r85718, lsdyna/mpp/ls980B1.011113, lsdyna/mpp/r611.79036, lsdyna/mpp/r611.80542, lsdyna/mpp/r711.88920, lsdyna/mpp/r800.95359 and lsdyna/mpp/r901.109912 have yet to be tested - if you want to use one but cannot get it working please open a ticket. Note that lines containing a space between "# and SBATCH" are comments. To activate such line(s) required removal of the space ie) "#SBATCH". Its recommended to use module ls-dyna-mpi on graham instead since the machine is far newer, faster and reliable.

====Multi Node Slurm Script====

<pre>
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob4.sh
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --constraint=xeon
#SBATCH --ntasks=48
#SBATCH --mem-per-cpu=4000M
#SBATCH --time=00:30:00
#SBATCH --job-name=mympijob4-Xnodes-48cores-xeon # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
# -------------------------------------------------------------
# To set license server when using legacy sharcnet modules:
export LSTC_LICENSE_SERVER=port@some.license.server
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.
# -------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`
mpirun --host $MACHINEFILE --mca plm_slurm_args '--cpu_bind=none' lsdyna_s i=airbag.deploy.k memory=250M memory2=250M
</pre>

====Single Node Slurm Script====

<pre>
[roberpj@orc-login2:~/samples/lsdyna/airbag] cat mympijob5.sh
#!/bin/bash
#SBATCH --account=def-roberpj-ab
#SBATCH --constraint=opteron
#SBATCH --nodes=1
# SBATCH --ntasks=24
# SBATCH --mem=0
#SBATCH --ntasks=16
#SBATCH --mem-per-cpu=1000M
#SBATCH --time=00:30:00
#SBATCH --job-name=mympijob5-1node-16cores-opteron # jobname
#SBATCH --output=slurm-%x-%N-%j.out # %x = jobname
# -------------------------------------------------------------
# To set license server when using legacy sharcnet modules:
export LSTC_LICENSE_SERVER=port@some.license.server
# since ~/.licenses/ls-dyna.lic does not work with legacy modules.
# -------------------------------------------------------------
module purge --force
export MODULEPATH=/opt/sharcnet/modules
module load intel/12.1.3 openmpi/intel/1.6.5 lsdyna/mpp/r712.95028
MACHINEFILE=`srun hostname -s | tr '\n' ':' | sed 's/:/,/g' | sed 's/.$//'`
mpirun --host $MACHINEFILE lsdyna_s i=airbag.deploy.k memory=250M memory2=250M
</pre>

where memory = 1000M / 4 (bytes/word) = 250M words

==Legacy Clusters==

STEP1) The following shows sqsub submission of the [http://www.dynaexamples.com/examples-manual/misc/airbag airbag] example to the mpi queue. Its recommended to first edit airbag.deploy.k and change endtim to 3.000E-00 so the job runs long enough to perform the restart in steps 2 and 3 below:
<pre>
cp -a /opt/sharcnet/lsdyna/r611.79036/examples /scratch/$USER/test-lsdyna
cd /scratch/$USER/test-lsdyna/misc/airbag
gunzip airbag.deploy.k.gz
cp airbag.deploy.k airbag.deploy.restart.k
nano deploy.restart.k (reduce plot file creation frequency)
*DATABASE_BINARY_D3PLOT
$ dt lcdtdeploy.restart.k
1.000E-01 <--- change from 5.000E-04
export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet
module unload intel openmpi lsdyna
module load intel/11.1.069 openmpi/intel/1.6.4 lsdyna/mpp/r611.79036
sqsub -r 10m -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.k
</pre>

STEP2) With the job still running, use the echo command as follows to create a file called "D3KIL" that will trigger generation of restart files at which point the file D3KIL itself will be erased. Do this once a day if data loss is critical to you OR once or twice just before the sqsub -r time limit is reached. Further information can be found here http://www.dynasupport.com/tutorial/ls-dyna-users-guide/sense-switch-control:

<pre>
echo "sw3" > D3KIL
sqkill job#
</pre>

STEP3) Before the job can be restarted, the following two lines must be added to the airbag.deploy.restart.k file:

<pre>
*CHANGE_CURVE_DEFINITION
1
</pre>

STEP4) Now resubmit the job as follows using "r=" to specify the restart file:

<pre>
sqsub -r 1h -q mpi --mpp=2G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.restart.k r=d3dump01
</pre>

=General Notes=

==Graham Cluster==

==Command Line Use==

Parallel lsdyna jobs can be run interactively on the command line (outside the queue) for short run testing purposes as follows:

o Orca Development Node MPP (mpi) Example

ssh orca.sharcnet.ca
ssh orc-dev1 (or dev2,3,4)
module unload intel openmpi lsdyna
module load intel/12.1.3 openmpi/intel/1.6.2 lsdyna/mpp/r711.88920
mpirun -np 8 lsdyna_d i=airbag.deploy.k

o Orca Development Node SMP (threaded) Example

ssh orca.sharcnet.ca
ssh orc-dev1 (or dev2,3,4)
module unload intel openmpi lsdyna
module load lsdyna/smp/r711.88920
lsdyna_d i=airbag.deploy.k ncpu=8

==Memory Issues==

A minumum of mpp=2G is recommended although the memory requirement of a job may suggest much less is required. For instance setting mpp=1G for the airbag test job above will result in the following error when running a job in the queue:

<pre>
--------------------------------------------------------------------------
An attempt to set processor affinity has failed - please check to
ensure that your system supports such functionality. If so, then
this is probably something that should be reported to the OMPI developers.
--------------------------------------------------------------------------
</pre>

Another error message which can occur after a job runs for a while if mpp is chosen to small is:

<pre>
>>>>> Process 45 <<<<<
>>>>> Signal 11 : Segmentation Violation <<<<<
</pre>

To get an idea of the amount of memory a job used run grep on the output file:

<pre>
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep Memory
| Distributed Memory Parallel |
Memory size from command line: 500000, 500000
Memory for the head node
Memory installed (MB) : 32237
Memory free (MB) : 11259
Memory required (MB) : 0
Memory required to process keyword : 458120
Memory required for decomposition : 458120
Memory required to begin solution (memory= 458120 memory2= 230721)
Max. Memory reqd for implicit sol: max used 0
Max. Memory reqd for implicit sol: incore 0
Max. Memory reqd for implicit sol: oocore 0
Memory required to complete solution (memory= 458120 memory2= 230721)
</pre>

==Version and Revision==

Again run grep on the output file to extract the major and minor revision:

<pre>
[roberpj@orc-login1:~/samples/lsdyna/airbag] cat ofile.2063302.orc-admin2.orca.sharcnet | grep 'Revision\|Version'
| Version : mpp s R6.1.1 Date: 01/02/2013 |
| Revision: 78769 Time: 07:43:30 |
| SVN Version: 80542 |
</pre>

==Check License Status==

===Running and Queued Programs===

To get a summary of running and/or queued jobs use the lstc_qrun command as follows, where queued means the job has started running according to the sqsub command but its actually sitting waiting for license resources to come available. Once these are acquired the job will start running on the cluster and appear as a Running Program according to lstc_qrun ie)

<source lang="bash">
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet

[roberpj@hnd19:~] lstc_qrun
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
Running Programs
User Host Program Started # procs
-----------------------------------------------------------------------------
dgierczy 20277@saw61.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1
dgierczy 8570@saw32.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1
dscronin 25486@hnd6 MPPDYNA_971 Tue Apr 9 20:19 6
dscronin 14897@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 14971@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 15046@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6
dscronin 31237@hnd16 MPPDYNA_971 Tue Apr 9 21:53 6
dscronin 31313@hnd16 MPPDYNA_971 Tue Apr 9 21:54 6
dscronin 6396@hnd15 MPPDYNA_971 Tue Apr 9 21:54 6
csharson 28890@saw175.saw.sharc MPPDYNA_971 Wed Apr 10 16:48 6
No programs queued
</source>

To show license file expiration details, append "-R" to the command:

<source lang="bash">
[roberpj@hnd19:~] export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet

[roberpj@hnd19:~] lstc_qrun -R
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
**** LICENSE INFORMATION ****
PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE
---------------- ---------- ----- ------ ------ | -----
LS-DYNA_971 12/31/2013 - 966 1024 | 0
dgierczy 20277@saw61.saw.sharcnet 1
dgierczy 8570@saw32.saw.sharcnet 1
MPPDYNA_971 12/31/2013 - 966 1024 | 0
dscronin 25486@hnd6 6
dscronin 14897@hnd18 6
dscronin 14971@hnd18 6
dscronin 15046@hnd18 6
dscronin 31237@hnd16 6
dscronin 31313@hnd16 6
dscronin 6396@hnd15 6
csharson 28890@saw175.saw.sharcnet 6
LICENSE GROUP 58 966 1024 | 0

PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE
---------------- ---------- ----- ------ ------ | -----
LS-OPT 12/31/2013 0 1024 1024 | 0
LICENSE GROUP 0 1024 1024 | 0
</source>

===Killling A Program===

The queue normally kills lsdyna jobs cleanly. However its possible that licences for a job (which is no longer running in the queue and therefore no longer has any processes running on the cluster) will continue to be tied up according to the lstc_qrun command. To kill such a program determine the pid@hostname from the lstc_qrun command then run the following kill command. The following example demonstrates the procedure for username roberpj:

<pre>
[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj
roberpj 45312@orc329.orca.shar MPPDYNA Wed Dec 5 15:50 12

[roberpj@orc-login2:~] lstc_qkill 45312@orc329.orca.sharcnet.ca
Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable
Program queued for termination

[roberpj@orc-login2:~] lstc_qrun -R | grep roberpj
[roberpj@orc-login2:~]
</pre>

==Legacy Instructions==

The following binaries remain available on saw and orca for backward compatibility testing:

<pre>
[roberpj@orc-login2:~] cd /opt/sharcnet/local/lsdyna
[roberpj@orc129:/opt/sharcnet/local/lsdyna] ls ls971*
ls971_d_R3_1 ls971_d_R4_2_1 ls971_s_R3_1 ls971_s_R4_2_1 ls971_s_R5_1_1
ls971_d_R4_2_0 ls971_d_R5_0 ls971_s_R4_2_0 ls971_s_R5_0
</pre>

There are currently no sharcnet modules for these versions, hence jobs should be submitted as follows:

<pre>
module load lsdyna
export PATH=/opt/sharcnet/local/lsdyna:$PATH
export LSTC_LICENSE_SERVER=XXXXX@license3.uwo.sharcnet
cp /opt/sharcnet/local/lsdyna/examples/airbag.deploy.k airbag.deploy.k
SERIAL JOB: sqsub -q serial -r 1d -o ofile.%J ls971_d_R4_2_1 i=airbag.deploy.k
THREADED JOB: sqsub -q threaded -n 4 -r 1d -o ofile.%J ls971_d_R4_2_1 ncpu=4 para=2 i=airbag.deploy.k
</pre>

Note! the ls971_s_R3_1 and ls971_d_R3_1 binaries do not work, a fix is being looked for.

=References=

o LSTC LS-DYNA Homepage 
http://www.lstc.com/products/ls-dyna

o LSTC LS-DYNA Support (Tutorials, HowTos, Faq, Manuals, Release Notes, News, Links) 
http://www.dynasupport.com/

o LS-DYNA Release Notes 
http://www.dynasupport.com/release-notes

o LS-PrePost Online Documentation FAQ 
http://www.lstc.com/lspp/content/faq.shtml

o LSTC Download/Install Overview Page 
Provides links to binaries for LS-DYNA SMP/MPP, LS-OPT, LS-PREPOST, LS-TASC. 
http://www.lstc.com/download

==Memory Links==

o Implicit: Memory notes 
http://www.dynasupport.com/howtos/implicit/implicit-memory-notes

o LS-DYNA Support Environment variables 
http://www.dynasupport.com/howtos/general/environment-variables

o LS-DYNA and d3VIEW Blog (a few words on memory settings) 
http://blog2.d3view.com/a-few-words-on-memory-settings-in-ls-dyna/

o Convert Words to GB ie) memory=500MW (3.73GB) 
http://deviceanalytics.com/memcalc.php