From Documentation
Jump to: navigation, search
(Blanked the page)
(4 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= Key Features =
Unix Architecture page on Wikipedia
<blockquote>Files are stored on disk in a hierarchical file system, with a single top location throughout the system (root, or &quot;/&quot;), with both files and directories, subdirectories, sub-subdirectories, and so on below it.
With few exceptions, devices and some types of communications between processes are managed and visible as files or pseudo-files within the file system hierarchy. This is known as ''everything's a file''.
Doug McIlroy (inventor of Unix pipes)
<blockquote>This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface
= File system =
Key differences from Windows
* there are mount points instead of <tt>A:</tt>, <tt>C:</tt>, etc.,
* directories and files are case sensitive, and
* the separation character is <tt>/</tt> instead of <tt>\</tt>
What would appear as a separate media hierarchy in Windows (e.g., <tt>A:\MyDir\MyCode.c</tt>) simply appears under a separate directory (known as a mount point) in Unix (e.g., <tt>/media/disk/MyDir/MyCode.c</tt>).
== Root (<tt>/</tt>) ==
; <tt>/boot</tt>
: boot loader files
; <tt>/etc</tt>
: configuration files
; <tt>/dev</tt>
: device files
; <tt>/bin</tt>
: user programs required for booting
; <tt>/sbin</tt>
: system programs required for booting
; <tt>/lib{,32,64}</tt>
: libraries required for booting
; <tt>/usr</tt>
: programs, libraries, and such not required for booting
; <tt>/root</tt>
: superuser directory
; <tt>/home</tt>
: users directories (shared by all clusters)
; <tt>/tmp</tt>
: temporary files
; <tt>/var</tt>
: variable data (spool files, log files, etc.)
; <tt>/opt</tt>
: add on package directory
; <tt>/media</tt>
: mount point for removable media
; <tt>/proc</tt>
: process information pseudo-file system
; <tt>/sys</tt>
: system information pseudo-file system
== User (<tt>/usr</tt>) ==
The <tt>/usr</tt> directory is split off from the <tt>/</tt> directory mostly because disk space used to be precious.
; <tt>/usr/bin</tt>
: user programs not required for booting
; <tt>/usr/sbin</tt>
: system programs not required for booting
; <tt>/usr/lib{,32,64}</tt>
: libraries not required for booting
; <tt>/usr/games</tt>
: game programs
; <tt>/usr/share</tt>
: architecture independent data
; <tt>/usr/man</tt>
: on-line manuals
; <tt>/usr/src</tt>
: source code
; <tt>/usr/include</tt>
: header files
== User Local (<tt>/usr/local</tt>) ==
The <tt>/usr/local</tt> directory is a place to locally install programs without messing up <tt>/usr</tt>.
; <tt>/usr/bin</tt>
: user programs not required for booting
; <tt>/usr/sbin</tt>
: system programs not required for booting
; <tt>/usr/lib{,32,64}</tt>
: libraries not required for booting
; <tt>/usr/games</tt>
: game programs
; <tt>/usr/share</tt>
: architecture independent data
; <tt>/usr/man</tt>
: on-line manuals
; <tt>/usr/src</tt>
: source code
; <tt>/usr/include</tt>
: header files
; <tt>/work</tt>
: user data files (shared by all cluster)
; <tt>/scratch</tt>
: user temporary data files (local to each cluster)
= Devices =
Some of the special <tt>/dev</tt> files are
; <tt>/dev/null</tt>
: discards all data written and provides no data
; <tt>/dev/zero</tt>
: provides a constant stream of ''NULL'' characters
; <tt>/dev/random</tt>
: provides a stream of random characters
; <tt>/dev/urandom</tt>
: provides a constant stream of pseudo-random characters
= Commands =
Programs are run by specifying the command followed by the arguments separated by spaces.
<blockquote>''program'' <tt>[</tt>''argument''...<tt>]</tt>
By convention, arguments are switches followed by strings (e.g., regexps, paths, file names, etc.). Switches are usually single dashes followed by letter for each switch or a double dash followed by a descriptive string (e.g., <tt>rm -fr mydir</tt> or <tt>rm --force --recurse mydir</tt>). Most commands also understand
; <tt>-</tt>
: as a file name means read or write to the terminal
; <tt>--</tt>
: the end of switches and the start of the strings (in case the string needs to start with <tt>-</tt> or <tt>--</tt>).
== Help ==
Traditionally man pages (a single help page) have been the de facto documentation source, however, some software suites have been switching to info pages (a collection of hyperlinked pages). Help for the shell built in commands is available by the built in <tt>help</tt>.
; <tt>man</tt> ''command''
: on-line reference manuals
; <tt>apropos</tt> <tt>[-a]</tt> ''keyword'' ...
: search on-line reference manuals (same as <tt>man -k</tt>)
; <tt>info</tt> ''item''
: info documents
== Directories ==
The current directory is <tt>.</tt> and the parent directory is <tt>..</tt>.
; <tt>pwd</tt>
: current directory
; <tt>cd</tt> ''directory''
: change directory
; <tt>mkdir</tt> ''directory''
: make directory
; <tt>rmdir</tt> ''directory''
: remove directory
== Files ==
Files beginning with <tt>.</tt> are considered hidden and not normally shown.
; <tt>ls</tt> <tt>[-a]</tt> <tt>[-l]</tt> ''destination''
: list files
; <tt>cp</tt> <tt>[-a|-p]</tt> <tt>[-r]</tt> <tt>[-s]</tt> ''source'' ... ''destination''
: copy files
; <tt>ln</tt> <tt>[-s]</tt> ''target'' ''name''
: link to file
; <tt>mv</tt> ''source'' ... ''destination''
: move files
; <tt>rm</tt> <tt>[-r]</tt> <tt>[-f]</tt> ''destination'' ...
: remove files
== Permissions ==
Standard permissions are <tt>r</tt>ead, <tt>w</tt>rite, and e<tt>x</tt>ecute for <tt>u</tt>ser, <tt>g</tt>roup, and <tt>o</tt>ther. They are frequently abbreviated as three octal numbers (0=000, 1=001, 2=010, 3=011, 4=100, 5=101, 6=110, 7=111) corresponding to user read, write, and execute; group read, write, and execute; other read, write, and execute.
For directories, <tt>r</tt>ead allows the contents to be listed, <tt>w</tt>rite allows files to be added or removed, and e<tt>x</tt>ecute allows the directory to be traversed.
; <tt>chmod</tt> <tt>[u|g|o|a]</tt>...<tt>[+|-|=][r|w|x|X]</tt>... <tt>[-R]</tt> ''destination'' ...
: change mode (user/group/other permissions)
; <tt>chown</tt> <tt>[-R]</tt> ''user'' ''destination'' ...
: change owner
; <tt>chgrp</tt> <tt>[-R]</tt> ''group'' ''destination'' ...
: change group
; <tt>setfacl</tt> <tt>[-m|-x]</tt> <tt>[-R]</tt> <tt>[[u|g|o|m]...:</tt>''user''<tt>:[r|w|x|X]...]</tt> ''destination'' ...
: set file access control list (individual users)
; <tt>getfacl</tt> ''destination'' ...
: get file access control list (individual users)
== View Files ==
The <tt>space</tt> key will advance a page and the <tt>q</tt> key will quit in <tt>more</tt> and <tt>less</tt>. In addition, the arrow keys will move in the appropriate direction in <tt>less</tt>.
; <tt>more</tt> ''file''
: view one page at a time
; <tt>less</tt> ''file''
: view forward and backwards
; <tt>cat</tt> <tt>[</tt>''file'' ...<tt>]</tt>
: concatenate files in sequence
; <tt>head</tt> [-n ''lines''] <tt>[</tt>''file'' ...<tt>]</tt>
: first part of files
; <tt>tail</tt> [-n ''lines''] [-f] <tt>[</tt>''file'' ...<tt>]</tt>
: last part of files
; <tt>paste</tt> <tt>[-d</tt> ''deliminator''<tt>]</tt> <tt>[</tt>''file'' ...<tt>]</tt>
: concatenate files in parallel
; <tt>cut</tt> [<tt>-d</tt> ''deliminator''<tt>]</tt> [<tt>-f</tt> ''range''<tt>]</tt> <tt>[</tt>''file'' ...<tt>]</tt>
: extract columns
; <tt>sort</tt> <tt>[-g]</tt> <tt>[-f]</tt> <tt>[-u]</tt> <tt>[</tt>''file'' ...<tt>]</tt>
: sort lines
== Comparison ==
Digests are numbers computed from the content of files such that it is extremely difficult to come up with two different files with the same number.
; <tt>diff</tt> <tt>[-w]</tt> <tt>[-i]</tt> <tt>[-u</tt> ''number''<tt>|-y]</tt> ''file1'' ''file2''
: compare files line by line
; <tt>sdiff</tt> <tt>[-W]</tt> ''file1'' ''file2''
: compare files side by side (similar to <tt>diff -y</tt>)
; <tt>md5sum</tt> <tt>[</tt>''file'' ...<tt>]</tt>
: compute MD5 digest
; <tt>sha256sum</tt> <tt>[</tt>''file'' ...<tt>]</tt>
: compute SHA256 digest
== Searching ==
; <tt>egrep</tt> <tt>[-i]</tt> <tt>[-v]</tt> ''regexp'' <tt>[</tt>''file'' ...<tt>]</tt>
: find lines matching regexp in files (same as <tt>grep -E</tt>)
; <tt>fgrep</tt> <tt>[-i]</tt> <tt>[-v]</tt> ''strings'' <tt>[</tt>''file'' ...<tt>]</tt>
: find lines matching strings in files (same as <tt>grep -F</tt>)
; <tt>find</tt> ''directory'' ... ''predicates''
: find files satisfying predicates in directories
== Process ==
Each process (a running programs) is identified by a unique number.
; <tt>ps</tt> <tt>[-A|-U</tt> ''user''<tt>]</tt> <tt>[-H]</tt> <tt>[-f]</tt>
: process list
; <tt>kill</tt> <tt>[-s</tt> ''signal''<tt>]</tt> ''process'' ...
: signal process
; <tt>nohup</tt> ''command''
: command will continue to run even if user disconnects on clusters which are set up for that to happen.  Most clusters nowadays don't require this command.
; <tt>nice</tt> ''command''
: low priority command
== Remote ==
; <tt>ssh</tt> <tt>[</tt>''user''<tt>@]</tt>''host'' <tt>[</tt>''command''<tt>]</tt>
: login to remote system
; <tt>scp</tt> <tt>[[</tt>''user''<tt>@]</tt>''host''<tt>:]</tt> ''source'' ... <tt>[[</tt>''user''<tt>@]</tt>''host''<tt>:]</tt>''destination''
: copy remote files
; <tt>unix2dos</tt> ''file'' ...
: convert to DOS line breaks
; <tt>dos2unix</tt> ''file'' ...
: convert to Unix line breaks
== Other ==
; <tt>sleep</tt> ''seconds''
: waits given number of seconds
; <tt>echo</tt> <tt>[-n]</tt> <tt>[-e]</tt> ''strings''
: prints strings
; <tt>test</tt> ''tests''
: perform various string (e.g., equality) of file (e.g., existence) tests
= Editors =
The two most popular Unix editors are <tt>vi</tt> and <tt>emacs</tt>. Both are extremely powerful and very complex. A simpler editor is <tt>nano</tt>.
; <tt>vi</tt> <tt>[</tt>''file'' ...<tt>]</tt>
: common Unix editor
; <tt>emacs</tt> <tt>[-nw]</tt> <tt>[</tt>''file'' ...<tt>]</tt>
: common Unix editor
; <tt>nano</tt> <tt>[</tt>''file'' ...<tt>]</tt>
: simple U
== Vi ==
Vi distinguishes between command and insert mode. Command mode allows you to move around and enter commands. Insert mode allows you to edit text.
; <tt>:h</tt>
: help
; <tt>:w[!] [</tt>''file''<tt>]</tt>
: write file (excalmation forces it)
; <tt>:e</tt> ''file''
: edit file
; <tt>:q[!]</tt>
: quit Vi (exclamation forces it)
; <tt>:n[!]</tt>
: next file (excalmation forces it)
; <tt>[a|A]</tt>
: append after cursor or at end of line
; <tt>[i|I]</tt>
: insert (capital for beginning of line)
; <tt>[c[w|c]|C]</tt>
: change word/line or to end of line
; <tt>[d[w|d]|D]</tt>
: delete word/line or to end of line
; <tt>[y[w|y]|Y]</tt>
: copy word/line or to end of line
; <tt>[p|P]</tt>
: paste before or after cursor/line
; <tt>J</tt>
: join lines
; <tt>[u|U]</tt>
: undo (captial for current line)
; <tt>ESC</tt>
: revert to command mode
== Emacs ==
Emacs is a more traditional single mode editor. Partially typed entries can be completed by pressing ''TAB'' (twice to list).
; <tt>CTRL+h</tt>
: help (<tt>b</tt> list keys and <tt>k</tt> describes keys)
; <tt>CTRL+g</tt>
: abort current operation
; <tt>CTRL+[1|2|3]</tt>
: single window or split vertical/horizontal window
; <tt>CTRL+x</tt> <tt>CTRL+s</tt>
: save current buffer
; <tt>CTRL+x</tt> <tt>b</tt>
: switch current buffer
; <tt>CTRL+x</tt> <tt>k</tt>
: quit current buffer
; <tt>CTRL+x</tt> <tt>CTRL+c</tt>
: quit Emacs
; <tt>CTRL+SPACE</tt>
: mark start of region
; <tt>ALT+w</tt>
: copy from start of region to cursor
; <tt>CTRL+y</tt>
: past copied region
; <tt>CTRL+k</tt>
: delete to end of line or line if start of line
; <tt>CTRL+s</tt>
: search for text
; <tt>CTRL+_</tt>
: undo
; <tt>ALT+x</tt>
: enter command (''TAB'' twice to list)
= Command Line =
The shell is a command line interpreter that lets users run programs. It provides ways to start programs and to manipulate/setup the context in which they run. The main parts of this are
* arguments,
* environment,
* standard input (stdin),
* standard output (stdout),
* standard error (stderr), and
* return value
A standard command looks like so
<blockquote>''command'' <tt>[&lt;</tt>''stdinfile''<tt>]</tt> <tt>[&gt;[&gt;]</tt>''stdoutfile''<tt>]</tt> <tt>[2&gt;[&gt;]</tt>''stderrfile''<tt>]</tt> <tt>[&amp;]</tt>
== Arguments ==
Options passed to the program to tweak it's behaviour. Traditionally switches (e.g., <tt>-xzf</tt> or <tt>--extract</tt> <tt>--gzip</tt> <tt>--file</tt>) followed by strings (e.g., regexp, paths, file names, etc.). Partially typed file names and directories can be completed by pressing ''TAB'' (twice to list).
; ...<tt>{</tt>...<tt>}</tt>... (brace expansion)
: if not quoted, expands once for each comma separated list or once for each number in <tt>..</tt> separated range
; <tt>~</tt>... (tilde expansion)
: if not quoted, expands to home directory of user following the tilde or the current user if no user specified
; <tt>${...}</tt> (parameter and variable expansion)
: if not single quoted, expands to environment variable specified or the corresponding parameter if number specified (<tt>{</tt> and <tt>}</tt> are not always required)
; <tt>$(...)</tt> (command substitution)
: if not single quoted, expands to output for command (<tt>`</tt>...<tt>`</tt> is an alternative syntax)
; <tt>$((...))</tt> (arithmetic substitution)
: if not single quoted, expands to evaluated result of the expression
; ... (word splitting)
: if not quoted, splits into separate arguments anywhere an IFS character (by default space, tab, and newline) occurs
; ...<tt>[*|?|[</tt>...<tt>]]</tt>... (path name expansion)
: if not quoted, is considered a pattern and replaced with matching file names (<tt>*</tt> matches any string, <tt>?</tt> matches any character, and <tt>[</tt>...<tt>]</tt> matches all the enclosed characters)
== Quoting ==
Special characters can be escaped with <tt>\</tt> to remove their special meaning. Single and double quoting strings affect escaping as well as which expansions and substitutions are preformed.
; <tt>'</tt>...<tt>'</tt>
: no expansion or substitutions is preformed
; <tt>&quot;</tt>...<tt>&quot;</tt>
: only escaping, parameter and variable expansion, command substitutions, and arithmetic substitutions occur
== Environment ==
A set of key value pairs (e.g., <tt>USER=root</tt>) that programs can look up and use. Each program gets a fresh copy (i.e., changing it will not change the original) of all environment variables marked for export.
; ''key''<tt>=</tt>''value''
: make a local environment variable
; <tt>export</tt> ''key''<tt>[=</tt>''value''<tt>]</tt>
: mark an environment variable for export
; <tt>unset</tt> *key
: delete an environment variable
Two important environment variables are
; <tt>PATH</tt>
: list of <tt>:</tt> separated directories to look for programs in
; <tt>LD_LIBRARY_PATH</tt>
: list of <tt>:</tt> separated directories to look for libraries in (ahead of the system defaults specified in <tt>/etc/</tt>)
== Input and Output ==
Programs are run with a standard place to read input from, a standard place to write output to, and a standard place to write error messages to. By default these are all the terminal window in which the program is run. This can be changed via
; <tt>&lt;</tt> ''file''
: read standard input from file
; <tt>[&gt;|&gt;&gt;]</tt> ''file''
: write standard output to file (overwriting or appending)
; <tt>[2&gt;|2&gt;&gt;]</tt> ''file''
: write standard error to file (overwriting or appending)
; <tt>[&amp;&gt;|&amp;&gt;&gt;]</tt> ''file''
: write standard output and error to file (overwriting or appending)
== Status ==
Programs return an integer exit status. The stats of the most recent executed foreground command is available as <tt>$?</tt>.
: TORQUE Resource Manager scheduler exit code, job could not start ([ list])
; 0
: program completed successfully
; 1...127
: program specific error code
; 128...255
: program terminated by signal 127+''signal''
== Job Control ==
Programs run in the foreground by default. Background jobs will suspended if they require input. Existing jobs will be sent <tt>SIGHUP</tt> when the shell exits.
; <tt>jobs</tt>
: list jobs
; <tt>fg</tt> ''id''
: switch job to foreground
; <tt>bg</tt> ''id'' ...
: switch jobs to background
; <tt>disown</tt> ''id'' ...
: release jobs from job control
Foreground jobs usually respond to the following key combinations
; ''CTRL+Z''
: suspend program
; ''CTRL+C''
: abort program
; ''CTRL+D''
: end of input
== Multiple Commands ==
Commands can be combined in several ways.
; ... <tt>;</tt> ...
: run first command and then second (same as pressing ''ENTER'')
; ... <tt>&amp;</tt> ...
: run first command in background at the same time as second
; ... <tt>|</tt> ...
: run first command in background with its output going to the second as input
; ... <tt>&amp;&amp;</tt> ...
: run first command and then second only if first returns success
; ... <tt>||</tt> ...
: run first command and then second only if first returns failure
; <tt>{</tt>...<tt>}</tt>
: group command in current shell -- has to end with <tt>;</tt> or newline
; <tt>(</tt>...<tt>)</tt>
: group command in sub shell -- does not have to end with <tt>;</tt> or newline
= Scripting =
Executable text files that start with <tt>#!</tt>''command'' (#!/bin/bash` for shell scripts) are run as ''command'' ''file''.
== Parameters ==
; <tt>$#</tt>
: number of parameters
; <tt>$0</tt>
: name of shell or shell script
; <tt>$</tt>''number''
: positional parameter
; <tt>$*</tt>
: all positional parameters (in double quotes expands as one arguement)
; <tt>$@</tt>
: all positional parameters (in double quotes expands as separate arguements)
The following functions manipulate parameters
; <tt>shift</tt> <tt>[</tt>''number''<tt>]</tt>
: drop specified number of parameters (one if unspecified)
; <tt>set</tt> ''parameter'' ...
: set parameters to given parameters
== Programming ==
; <tt>if</tt> ''command'' ...<tt>; then</tt> ''command'' ...<tt>; [elif</tt> ''command'' ...<tt>; then</tt> ''command'' ...<tt>;]</tt> ... <tt>[else</tt> ''command'' ...<tt>;] fi</tt>
: conditionally run commands depending on success <tt>if</tt> and <tt>elif</tt> commands
; <tt>for</tt> ''key'' <tt>in</tt> ''value'' ...<tt>; do</tt> ''command'' ...<tt>; done</tt>
: for each value, set key to value and run commands
; <tt>while</tt> ''command'' ...<tt>; do</tt> ''command'' ...<tt>; done</tt>
: repeatedly run commands until <tt>while</tt> commands fail
; <tt>case</tt> ''value'' <tt>in [</tt>''pattern'' <tt>[|</tt> ''pattern''<tt>]</tt>... <tt>)</tt> ''command'' ...<tt>;;]</tt> ... <tt>esac</tt>
: run commands that where first pattern matches (same as path name expansion)
; <tt>continue</tt> <tt>[</tt>''number''<tt>]</tt>
: next iteration of enclosed loop (last if not specified)
; <tt>break</tt> <tt>[</tt>''number''<tt>]</tt>
: exit enclosed loop (last if not specified)
; <tt>function</tt> ''name'' <tt>{</tt> ''command''<tt>; }</tt> ...
: create a command that runs the commands with passed parameters
; <tt>return</tt> <tt>[</tt>''number''<tt>]</tt>
: return from function with given exit status (last command if not specified)
; <tt>exit</tt> <tt>[</tt>''number''<tt>]</tt>
: quit shell with given exit status (last command if not specified)
= Regular Expressions =
Regular expressions are strings where several of the non-alphanumeric characters have special meaning. They provide a concise and flexible means for matching strings, and are used by several Unix programs.
== Anchoring ==
; <tt>^</tt>
: match start of line
; <tt>$</tt>
: match end of line
== Characters ==
; ''character''
: the indicated character
; <tt>.</tt>
: any character
; <tt>[</tt>...<tt>]</tt>
: any character in the list or range (<tt>^</tt> inverts)
== Combining ==
; <tt>(</tt>...<tt>)</tt>
: group
; ...<tt>|</tt>...
: match either or
== Repetition ==
; <tt>?</tt>
: match zero or one times
; <tt>*</tt>
: match zero or more times
; <tt>+</tt>
: match one or more times
; <tt>{</tt>...<tt>}</tt>
: match a range of times

Latest revision as of 12:37, 3 September 2021