prologue and epilogue for PBS
using PAM to enforce PBS node assignments
May, 2001
Willy Weisz weisz@vcpc.univie.ac.at

Purpose
-------
In order to ensure that jobs use only nodes that PBS has assigned to
them, any kind of remote login/execution on nodes not assigned must
be blocked. As long as kernel patches and/or daemons that are PBS-aware
are not available this function must make use of authentication and
authorization software that is able to be controlled by PBS scripts.
The solution used on the cluster "gescher" at VCPC makes use of the
following facilities:
- prologue/epilogue feature of OpenPBS (version 2.3.12)
- PAM (Pluggable Autentication Modules)
  <http://www.us.kernel.org/pub/linux/libs/pam/>.

As a side effect the entries in the file limits.conf are used to control
any purging of left-overs actions, and to allow the use of idle cycles of
non-assigned nodes.  

HOWTO
-----

First the following files in the directory /etc/pam.d of any available node
have to be amended:
- login
- rsh
- sshd (if ssh is used to login to the target node)
- rexec
- any other name of a remote login/exection service on target nodes

You have to add the line:

  session  required       /lib/security/pam_limits.so

after the line

  session  required       /lib/security/pam_unix.so

in each of those files.

Then you have to insert an entry

  *               hard    maxlogins       0

in the file /etc/security/limits.conf on all nodes. If any users/groups
except for root are to be allowed access to nodes not allocated to PBS job
of their own lines like e.g.:
  user-name       hard    maxlogins       10 # for a user
  @group-name     hard    maxlogins       10 # for a group
have to be inserted. A last line (preferably a comment line) has to be
defined after this static block; an identifying part of it has to be defined
in the epilogue script. I'm just using the last line of the distributed
example file which reads: # End of file. There must be no line after this one
when the node isn't assigned to any job!

Each job that gets the node assigned will add a line of the form:
  user-name hard maxlogins 18 #PBS_job_id
and the epilogue script will delete this line.
You may ask: Why 18? That's _life_.

At last you have to install the files prologue, epilogue and any other program
or script called by them to the directory /var/spool/pbs/mom_priv, make them
owned by root and have access mode 700. In our case there are two more scripts:
- epilogue_clean_up which purges the node of all left-overs from the job (if no
  other job of the user is active on the node)
- epilogue_harness which contains the applications that use the idle cycles
  between two job assignments of the node and that are run as non-PBS jobs.

Questions, suggestions, comments, bug reports
---------------------------------------------
Send to $Author

Author
------
Willy Weisz

European Centre for Parallel Computing at Vienna (VCPC)
                 Liechtensteinstrasse 22
                 A-1090 Wien
                 http://www.vcpc.univie.ac.at
Tel: (+43 1) 4277 - 38824       Fax: (+43 1) 4277 - 38818
                e-mail: weisz@vcpc.univie.ac.at