Original question:
>I have been asked a question by one of our application people who is having
>problems with a third party product that bases some of its log file
>creation/reading on process ids. His question is what is the default limit
>for pid numbers and how are they allocated? The version of o/s is 4.0e.
Thanks to all that replied, but most thanks go to Dr. Tom Blinn, for really
getting to the guts of the answer. In summary the maximum PID number
available is 32767 but whether your system gets that far depends on other
kernel params you have set see details below. Also PID numbers are reused in
a cyclical fashion - i.e. starts from the bottom up then recycles round to
the bottom and uses the next available PID. I've attached Tom's responses to
answer the question in more detail.
Dr. Tom wrote:
It's tunable -- either in the kernel configuration file, or in the file
/etc/sysconfigtab (I believe it's in the proc subsystem).
Look at the tuning guide; it should be documented there, or else in the
system management guide. And look at param.c (in your kernel build),
it computes a variable value for "int nproc" from MAXPROC if that is
defined (as it would be if a line "maxproc <int value>" appears
in your kernel config file).
There is also a MAXUPRC parameter. Suffice it to say that the answer
isn't trivial, but the value is relatively large.
So, what is the real problem that your application programmer is seeing,
because it may be easier to address the problem than fully explain all
of the details about how the real limit of number of PIDs is determined.
(Suffice it to say it's also bounded by the maximum number of threads,
and there may be other limiting factors, such as maximum processes per
user, that are involved.)
>I've had a couple of replies back, thanks Dr. Tom Blinn and Nikola
>Milutinovic, which seem to pose more questions than answers! I thought this
>might be a trivial question, but it appears not. I shall try and clarify my
>question a bit further just to make sure I'm not confusing folks.
>
>The application experiencing the problem is Tivoli Maestro (a job scheduler
>amongst other things) that records log files by PID. What is occurring is
>that the available PIDs are being recycled too fast causing Maestro to
>become confused. I'm aware that the kernel allocates pids up to a certain
>value before restarting the numbering process and skipping over PIDs that
>are still in use. On a HP system I believe that value is something in the
>range 0-32767, which the application folks says works fine for them in that
>instance. However I have seen PIDs on this system in the order of 32000 -
so
>I think it might have the same range as the HP. So I just need to clarify
if
>the range of PIDs that can be assigned is 0-32767? And can it be increased?
>
>I've scoured the config guide and tuning manual but cannot find any
>information specific to this. The task-max value Nikola was referring to
>refers to the maximum number of simultaneous tasks not the allocation of
the
>PID range.
>
>Sorry Tom, but your reply had me scratching my head a lot! I was hoping you
>might say "it's this by default...and it's tuneable by changing this
>parameter.", or something like that. Again I think this may be referring to
>maximum processes, not the range of PID numbers assigned.
> The version of o/s is 4.0e.
And you thought this would be easy :^)
Looking at the kernel sources, and looking for references to "nproc", I
find in a module called kernel/bsd/proc_cfg.c in proc_configure I find
case CFG_OP_CONFIGURE:
if (!nproc) {
nproc = 20 + 8 * maxusers;
if (nproc > PID_MAX)
nproc = PID_MAX;
}
if (nproc < (task_max-1))
nproc = task_max-1;
if (!thread_max)
thread_max = nproc * 2;
if (!task_max)
task_max= (nproc + 1);
portmax = (task_max * 3 + thread_max) /* kernel */
+ (thread_max * 2) /* user */
+ 20000; /* slop for objects */
port_hash_max_num = 50 * portmax;
port_max_num = portmax;
port_reserved_max_num = portmax;
set_max_num = task_max + thread_max + 200; /* Max port sets */
if (maxuprc > task_max - 10) {
/*
* Do not allow the number of processes per-user to be larger
* than the total number of tasks (and leave room for some root
* processes).
*/
maxuprc = task_max - 10;
}
if (maxuthreads > thread_max - 10) {
/*
* Do not allow the number of threads per-user to be larger
* than the total number of threads (and leave room for some
root
* threads).
*/
maxuthreads = thread_max - 20;
}
if (maxuthreads != 0 && maxuthreads < maxuprc) {
/*
* We need at least as many threads as procs on a per-user
basis.
*/
maxuthreads = maxuprc;
}
if (thread_max < task_max) {
/*
* We need at least as many threads as tasks.
*/
thread_max = task_max;
}
/*
* Now initialize all the other variables that are scaled off
* the reconfigurable variables.
*/
/* number of pids is rounded up to a multiple of 2 to allow for
* fast lookup in pfind().
*/
npid = 3 * task_max;
npid = 1 << (msb(npid - 1) + 1);
pid_entry_mask = npid - 1;
if (npid > (PID_MAX+1)) {
npid = PID_MAX+1; /* already a power of 2 -1 */
pid_entry_mask = PID_MAX;
}
#ifdef OSF_1.2
max_free_files = 16;
/*
* The configuring of lvm should be moved to its own subsystem.
*/
if (lvm_var_init_ptr)
(*lvm_var_init_ptr)();
#endif /* OSF_1.2 */
/*
* Configure autonice time value in usecs.
*/
autonice_usecs = 1000000 * autonice_time;
/*
* Check for extended UID's enabled
*/
if (enable_extended_uids) {
uid_max = (uid_t) LARGE_UID_MAX; /* 256K - 1 */
reset_uid_configurability();
}
break;
-----------------------------
So, it looks sort of like npid (the number of pids) is the next larger
power of 2 greater than 3 times task_max unless that exceeds PID_MAX,
and if you look in /usr/include/limits.h you should find
#define PID_MAX SHRT_MAX
and in /usr/include/machine/machlimits.h you should find
#define SHRT_MAX 32767
but the number of PIDs that actually are used depends on the task_max
so if you happen to have task_max set small then PIDs will be limited
and if you have lots of active tasks in spite of a small task_max then
the PIDs will in fact recycle quickly as new processes are created.
Should I say "duh" at this point? Not really -- it's not obvious that
there would be a PID lookup table, but if you think it through, since
any table takes up space, you want to limit it. When assigning a PID
to a new process (task), you look for the next free slot in the table
and if there are relatively few free slots, you're going to re-use the
numbers relatively quickly. Remember, PID_MAX is an upper limit, but
it's not what is guaranteed to be used on a given system, the limit on
the numbers before they get reused might be much smaller. And, if you
have a large enough task_max that you'd be over PID_MAX using the rule
of "next power of two greater than three times task_max" and you've got
a LOT of active processes that stay around, the recycle rate is going
to be really quick.
> Tom,
>
> I forgot to ask how the PID table is used, i.e. when the maximum PID is
used
> does it then restart from 0 using the next available PID, or is there some
> other wacky way it reuses PIDs.
>
> Thanks,
> Dave.
To know FOR SURE, I'd have to go examine the code. However, I believe the
mechanism used is taken nearly verbatim from Berkeley UNIX (4.2 or 4.3 BSD)
and that it looks for the next higher available PID and when it gets to the
top of the range it wraps around to the beginning again and searches from
there. This belief is as much based on empirical evidence as anything.
Underneath the covers, a PID is a fairly artificial construct in Tru64 UNIX.
It prepresents a process context, but that in turn has a virtual memory map
or address space (managed as a VM or virtual memory object) as well as one
or more threads of execution. The "PID" construct is maintained largely for
compatibility with the legacy UNIX model of "process ID" being relevant; in
Tru64 UNIX, it's really just a way to gather together pointers to a bunch of
other things that are more relevant. So the allocation of the "process ID"
numbers is done relatively infrequently, and a sequential search for the
next
available number is pretty straight-forward. I suspect the restriction of
the
MAX_PID to 32767 (16 bit value) is a carry-over from older UNIX
implementations
and in my recollection it changes in V5.0 (it can impact user space things,
such as programs that think they know the size of a PID or believe that it
takes at most 5 characters to represent the number).
Received on Thu Feb 17 2000 - 11:16:16 NZDT