Tuesday, December 9, 2008

Do You Know Your Limits!?

To continue further our discussion of what are process resource limits, let’s see some real figures. Try putting following two printk() statements in FirstModuleSource.c posted here.

printk (KERN_INFO "Process Name: %s", current -> comm);
printk(KERN_INFO "current limit- # processes: %ul\n", current -> rlim[RLIMIT_NOFILE].rlim_cur);
printk(KERN_INFO "maximum limit- # processes: %ul\n", current -> rlim[RLIMIT_NOFILE].rlim_max);

These statements print soft and hard limits (for the maximum number of files the current user can open) imposed on current process (which is insmod). On similarly printing the limits of other resources, we see that all values of soft and hard limits are assigned limits from array INIT_RLIMITS (see #line 24 in snapshot of resource.h posted here).


To see the difference between resource limits of a kernel process with an ordinary user process, we shall print the resource limits of a user process by adding following lines of code in the FirstModuleSource.c

int some_pid = 3630; /*reader should change this value appropriately*/
struct task_struct *task = find_task_by_pid(some_pid);
printk (KERN_INFO "Process Name: %s", task -> comm);
printk (KERN_INFO "current limit- # processes: %ul\n", task -> rlim[RLIMIT_NOFILE].rlim_cur);
printk (KERN_INFO "maximum limit- # processes: %ul\n", task -> rlim[RLIMIT_NOFILE].rlim_max);

Here, 3630 is the pid of the shell command tail, which I got by running tail and looking for it in output of ps. The find_task_by_pid() kernel function returns the process descriptor belonging to some_pid. We can specify pid of any user process running in background; but that pid must be alive while the module is inserted.

On my i386/RHEL-4 machine, both sets of statements given above produce following output, respectively:

Nov 15 21:27:32 localhost kernel: Process Name: insmod<6>
Nov 15 21:27:32 localhost kernel: current limit- # processes: 1024l
Nov 15 21:27:32 localhost kernel: maximum limit- # processes: 1024l

Nov 15 21:27:32 localhost kernel: Process Name: tail<6>
Nov 15 21:27:32 localhost kernel: current limit- # processes: 1024l
Nov 15 21:27:32 localhost kernel: maximum limit- # processes: 1024l

On similarly printing the limits of other resources and analyzing the output, we see that all the user/kernel processes’ resource limits are initialized with the same set of values which are specified by array INIT_RLIMITS of resource.h. This small experiment suggests that by default, the kernel imposes similar resource limits on any user/kernel mode process. To verify this further, I changed some values in INIT_RLIMITS, and then ran the same program again after compiling the kernel. The changed values were reflected as-is in the output.

How are Limits Assigned?
Unless a user is assigned custom limits by the superuser, all the processes belonging to that user are assigned default limits as follows.

The first Linux process which is created during system bootup is the swapper process. After kernel initialization, the swapper creates another process init, which then takes full control of all the system activities and creates more processes. swapper's data structures are initialized through init_task.h. It is the only process, for which data structures are statically allocated like this. Data structures for all other processes are dynamically allocated.

As we see in the snapshot here, INIT_TASK macro initializes the task_struct of swapper. At #line 80, rlim field of task_struct has been initialized with INIT_RLIMITS structure defined in resource.h (see last post), #line 82 is the name of the process.
Since every process inherits properties of its parent process when it is created, init receives the same values as swapper for all of its fields, including rlim. Further, when init creates more processes, these values are inherited by every child process and to their child processes and so on. This goes on until and unless
  1. A process executes setrlimit() system call
  2. The superuser has configured custom values for a user.
We've already covered the first point in last post. For information on second point, do a `man limits.conf' and open your /etc/security/limits.conf. The custom values for a user/group are specified in this configuration file by the system superuser.

Therefore, if a particular limit for a resource is set for user X in limits.conf, that limit shall be assigned to all the processes created by user X, at the time of process creation.

** Try shell command `ulimit –a’ to see all soft limits for current user. `ulimit -s unlimited’ sets the stack size soft limit to maximum for all the processes belonging to current session of the current user. (Current session is nothing but the current shell on which ulimit is entered; the changes by ulimit do not affect other shells.) If a user/resource doesn't appear in limits.conf, ulimit shall display default limits (i.e. the values of INIT_RLIMIT array) for that user/resource.

Shweta is going to share some findings on this very soon... on the same page.
-------------------------------------------------------------

Shweta added:

The above explanation can be understood with the help of a small and quick experiment. This should make clear this behavior of resource limits assignment. This experiment has two parts:

First, clear the entries belonging to user X from limits.conf, (if any), then login to your machine with user account X and write a user program to print value of RLIMIT_NOFILE using getrlimit(). Also check this value using 'ulimit'. Both outputs should show the value specified in INIT_RLIMIT array. Now change the value of RLIMIT_NOFILE using setrlimit(), fork() a new process in the same program and print the value of RLIMIT_NOFILE for child process using getrlimit(). This should output the value changed in parent process using setrlimit().

Now for the second part of the experiment, make an entry for RLIMIT_NOFILE for user X in limits.conf and repeat part-1 of the experiment. The outputs this time would differ in the way that wherever previous experiment printed values from INIT_RLIMIT array, this experiment would print values specified in limits.conf file.

Saturday, November 22, 2008

Process Resource Limits

Having seen the task_struct structure representing a process in kernel, in this post we shall discuss one very important field of task_struct, named rlim (#line 460 in the snapshot of sched.h in last post). A process can never access the system resources how much ever it wants. Linux kernel imposes limits on the amount of resources a process is allowed to use, so that a single process cannot cause other processes to starve by itself consuming large amounts of resources. There are RLIM_NLIMITS types of resources a process can use. The kernel specifies limits on all of these RLIM_NLIMITS types of resources; for kernel 2.6, the value of RLIM_NLIMITS is 13. See the snapshot of file include/asm-i386/resource.h below, which contains the macros for all 13 resource types.

All Linux processes are assigned the same values of resource limits when they are created. However, after creation, a process can increase or decrease its limits. The resource limits of a process are stored in the
rlim field of the process descriptor. The rlim field is a 13 elements array of type struct rlimit, which is defined in file include/linux/resource.h. (Snapshot below)

The
rlimit structure contains two fields specifying current and maximum limits of a resource (also called soft and hard limits, respectively) to be used by a process. Using system calls getrlimit()/setrlimit(), any process can increase/decrease its soft limits up to corresponding hard limits. Hard limits of a process can only be increased if the particular process has superuser privileges. Therefore, a “normal” process may use getrlimit()/setrlimit() system calls to irreversibly decrease its hard limits.

If a process exceeds the consumption of a resource beyond the assigned soft limit, the kernel signals the process, or the system call (e.g.
malloc, fopen) trying to consume the resource fails with errno set to appropriate value. For example, the signals SIGXCPU, SIGXFSZ, SIGSEGV are fired when a process exceeds RLIMIT_CPU, RLIMIT_FSIZE, RLIMIT_STACK respectively.

This was the general background; in the next post we will practically see the actual limits imposed on Linux processes, and how these limits are assigned.

Tuesday, July 29, 2008

Kernel Process Data-Structures

In the simplest words, a process is defined as "execution context of a running program". Execution context is the collection of all the data structures, registers, memory addresses and other resources that the process uses while executing a program. Therefore, a program itself is not a process. Rather, it is a collection of data structures that fully describes how far the execution of program has progressed. In other words, a process is an active program, i.e., an in-memory representation of a program stored in hard-disk. An important point to note is that two or more processes can co-exist which are executing the same program, and a single process can execute two or more programs in its life span.


Running and managing a process is a complex activity for kernel. In this post (and probably the next few) I try to make a note of how beautifully the Linux kernel performs this task. Specifically in this post I note how the Linux kernel keeps track of the process currently scheduled to run on CPU.


Data Structure to Hold Entire Information About a Process:

Kernel identifies each process with a unique process descriptor represented by struct task_struct, which contains all the information about a specific process. List of all process descriptors is maintained in a circular doubly linked list.

The following screenshot shows a (partial) definition of the task_struct structure, defined in linux/sched.h. I have removed many fields here, which are not directly relevant to our discussion here.

getpid() Does Not Return Process id ?!?

But I have been always using it to get the pid of calling process without knowing this fact!!! Indeed, getpid() returns tgid (thread group id), which is the pid of the leader of threads executing within the same group, (which is usually the first process created within that group). Since Linux doesn't support multithreading directly, each process executes within a single thread in its own thread group. Therefore, a single threaded process naturally becomes its thread group leader. The difference occurs only when some library (e.g. POSIX) is used to create multithreaded processes, in which case, getpid() returns the value of calling process' tgid field. By the way, in Linux, threads are called light weight processes and they also have pids, since as we can see, there is nothing called thread id in task_struct.

Food for Thought:

1. What if I need to determine the pid of a light weight process, which is not a group leader?


In a uniprocessor system, only a single process can run at a time. The pointer to the task_struct of the process currently running on CPU is returned by a macro current. Thus, the statement


printk(KERN_ALERT "Process Name: %s\n" "PID: %d\n", current->comm, current->pid);


prints the name and pid of the currently executing process.


How Does the current Macro Determine the Currently Running task_struct?

Other than task_struct, there is one more data structure associated with a process, thread_info, which contains the low level information about a process when it is running on CPU. Both the task_struct and thread_info structures contain a field pointing to each-other, named thread_info and task respectively (line #439 in the task_struct snapshot). But unlike task_struct, which is a dedicated structure for each process, there is one thread_info structure per processor in the system. The thread_info field of task_struct (and eventually, the task field of thread_info) is updated as soon as the process is scheduled onto the processor. When a process is not running, the value of thread_info field of task_struct does not mean anything.

For each process running in kernel mode, kernel uses the following union spanning 8KB:

union thread_union{

struct thread_info thread_info;

unsigned long stack[2048]; /* (long: 4 bytes) x 2K = 8KB*/

};


Food for Thought:

2. What happens if I reverse the order of the two declarations above in thread_union?

3. What is wrong in following code (courtesy of Kernel Trap mailing list http://kerneltrap.org/node/5835)

char *get_sp() { asm("movl %esp, %eax"); }


int main(void){

struct thread_info *current_thread_info;

struct task_struct *current_task;

current_thread_info = (struct thread_info *)((unsigned long)get_sp()&~0x1FFF));

current_task = (struct task_struct *)(current_thread_info->task);

printf("my pid is: %d\n",(unsigned int)current_task->pid);

}

As soon as a process switches to kernel mode, the (8 KB) thread_union is "created" in the memory area kernel data segment which is designated for kernel by kernel only (address to kernel data segment is returned by macro __KERNEL_DS). The user mode processes cannot access the kernel data segment.


Since at any given time, esp register points to a memory location lying somewhere between this 8 KB area (adrressed by 13 bits), the starting address of thread_union can be obtained by masking out the 13 least significant bits of the esp register. Furthermore, since the first field of thread_union is thread_info, obtained value is a pointer to thread_info. This masking operation is performed by kernel routine current_thread_info() which looks like following:


movl $0xffffe000,%ecx /* store the mask */

andl %esp,%ecx /* mask the esp, ecx points to thread_info*/


Having acquired the address of thread_info structure, getting the pointer to task_struct is trivial. Hence, the current macro is:


current = current_thread_info()->task


This is how the current macro points to the task_struct of the process currently running on CPU.