Gfrktyl

How are other admins managing CPU resources on multi-core hosts that run jails, such as allocating the appropriate number of cores for exclusive use by specific jails, and likewise reserving the appropiate number of cores for exclusive use by the host system?

TL;DR

I'm learning how to manage CPUs using rctl(8), /etc/rctl.conf, cpuset(1) and related tools.

As an exercise, I am working with an 8-core system. I wish to dedicate CPUs 0-3 for use solely by the host system processes, and make CPUs 4-7 available solely to jailed processes.

I'm encountering two fundamental problems in my mental landscape of this problem. The first is that since cpuset can apply a CPU mask only to an existing target (process, jail, whatever), I presently have to start a jail first, and THEN restrict its CPU set. This means that jailed processes which are spawned before the cpuset restriction is created will run outside of the CPU set. If those jailed processes are long-lived then they must be restarted after cpuset is invoked in order to ensure that the jail stays within its restriction.

Second, for similar reasons as #1, it is not clear to me how to get the host system to restrict its own CPU usage, so that it doesn't use any of the CPUs that are dedicated to running jails. Again, since a CPU set restriction has to be established before a process is born, it seems that init(8) itself needs to be launched with a restricted CPU set. I cannot find any mechanism by which to do this.

To solve the first problem, that of being able to start a jail with an immediate cpuset restriction, this syntax didn't work in /etc/jail.conf:

path                = "/jail/$name";



test {

  jid = 42;

# This is the default value before tweaking:

#  exec.start        = "/bin/sh /etc/rc";

#  restrict this jail to CPU7 only:

  exec.start        = "/usr/bin/cpuset -c -l 7 /bin/sh /etc/rc";

  host.hostname     = "test";

  ip4.addr          = "public|10.160.161.162";

  mount.devfs       = true;

  allow.raw_sockets = true;

  allow.sysvipc     = true;

}

Another unsuccessful approach was to restore the default value of exec.start and start the jail with:

# cpuset -c -l 7 service jail start test

Regarding the second problem, that of restricting the host system to prevent non-jailed processes (jail id = 0) from using CPUs that are reserved for jailed processes (jail id's > 0, I'm not sure where to begin. The most elegant way would seem to be to restrict init(8) at its very outset, so that any processes it spawns inherit the same CPU restriction. A less elegant hack might be to use rctl.conf to create a CPU restriction based on a particular login class, and assign all users to that login class. That still seems to fail to restrict system daemons which are launched by init before rctl initializes. A naive look at rcorder(8) suggests that rctl runs relatively late in the rc start-up process, at 127th place out of 166 on my system:

# cd /etc/rc.d

# rcorder * | grep -xn rctl

127:rctl

# rcorder * | wc -l

     166

What are some solutions and best practices that have been developed to solve this kind of problem? For now I am reading some historical discussion from 2014 regarding the issue of starting init(8) with a specific cpuset. Sadly the patch proposed by the OP of that thread is no longer accessible.