Gfrktyl

Something is constantly causing mischief on my system, but I have not been able to identify which program it is. It happens a few times every hour. I've found the PIDs in a log, but the program is very short lived so by the time I can ps -fp pid, it's gone.

Can I set up logging somehow to see what all PIDs were afterwards? Which program say pid 12345 was with a timestamp? I've tried running find /proc/ -maxdepth 2 -name cmdline -mmin -3 -not -type d|while read l;do echo -n $l;cat $l;done and variants of that frequently, but without success. I'm running Ubuntu 16.04.

I used to administer a public shell host where users ran all sorts of processes that occasionally went rogue. We had root cron jobs that looked like this:

 * * * * * (date; ps auxwwww) >> /var/log/ps.log

 * * * * * (date; lsof -n) >> /var/log/lsof.log

We also installed rotation for the two log files so they wouldn't overflow.

When we observed problems with the system, we would review the ps log to catch the name of the rogue process and its arguments, and the lsof logs to find the process's executable image and any interesting files it had open.

However, if the rogue process you're looking for runs for a few seconds or less, then it's not likely to get caught by the above scripts which only run once a minute. You can write your own monitoring script that runs the above commands at a faster rate, say once every second.

If that's still not enough to catch it, you'll need something that vacuums up data from every process in the system. An Ubuntu 16.04 machine should be able to run relatively recent performance tools like bpf:

http://www.brendangregg.com/blog/2016-06-14/ubuntu-xenial-bcc-bpf.html

If that's not an option, you could try to use strace on the rogue process's parent process, assuming you knew what it was. If it's something being run out of cron or a user logging via ssh, a carefully written strace command that only captures process events and ignores everything else might work - but be very careful as it could be disruptive, especially if you trace too many things at once.

Finally, you don't specify what "mischief" is happening on your system - there are ways to trigger commands based on system events (update of a file, appearance of a packet on the network interface, CPU getting too high) that might work for you depending on your symptoms.

You can try to make a cron task that extract new process and log it to a file.

For example a command executing every minutes doing a ps wich select only process younger than the last cron execution and log it in a file. With chance you will catch it.