Fan control via ipmitool for Supermicro motherboards on Ubuntu Server 16.04
One of the biggest challenges I've faced getting my new home server setup has been fan control. My Supermicro motherboard only allows fan control via the IPMI interface, whereas I'm used to using Linux's lm-sensors, pwmconfig, and fancontrol systems. After a bunch of futzing about, I think I've got something functional.
Instead of going over all the "well this didn't work" things, I'm only going to discuss what is working.
Where This Script Comes From
I searched and searched for some method and kept getting routed back to a couple of Perl scripts made for FreeNAS. Since the subsystems involved to get CPU and hard drive temperatures for FreeNAS and Ubuntu are quite different, the scripts won't work out of the box. So, I turned to Fiverr to get a Perl programmer to help out changing up the bits that had to be changed. I know enough of scripting to get the idea of what something is doing, but not remotely enough to modify things as we'd need it.
The script we're using is by Kevin Horton, available here: https://forums.freenas.org/index.php?threads/pid-fan-controller-perl-script.50908
NOTE: Kevin's script is based on a script by FreeNAS forums user Stux, available here: https://forums.freenas.org/index.php?threads/script-hybrid-cpu-hd-fan-zone-controller.46159/
The core difference between Stux and Kevin's scripts are Stux's script is a bit "brute force" where as Kevin's involves a bit more "finesse." To be fair, Stux's script is great, but it's more raw. Kevin's iteration on the script utilizes PID control, allowing the fans to be run with more fine control. This appeals to me as the goal here is to keep the temperatures, and the fans, in a more controlled state.
What Needed to Change
I know what had to change in the script to get it working on Ubuntu: We need to change the subroutines that get the CPU and hard drive temperatures to utilize the programs sensors and hddtemp, respectively. However, since I haven't even attempted to program with Perl since I was 15 or 16, and even then it was a poor shot at it, so I sent Kevin's script to gperales on Fiverr.
Gerardo was able to replace Kevin's subs that obtain the drive temperatures with a simplified version that utilizes hddtemp instead. I didn't even know where to begin recoding those subroutines so I'm quite thankful for the help. The sub that gets the CPU temperature utilizes a command string and involves some awk and sed piped commands, which I'm far more comfortable adjusting on my own so that's what I did. After the tweaks were made I ran it for a few minutes while monitoring the temps and fan speeds and it's working mostly as expected. The fan speeds are a bit hectic, but I blame the Noctua fans I'm using as they're static pressure fans, not high air flow, and the replacements should be in Monday. Once the high air flow fans are installed I expect them to be more capable of maintaining the drive temps at more stable speeds. Once installed I'll update here to let anyone interested know if it's more stable.
For now I'm disabling the script since the fan speeds are bouncing back and forth. It appears to me to be running properly, it's just that the fans aren't getting enough air moving at the lower speeds. There may need to be some tuning to the script later to get the fan speeds to run at different RPM's at a given duty cycle, but I'd rather do that with the replacement fans.
Try it Yourself
If you'd like to give a shot you can download these two files and put them on your server yourself. For security I've renamed the files to have a .txt extension, but for your own sake please do scan through both files yourself to ensure they're safe before using them! NOTE: I'm assuming you'll know how to download the files and get them to your server on your own, so the second command below is simply changing directory to the location you put the files on the server.
Also, at this point there are likely a number of comments and subs that are slightly "out of sync" with each other due to what's changed. I'm leaving it on purpose, I'd rather let the original author, Kevin, decide what to do with the changes. As is it's working, which is what matters to me.
Download
Notes
You'll want to open the IPMIFanControl.pl.txt file and change a few variables. I've set it to be configured for my own setup but you'll have your own changes to make. Look through the CONFIGURATION, FAN CONFIGURATION, and MISC sections to make sure everything is correct, and then look under the GLOBALS section and ensure the @hd_list variable contains all the drives you'll need monitored.
If you download both files, the following commands will install dependencies, move the files to their necessary locations, change permissions and setup the service so it can start automatically.
Setup
sudo apt install perl hddtemp lm-sensors ipmitool cd <LOCATION YOU DOWNLOADED FILES TO> sudo mv IPMIFanControl.pl.txt /usr/local/bin/IPMIFanControl.pl sudo mv IPMIFanControl.service.txt /lib/systemd/system/IPMIFanControl.service sudo systemctl daemon-reload sudo systemctl enable IPMIFanControl.service sudo service IPMIFanControl start
Final Words of Advice
With that the service should start and you can start monitoring the temps and fans with this command:
watch -n5 sudo ipmitool sensor
Allow it to run for a while to get a feel for what it's doing out of the box. If you need to tune anything, the main things to look at is the PID CONTROL GAINS settings, the FAN SPEEDS, and you may want to change the FAN DUTY LEVELS as those are basically a percentage of the full speed of fans and are the "stages" for each duty. For instance, hd_fan_duty_med_high being set to 80 means when the script moves those fans to a medium/high fan duty cycle, the fans run at 80% speed. You can adjust those numbers a bit to find a better curve for your fans.