“We’re having high overload in the php fpm service (it loads the 16 VCPU with 100%). I’ve to restart the service 4 times. We need your help.”
We got this request recently from a server owner who was facing high CPU load due to PHP-FPM processes.
In our role as Technical Support Services for web hosting companies, configuring and managing the web servers for best performance, is a major task we do.
Analyzing the reasons for high server load and resolving it, form a part of that service. Here’s how we tackled this issue.
What is PHP-FPM? Why does it cause high load?
PHP-FPM is PHP FastCGI Process Manager. It is a FastCGI daemon that allows a website to handle high load. But sometimes this service itself causes high load.
To debug the issue, the first thing we did is to examine the PHP-FPM error logs, which is available in a location such as ” /var/log/php7.0-fpm.log “.
The location for the error logs varies based on the installation path and PHP version. While checking the error log, we found some warnings like these.
[12-Nov-2018 11:12:11] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 9 idle, and 89 total children
This message means that the PHP-FPM pool is very busy and is unable to handle the high traffic with the existing configuration parameters.
PHP-FPM can lead to high server load due to many reasons:
- Too many processes running due to high traffic or attacks
- A related service such as MySQL or web server being abused
- PHP-FPM configuration is not properly mapped to available server resources
- Poorly coded or buggy applications abusing the server resources
How to perform PHP-FPM tuning for high load
The first thing server owners tend to do for an immediate fix, is to restart the PHP-FPM service. But simply restarting the service is only a band-aid solution, and you will soon see the load spiking.
When we debug such load issues, we analyze the logs and server performance over a period of time. This helps us to pinpoint the culprit and take prompt actions to fix it.
1. Switch process manager
Process manager is mainly of 3 types – dynamic, ondemand and static. The performance of the service drastically varies based on this type.
In dynamic type, the number of child processes is set dynamically based on the PHP-FPM parameters in conf file. But it is a bit memory-intensive type.
In static type, the number of child processes is fixed by pm.max_children parameter, but this type is not flexible for a server with changing web traffic. It also consumes too much memory.
In ondemand type, the PHP-FPM processes are spawned only on demand, based on the traffic. This type helps to manage varying traffic in memory restrained servers. But the overhead increases when there is so much traffic fluctuation.
The PHP-FPM process manager is determined after assessing the available server memory, traffic spikes, website pages, CPU usage, etc.
2. Tweak PHP-FPM parameters
PHP-FPM has a lot of configuration parameters which determine the way it performs. These parameters have to be determined based on available server resources such as RAM and CPU.
For instance, the total processes that run in a server would be approximately = (Total RAM / Memory per process) . In a server with multiple services, all of them are taken into account for the tuning process.
The PHP-FPM configuration file will be available at /etc/php-fpm.conf or some other location based on the service path. The contents of the file would look like:
listen.allowed_clients = 127.0.0.1 pm = dynamic pm.max_children = 5 pm.start_servers = 2 pm.min_spare_servers = 2 pm.max_spare_servers = 5 pm.max_requests = 150
Some of the major parameters that we tune for performance are:
-
- pm.max_children – This is used to set the total number of processes allowed
- pm.start_servers – The number of child processes created on startup is defined by this
- pm.min_spare_servers – Defines the minimum number of idle processes
- pm.max_spare_servers – Sets the maximum number of idle server processes
- pm.process_idle_timeout – The number of seconds an idle process will be alive
- pm.max_requests – This sets the execution time of each child process and is used to curb memory leaks.
A common mistake server owners do, is to set very high values for these parameters, to get maximum performance. But if there are not enough resources, such high values can crash the server.
The value for ‘pm.max_children’ parameter is first determined based on the server resources, website traffic and content of the sites. Other parameters are then derived upon further testing.
3. Enable slow log
In cases where the parameters are fine, but a certain application or site is suspected to cause the high load, we enable the slow log option in PHP-FPM.
request_slowlog_timeout = 6s slowlog = /var/log/php-fpm/slowlog-site.log
The log file mentioned for ‘slowlog’ parameter in the PHP-FPM config file records the requests that take more than 6 seconds to execute.
Analyzing these requests enable us to track and fix the resource-intensive requests and applications in the server.
4. Managing multiple pools
PHP-FPM has the option to configure multiple resource pools for separate applications. This helps to contain resources per application and for additional security.
By configuring parameters based on the pool, we restrict a single pool from using up all server resources. Unused pools are deleted for further management.
PHP-FPM tuning for high load – An ongoing process!
Tweaking PHP-FPM for best performance, is not just a template fix. The configuration and parameters for the service would vary based on the server needs and resources.
During peak traffic, the performance would be different from that of normal hours. So, after doing the tweaks, we conduct stress tests to see how the service performs during high load.
Based on the server performance during stress test, we do further fine-tuning of the service. We also monitor the server load for a period of time, to ensure that the changes has indeed helped.
In this scenario, we were able to reduce the server load drastically, by performing the PHP-FPM tweaks.
But PHP-FPM tuning alone may not help in certain scenarios where services such as MySQL or web server too are culprits. In those cases, we perform additional tasks such as:
- Fine-tuning the MySQL server and web server such as Apache and Nginx
- Limiting server resources based on service, applications, users, etc. to avoid server crash
- Setting up load balancers or clusters to distribute services in high traffic servers
- Monitoring the server 24/7 and conducting periodic server audits to detect and resolve issues then and there
- Enabling page caching and optimizing code and queries to avoid applications from overloading server
Conclusion
Today we saw how our Support Engineers fix high load issue due to PHP-FPM service. In our customers’ servers, we monitor the server performance 24/7 and perform periodic server tweaks to ensure that server load stays normal.