MySQL failing to start with message “Can’t find file: ‘./mysql/plugin.frm’ (errno: 23)”

Recently I had a problem, where MySQL service was failing to start.
When tailing the MySQL log the following would be recorded while service was being started.

131224 06:04:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: Can't find file: './mysql/plugin.frm' (errno: 23)
131224 6:04:53 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
/usr/libexec/mysqld: Can't create/write to file '/tmp/ibqcFQMW' (Errcode: 23)
131224 6:04:53 InnoDB: Error: unable to create temporary file; errno: 23
131224 6:04:53 [ERROR] Plugin 'InnoDB' init function returned error.
131224 6:04:53 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
131224 6:04:53 [ERROR] Can't create IP socket: Too many open files in system
131224 6:04:53 [ERROR] Aborting

131224 6:04:53 [Note] /usr/libexec/mysqld: Shutdown complete

131224 06:04:53 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

Now, the message ‘Can’t find file: ‘./mysql/plugin.frm’ (errno: 23)‘ at the begging of the startup process might make you think that the problems is a missing file, but you can see that at the end of a startup process  this message is logged[ERROR] Can’t create IP socket: Too many open files in system.

This suggest that there is a problem with the number of files that is open on the system.
You can confirm this by using a perror utility, which prints system error messages.

If we check the error number 23, which is reported in the error message, we see that the cause of the failure is not a missing file, but that the file can’t be open because to many file handles are in use.

# perror 23
OS error code 23: Too many open files in system

You can check the current maximum number of file descriptors by checking the fs.file-max value in /etc/sysctl.conf, or use the sysctl command to check the current value.

# sysctl -a | grep file-max
 fs.file-max = 65536

To increase the maximum number of file handlesm you can edit /etc/sysctl.conf, change the value of fs.file-max to 200000 or some other value higher then the one you currently have, and then run sysctl -p to apply the new value to the system.

Now, after the file handle number has been increased, you should be able to start the MySQL service normally.

New app registered with whostmgr AppConfig: csf

A lot of people who have server with WHM/cPanel has been getting messages like this tonight.

A new app has been registered with AppConfig.

Name: csf
Service: whostmgr
ACLS required: software-ConfigServer-csf
System User: root
URL(s): /cgi/configserver/csf.cgi
Display Name: ConfigServer Security&Firewall
Entry URL: configserver/csf.cgi
You can view all current registered apps here: Apps Managed by AppConfig

 

And a lot of server owners has wondered if this is a genuine message, or are they been a victim of some sort of attack on their servers.

There is no need for the alarm, these message is caused by the WHM/cPanel update to 11.38.1., it is perfectly normal and there is no need for any kind of action.

In the new  version there were changes implemented by cPanel for addons software, and that is the cause of the message showing up for addons like ConfigServer software.

You can see the release note for version 11.38.1 here, and a post about the notification on cPanel forum here.

The message can be sent for other ConfigServer plugins, not only for ConfigServer FIrewall.

 

Enhanced by Zemanta

ConfigServer Internal Server Error 500, after cPanel update: fixed

After the WHM/cPanel update to 11.38 you might get the an error like this in your browser:

Internal Server Error
500
No response from subprocess (/usr/local/cpanel/whostmgr/docroot/cgi/addon_csf.cgi): subprocess exited with status 2

And something like this in /usr/local/cpanel/logs/error_log:

BEGIN failed–compilation aborted at /usr/local/cpanel/Cpanel/JSON.pm line 15.
Compilation failed in require at /usr/local/cpanel/Cpanel/Template/Plugin/JSON.pm line 12.
BEGIN failed–compilation aborted at /usr/local/cpanel/Cpanel/Template/Plugin/JSON.pm line 12.
Compilation failed in require at /usr/local/cpanel/Cpanel/Template.pm line 53.
BEGIN failed–compilation aborted at /usr/local/cpanel/Cpanel/Template.pm line 53.
Compilation failed in require at /usr/local/cpanel/Whostmgr/HTMLInterface.pm line 12.
BEGIN failed–compilation aborted at /usr/local/cpanel/Whostmgr/HTMLInterface.pm line 12.
Compilation failed in require at /usr/local/cpanel/whostmgr/docroot/cgi/addon_cmc.cgi line 25.
BEGIN failed–compilation aborted at /usr/local/cpanel/whostmgr/docroot/cgi/addon_cmc.cgi line 25.
For help, please send mail to this site’s webmaster, giving this error message and the time and date of the error.

when going to ConfigServer cPanel plugins in WHM like:

ConfigServer Explorer
ConfigServer Mail Manage
ConfigServer Mail Queues
ConfigServer ModSecurity Control
ConfigServer Security & Firewall

This happens if  the installed ConfigServer scripts on a cPanel/WHM server don’t get updated.

The solution for this error is simple.

To resolve this error simply SSH into your server as a root user and run the following command from command line:

curl -s configserver.com/free/csupdate | perl

This script will update: cmm, cmc, cmq, cse, csf, cxs, msinstall, msfe

You can see the ConfigServer blog post about the update here.