Apache’s CGI mode allows webmaster’s to reduce memory usage. It is the preferred way to run many low traffic websites.

But the CGI mode is pretty sensitive to things such as permissions and file encoding, that leads to 500 Internal Server Error.

In our role as Support Engineers for web hosts, we’ve seen a wide range of causes for this error.

Some common causes and their fixes are:

• Wrong file or folder permissions – Script files and directories must have EXACTLY 755 permissions. In some servers, directories may need 750 permissions.
• Wrong ownership (user & group issues) – The ownership of the file should be set to “USER : ApacheUser” or “USER : USER” depending on Apache’s configuration.
• Incorrect encoding – Files uploaded in Binary mode will fail execution, and should be re-uploaded as ASCII.

Almost 80% of error 500 issues would be solved by these fixes.

This article is about the rest 20% of causes which are hard to find and fix.

 

Uncommon causes for Apache CGI Internal Server Error

500 Internal Server Error is a web server’s way of saying, “Something has gone wrong when I tried to display the page. Not sure what.”

If it is not permission or ownership issues, it could be anything from application bugs & Apache misconfigurations to firewall blocks & file system errors.

Here are the top 10 uncommon issues we’ve seen in our course of work.

 

1. Missing modules

Web applications rely on a lot of PHP or Perl “modules” for specific functions.

Some such modules are part of a standard server setup, but many are not.

We’ve seen that after a migration or a during a new website setup, scripts often fail because all required modules are not present.

The error log will say there are “undefined” functions or “methods” do not exist.

To fix this, we go through the app requirement specifications, and install all missing modules.

 

2. Network timeouts (of API calls, remote scripts, etc.)

Some web apps get data from remote servers (eg. weather data).

If for some reason this connection to the remote server is interrupted, the web app will wait for a while and then show a time-out error.

Apache then reports this status as an Error 500.

This issue can be hard to troubleshoot as it might not leave a log entry.

We detect timeout issues by inspecting long pending outgoing connections, and fix it by disabling the app feature that requires the outbound connections.

 

3. Old program (compiler) paths

The first line in every CGI app (called Shebang) contains the path to the program that’s supposed to execute it.

For eg. Perl scripts have this path:

#!/usr/bin/perl

But this path can differ based on operating system vendors, and custom compilation settings.

So, the script will look for the program file in a particular location, and if it’s not found there, the script will show an Error 500.

Two effective methods are:

• Use the paths defined in the server environment – The paths of all programs are stored in the server’s “environment” variable. So, we change the first line by calling the environment, like so:
  • #!/usr/bin/env perl
  • This will make sure that no matter where Perl is setup, the path will be available to the script.
• Define the program paths in Apache config – Use the “AddHandler” and “Action” directives of Apache to execute all files with the right program. For eg. here’s how to setup PHP as a CGI:
  • AddHandler application/x-httpd-php5 php
  • Action application/x-httpd-php5 /local-bin/php-cgi

 

4. Errors in application code

Website owners can make small errors like deleting a “;” or inserting a stray character while editing application or configuration files.

This will result in the application failing to execute, but Apache will only spit out the cryptic “Internal Server Error” message.

We detect such issues by analyzing the log files, and resolve it by either fixing the code error, or by disabling the plugin / addon / theme that’s causing the error.

 

5. Old config files with wrong module paths

Websites are often customized for the server’s environment and program paths.

When sites are migrated, their configuration files will contain these old paths which might not be compatible with the new server.

We’ve seen cases where web applications use custom configuration files (say php.ini) to load library functions. This will fail when the path changes in a new server.

In such cases we remove the hard-coded program paths and instead use environment variables to automatically acquire program locations.

 

6. Security restrictions (Web application firewall, SELinux, etc.)

Most servers these days have Web Application Firewalls such as mod_security or ComodoWAF.

These firewalls can cause an Error 500 if it interprets a page request as an attack.

For eg. we’ve seen web apps failing because an external file calls were interpreted as a Remote File Inclusion attack by mod_security.

In such cases, we create custom firewall exclusion rules so that the app will no longer be blocked, while the server security is not affected.

Changing SELinux setting from “Enforcing” to “Permissive” has also helped us fix this error.

 

7. No Exec (no execution) restriction on the partition

CGI apps are executed as programs.

For that it needs something called an “Execute” privilege.

This “Execute” privilege is turned off for public accessible partitions such as “/tmp” and data-store partitions such as “/backup” to prevent malware execution.

We’ve seen cases where this “Execute” bit is turned off for website homes, thereby causing the script execution to fail.

We restore exec by:

• # mount -o remount,exec /home

 

8. Apache & .htaccess misconfiguration (no ExecCGI, no AllowOverride, etc.)

Apache needs to know which directories contain CGI scripts.

That is denoted using the directive “ExecCGI”.

For eg. to denote /cgi-bin/ as a CGI directory, Apache should be configured with:

<Directory /path/to/cgi-bin>
Options +ExecCGI
</Directory>

In some servers we’ve seen “ExecCGI” disabled in cgi-bin due to misconfiguration in either Apache conf file or an .htaccess in the parent directory.

Similarly, we’ve seen CGI related directives in .htaccess becoming ineffective because .htaccess was not enabled through an “AllowOverride” directive in Apache config.

All this can result in a failed script execution, and consequently an Error 500.

 

9. Wrong permissions on home directory

CGI directories are usually placed within the home directory. For eg. the path to a cgi-bin in cPanel servers is:

/home/username/public_html/cgi-bin/

Many webmasters take care to change the permission of the cgi-bin directory and the web application directories to 755, but forget to check the home directory’s permissions.

In some Apache configurations (eg. when using SuExec), the home directory should also have 755 permissions. Anything other than that (eg. 777), will cause the script execution to fail.

So, one of the first things we check along with script file permissions is to check the permissions of all directories in the file’s path.

We set all of it just right to make sure the script executes without errors.

 

10. Chroot related errors that breaks file path

In some shared servers, security is enforced by confining each user to a jailed environment.

This prevents one user from accessing the files of any other user.

But such environments can break softlinks to program files.

For eg. if programs are stored in “/opt/php/bin/” and it is linked to from “/usr/bin/php/”, those references can get broken.

These are pretty rare cases, and we fix it by re-configuring the server (or site) to use the right path instead of softlinks.

 

Conclusion

Incorrect permission and ownership settings are the biggest causes for Apache CGI Internal Server Errors. But, there are more issues that can cause this error. Today we’ve covered the top 10 uncommon causes for this error as seen by our Support Engineers.