3

TL:DR: PHP/Apache in docker is continuously creating 'files' in /proc/*/fd/ eventually making the site unusable.

We are running a PHP app in docker (php:5.6-apache). We deployed this stack to production a few weeks ago. After a few days costumers had problems accessing the site, which was caused by random 403 errors for various files, like javascript, image, etc.
As far as I can tell this was caused by this error 

[core:crit] [pid 17] (24)Too many open files: AH00529: /var/www/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable and that '/var/www/' is executable

So I went and checked what files where opened by the Apache processes. lsof -a -p 15 output: 

apache2  15 www-data   35   unknown                      /proc/15/fd/35 (readlink: Permission denied)
apache2  15 www-data   37   unknown                      /proc/15/fd/37 (readlink: Permission denied)
apache2  15 www-data   38   unknown                      /proc/15/fd/38 (readlink: Permission denied)

The fd number increments continuously when the website is accessed, until eventually it reaches the open file limit.

EDIT: The reason for (readlink: Permission denied) was a docker security feature, so all those handles are actually open TCP sockets. 

apache2  15 www-data  198u     sock    0,8      0t0 794575 protocol: TCP
apache2  15 www-data  200u     sock    0,8      0t0 795679 protocol: TCP
apache2  15 www-data  201u     sock    0,8      0t0 795681 protocol: TCP

However netstat lists no unusual open connections. netstat -a output: 

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.11:42045        0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
udp        0      0 127.0.0.11:59658        0.0.0.0:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
Giacomo1968
  • 53,069
  • 19
  • 162
  • 212
Mormund
  • 151
  • 5
  • Never seen a LAMP stack fail like this. Please check this [Stack Overflow answer for some insight](https://stackoverflow.com/a/14749074/117259). – Giacomo1968 Aug 20 '18 at 16:10
  • @JakeGould Problem is that it never closes the file handles so increasing the limit would only delay the issue. – Mormund Aug 20 '18 at 16:16
  • The problem is I have never seen a PHP server behave this way. The reality is you seem to have a coding/engineering problem if you have streams and files open for that long. PHP is not inherently designed to work this way. Thus you are hitting these walls. The only solution I can see is you need to close file handles more often. But without seeing your code, no other advice. And posting your code is impractical; what I am looking at here is a complete refactoring of PHP code or moving processes to a different language. – Giacomo1968 Aug 20 '18 at 16:56

1 Answers1

2

Turns out we had the xdebug extension installed which doesn't seem to close its sockets correctly.

socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 306
20    fcntl(306, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
20    connect(306, {sa_family=AF_INET, sin_port=htons(9000), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EINPROGRESS (Operation now in progress)
20    select(307, [306], [306], [306], {tv_sec=0, tv_usec=200000}) = 2 (in [306], out [306], left {tv_sec=0, tv_usec=199998})
20    getpeername(306, 0x7ffe1d6453f0, [16]) = -1 ENOTCONN (Transport endpoint is not connected)20
Giacomo1968
  • 53,069
  • 19
  • 162
  • 212
Mormund
  • 151
  • 5
  • 1
    Good detective work! [Check out this answer here](https://superuser.com/a/829413/167207) for more details. XDebug is a valuable tool for non-production servers, so disabling it is a good idea in this case. But still it is nice to know what might solve this issue past outright disabling it. – Giacomo1968 Aug 20 '18 at 20:05