Nginx worker process constantly crash

[0x5446]

9063 2012/12/17 14:10:05 [notice] 12334#0: signal 17 (SIGCHLD) received
9064 2012/12/17 14:10:05 [alert] 12334#0: worker process 30114 exited on signal 11 (core dumped)
9065 2012/12/17 14:10:05 [notice] 12334#0: start worker process 30267
9066 2012/12/17 14:10:05 [notice] 12334#0: signal 29 (SIGIO) received
9067 2012/12/17 14:10:09 [notice] 12334#0: signal 17 (SIGCHLD) received
9068 2012/12/17 14:10:09 [alert] 12334#0: worker process 30183 exited on signal 11 (core dumped)
9069 2012/12/17 14:10:09 [notice] 12334#0: start worker process 30268
9070 2012/12/17 14:10:09 [notice] 12334#0: signal 29 (SIGIO) received
9071 2012/12/17 14:10:12 [notice] 12334#0: signal 17 (SIGCHLD) received
9072 2012/12/17 14:10:12 [alert] 12334#0: worker process 30226 exited on signal 11 (core dumped)
9073 2012/12/17 14:10:12 [notice] 12334#0: start worker process 30297
9074 2012/12/17 14:10:12 [notice] 12334#0: signal 29 (SIGIO) received
9075 2012/12/17 14:10:14 [notice] 12334#0: signal 17 (SIGCHLD) received
9076 2012/12/17 14:10:14 [alert] 12334#0: worker process 30297 exited on signal 11 (core dumped)
9077 2012/12/17 14:10:14 [notice] 12334#0: start worker process 30298
9078 2012/12/17 14:10:14 [notice] 12334#0: signal 29 (SIGIO) received
9079 2012/12/17 14:10:24 [notice] 12334#0: signal 17 (SIGCHLD) received
9080 2012/12/17 14:10:24 [alert] 12334#0: worker process 30298 exited on signal 11 (core dumped)
9081 2012/12/17 14:10:24 [notice] 12334#0: start worker process 30304
9082 2012/12/17 14:10:24 [notice] 12334#0: signal 29 (SIGIO) received
9083 2012/12/17 14:10:52 [notice] 12334#0: signal 17 (SIGCHLD) received
9084 2012/12/17 14:10:52 [alert] 12334#0: worker process 30304 exited on signal 11 (core dumped)
9085 2012/12/17 14:10:52 [notice] 12334#0: start worker process 30349
9086 2012/12/17 14:10:52 [notice] 12334#0: signal 29 (SIGIO) received
9087 2012/12/17 14:11:10 [notice] 12334#0: signal 17 (SIGCHLD) received
9088 2012/12/17 14:11:10 [alert] 12334#0: worker process 30349 exited on signal 11 (core dumped)
9089 2012/12/17 14:11:10 [notice] 12334#0: start worker process 30375
9090 2012/12/17 14:11:10 [notice] 12334#0: signal 29 (SIGIO) received
9091 2012/12/17 14:11:17 [notice] 12334#0: signal 17 (SIGCHLD) received
9092 2012/12/17 14:11:17 [alert] 12334#0: worker process 30375 exited on signal 11 (core dumped)
9093 2012/12/17 14:11:17 [notice] 12334#0: start worker process 30406
9094 2012/12/17 14:11:17 [notice] 12334#0: signal 29 (SIGIO) received

[tianfeng@zdi01v /home/q/system/corefiles]$ sudo gdb /usr/local/nginx/sbin/nginx
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(gdb) core-file
No core file now.
(gdb) core-file core-nginx-26791-1355722867
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/local/lib/libluajit-5.1.so.2...done.
Loaded symbols for /usr/local/lib/libluajit-5.1.so.2
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libpcre.so.0...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /home/q/system/se_msg-2710/lib/cjson.so...done.
Loaded symbols for /home/q/system/se_msg/lib/cjson.so
Core was generated by `nginx: worker process '.
Program terminated with signal 11, Segmentation fault.
[New process 26791]
#0 0x00000000004c29b9 in ngx_http_lua_socket_tcp_finalize (r=0x33e12d0, u=0x413267a8) at lua-nginx-module-0.7.3//src/ngx_http_lua_socket_tcp.c:2185

warning: Source file is more recent than executable.
2185 cl->buf->pos = cl->buf->last;
(gdb) bt
#0 0x00000000004c29b9 in ngx_http_lua_socket_tcp_finalize (r=0x33e12d0, u=0x413267a8) at lua-nginx-module-0.7.3//src/ngx_http_lua_socket_tcp.c:2185
#1 0x00000000004c2b2a in ngx_http_lua_socket_tcp_cleanup (data=0x2a) at lua-nginx-module-0.7.3//src/ngx_http_lua_socket_tcp.c:2162
#2 0x000000000046b221 in ngx_http_free_request (r=0x33e12d0, rc=0) at src/http/ngx_http_request.c:3015
#3 0x000000000046b41d in ngx_http_close_request (r=, rc=0) at src/http/ngx_http_request.c:2990
#4 0x000000000046cc03 in ngx_http_lingering_close_handler (rev=0x330dd80) at src/http/ngx_http_request.c:2857
#5 0x00000000004570aa in ngx_event_process_posted (cycle=, posted=0x33e4238) at src/event/ngx_event_posted.c:40
#6 0x0000000000456f6f in ngx_process_events_and_timers (cycle=0x32ddbe0) at src/event/ngx_event.c:274
#7 0x000000000045cc4d in ngx_worker_process_cycle (cycle=0x32ddbe0, data=) at src/os/unix/ngx_process_cycle.c:808
#8 0x000000000045b597 in ngx_spawn_process (cycle=0x32ddbe0, proc=0x45cb85 <ngx_worker_process_cycle>, data=0x0, name=0x5be1b9 "worker process", respawn=0) at src/os/unix/ngx_process.c:198
#9 0x000000000045d431 in ngx_master_process_cycle (cycle=0x32ddbe0) at src/os/unix/ngx_process_cycle.c:622
#10 0x000000000044224d in main (argc=28, argv=0x32dd760) at src/core/nginx.c:410

(gdb)

[tianfeng@zdi01v /usr/local/nginx/logs]$ nginx -V
nginx version: nginx/1.2.4
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-46)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx/ --with-http_stub_status_module --with-http_ssl_module --with-openssl=openssl-1.0.0b --add-module=nginx_upload_module-2.2.0 --add-module=masterzen-nginx-upload-progress-module-82b35fc --with-http_realip_module --add-module=nginx_upstream_hash-0.3.1 --add-module=ngx_devel_kit-0.2.17rc2/ --add-module=lua-nginx-module-0.7.3/

I have read this issue: #110. But My lua-nginx-module's version is 0.7.3, I also checked the source code and found it has indeed been fixed already, so it must be some other things that caused the problem.

[0x5446]

What will happen when I do not put "memc:set_keepalive" neither "memc:close" at the end of communication with the upstream memcachd? Did it caused the crash?? Thanks!

[agentzh]

Hello!

On Sun, Dec 16, 2012 at 10:20 PM, dearGod notifications@github.com wrote:

9063 2012/12/17 14:10:05 [notice] 12334#0: signal 17 (SIGCHLD) received
9064 2012/12/17 14:10:05 [alert] 12334#0: worker process 30114 exited on
signal 11 (core dumped)

Could you please use tools like gdb to obtain the backtrace from the core
dump files for the crashes? See details here:

http://wiki.nginx.org/Debugging#Core_dump

Otherwise, there's too little information here to diagnose the problem.

Best regards,
-agentzh

[agentzh]

Hello!

On Wed, Dec 19, 2012 at 2:39 AM, dearGod notifications@github.com wrote:

What will happen when I do not put "memc:set_keepalive" neither
"memc:close" at the end of communication with the upstream memcachd?

When your resty.memc object is collected by the Lua GC, Lua GC will
automatically call the "close" method on it.

Did it caused the crash?? Thanks!

I don't think so. You have to get the backtrace for the crash first.

Best regards,
-agentzh

[agentzh]

Any updates on this issue please?

[agentzh]

Consider it resolved.