ngx_mruby vs lua-nginx-module memo - 人間とウェブの未来

Configuration, Environment

Machine

Ubuntu14.04 on VMWare
Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz 4core
Memory 8GB

Software version

ngx_mruby GitHub latest master
- df050067427fc8448a3c8589403fdfe336b82136
lua-nginx-module GitHub latest master
- bc120f9a2279a384e4335bedf765e844b4d22adb
nginx 1.7.6
Didn't tune up kernel parameters

OS

$ uname -a
Linux ubuntu14-04-64 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

nginx binary

$ ldd build/nginx/sbin/nginx
        linux-vdso.so.1 =>  (0x00007fff28946000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007feca7446000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007feca720d000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007feca6f06000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007feca6b2c000)
        libhiredis.so.0.10 => /usr/lib/x86_64-linux-gnu/libhiredis.so.0.10 (0x00007feca6922000)
        libluajit-5.1.so.2 => /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2 (0x00007feca66b2000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007feca6474000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007feca625b000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007feca5e94000)
        /lib64/ld-linux-x86-64.so.2 (0x00007feca7677000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007feca5c90000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007feca5a7a000)

nginx build options

$ ./build/nginx/sbin/nginx -V
nginx version: nginx/1.7.6
built by gcc 4.8.2 (Ubuntu 4.8.2-19ubuntu1) 
configure arguments: --add-module=/home/matsumotory/DEV/ngx_mruby --add-module=/home/matsumotory/DEV/ngx_mruby/dependence/ngx_devel_kit --prefix=/home/matsumotory/DEV/ngx_mruby/build/nginx --add-module=/home/matsumotory/DEV/lua-nginx-module

nginx.conf

worker_processes auto;

events {
    worker_connections 1024;
    multi_accept on;
}

daemon off;

http {
    include mime.types;
    server {
        listen 58080;
        server_name  localhost;
        root /home/matsumotory/DEV/ngx_mruby/build/nginx/html/;

        location /mruby {
            mruby_content_handler_code "Nginx.echo 'hello ngx_mruby world.'";
        }

        location /lua {
            content_by_lua "ngx.say('Hello ngx_lualu world.')";
        }

        location /static {
            # create static/inex.html
        }
    }
}

Benchmark Commands

$ ab -c 100 -n 1000000 -k http://127.0.0.1:58080/mruby
$ ab -c 100 -n 1000000 -k http://127.0.0.1:58080/lua
$ ab -c 100 -n 1000000 -k http://127.0.0.1:58080/static/index.html

Benchmark Results

ngx_mruby

This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests


Server Software:        nginx/1.7.6
Server Hostname:        127.0.0.1
Server Port:            58080

Document Path:          /mruby
Document Length:        23 bytes

Concurrency Level:      100
Time taken for tests:   11.708 seconds
Complete requests:      1000000
Failed requests:        0
Keep-Alive requests:    990044
Total transferred:      143950220 bytes
HTML transferred:       23000000 bytes
Requests per second:    85414.56 [#/sec] (mean)
Time per request:       1.171 [ms] (mean)
Time per request:       0.012 [ms] (mean, across all concurrent requests)
Transfer rate:          12007.27 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0      28
Processing:     0    1   1.5      1     111
Waiting:        0    1   1.5      1     111
Total:          0    1   1.6      1     114

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      2
  90%      2
  95%      2
  98%      4
  99%      5
 100%    114 (longest request)

lua-nginx-module

This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests


Server Software:        nginx/1.7.6
Server Hostname:        127.0.0.1
Server Port:            58080

Document Path:          /lua
Document Length:        23 bytes

Concurrency Level:      100
Time taken for tests:   15.227 seconds
Complete requests:      1000000
Failed requests:        0
Keep-Alive requests:    990053
Total transferred:      169950265 bytes
HTML transferred:       23000000 bytes
Requests per second:    65674.42 [#/sec] (mean)
Time per request:       1.523 [ms] (mean)
Time per request:       0.015 [ms] (mean, across all concurrent requests)
Transfer rate:          10899.79 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0      16
Processing:     0    2   2.5      1     143
Waiting:        0    1   2.5      1     143
Total:          0    2   2.5      1     143

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      2
  75%      2
  80%      2
  90%      3
  95%      3
  98%      5
  99%      8
 100%    143 (longest request)

Static Content

This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests


Server Software:        nginx/1.7.6
Server Hostname:        127.0.0.1
Server Port:            58080

Document Path:          /static/hello.html
Document Length:        22 bytes

Concurrency Level:      100
Time taken for tests:   16.687 seconds
Complete requests:      1000000
Failed requests:        0
Keep-Alive requests:    990049
Total transferred:      256950245 bytes
HTML transferred:       22000000 bytes
Requests per second:    59927.44 [#/sec] (mean)
Time per request:       1.669 [ms] (mean)
Time per request:       0.017 [ms] (mean, across all concurrent requests)
Transfer rate:          15037.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0      58
Processing:     0    2   3.8      1     449
Waiting:        0    2   3.8      1     449
Total:          0    2   3.9      1     449

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      2
  75%      2
  80%      2
  90%      3
  95%      4
  98%      8
  99%     12
 100%    449 (longest request)

one mistake; response data byte is different from others. If you write the following into nginx.conf, more faster, maybe.

accept_mutex_delay 100ms;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_requests 100000;

Thoughts

name	result
ngx_mruby	Requests per second: 85414.56 [#/sec] (mean)
lua-nginx-module	Requests per second: 65674.42 [#/sec] (mean)
static contents	Requests per second: 59927.44 [#/sec] (mean)

It's a just hello world benchmark, so this benchmark is hobby. But, very interesting.