Hacking Book | Free Online Hacking Learning


using open source software to build large scale waf cluster

Posted by agaran at 2020-03-16


Most Internet companies can't develop their business without web services, and web protection has become a problem that many security teams of party a need to face. In this paper, combined with practical experience, how to use open-source software to build a large-scale WAF cluster is described. In order to be more general, some self-developed modules use open source components instead. The development and maintenance of self built large-scale WAF cluster needs considerable manpower. You need to be cautious. We have experienced more than two years before it is stable. Both BAT3 and the traditional top 4 have solutions. You can really consider the commercial mature solutions.

WAF cluster architecture

The main components are: four layers of load balancing, WAF service, WAF log processing, WAF configuration management, which support cluster deployment and rapid horizontal expansion.

Four layer load balancing

From the perspective of basic network security, we should strictly limit the direct use of public IP to provide external services, so that the attack surface will be very large and uncontrollable. All tested services should use load balancing to publish web services to the outside world, and other services are not directly published to the outside world, which can strictly control the attack surface of hackers. The main functions of the four layer load are:

Four layer load balancing, which refers to user request forwarding to back-end WAF service

Publish the back-end WAF service in the form of public IP

It's better to have the ability to resist the attack of four layers

The common open source solution is LVS. LVS is the abbreviation of Linux virtual server, which means Linux virtual server. It is a virtual server cluster system, which supports three modes: vs-nat, vs-tun and vs-dr

Vs-nat: it is to change the destination address of the IP header of the data packet sent from the client to the IP address of one of the RS's on the load balancer, and then process it to rs. after RS processing, the data will be handed over to the load balancer, and then the load balancer will change the original IP address of the data packet to its own IP address, and change the destination address to the client's IP address. During this period, regardless of the incoming traffic, Or the flow out must go through the load balancer

Vs-tun: encapsulate a new IP header mark (only purpose IP) sent to rs by the data package sent from the client. After RS receives it, first untie the header of the data package, restore the data package, and then directly return it to the client without passing through the load balancer. Note that RS needs to restore the data package sent from the load balancer, so it must support the iptunnel protocol, In the RS kernel, you must compile and support the iptunnel option.

Vs-dr: both the load balancer and RS use the same IP for external service. Because the load balancer needs to change the layer 2 packet header, the load balancer and RS must be in a broadcast domain, which can also be simply understood as on the same switch

Because we hope that only LVS will be exposed to the public network, the deployment of back-end WAF and web services will not be limited by too many physical cabinet network segments. It is recommended to use vs-nat mode. Although all requests and responses will pass LVS, LVS may become the bottleneck of the whole system, but the actual experience is that LVS performance is good enough, and the single machine performance bottleneck is often the network card, and the CPU has spare power when the 10 Gigabit network card is full 。

As an example, the LVS is configured as follows:

/sbin/ipvsadm -C

/sbin/ipvsadm -a -t -r -m -w 1

/sbin/ipvsadm -a -t -r -m -w 1


Note: VRRP virtual VIP can be used between two LVS to achieve high availability. Please supplement more optimization of production environment.

WAF services

Common open source WAFS include mod_security and nginx + Lua. Due to the excellent performance of nginx and the programming level flexibility of lua, nginx + Lua has become the choice of more people. A code example is as follows:

location / {          default_type      'text/plain';          access_by_lua '                  local waf = require "waf"                  waf.execute()          ';}

All the real WAF work is done by the WAF module written by Lua. This part is written very clearly in loveshell (https://github.com/loveshell/ngx_lua_waf), and Chunge's is not bad (https://github.com/openresty)

I have slightly simplified the installation process:

Install luajit

wget http://luajit.org/download/LuaJIT-2.0.0.tar.gz

tar zxvf LuaJIT-2.0.0.tar.gz

cd LuaJIT-2.0.0


make install PREFIX=/usr/local/lj2

ln -s /usr/local/lj2/lib/libluajit-5.1.so.2 /lib64/

Install devel Kit

wget https://github.com/simpl/ngx_devel_kit/archive/v0.2.17rc2.zip

unzip v0.2.17rc2

Install Lua nginx module

wget https://github.com/chaoslawful/lua-nginx-module/archive/v0.7.4.zip

Unzip v0.7.4

Install PCRE

wget http://blog.s135.com/soft/linux/nginx_php/pcre/pcre-8.10.tar.gz

tar zxvf pcre-8.10.tar.gz

cd pcre-8.10/


make && make install

Compile nginx

wget 'http://nginx.org/download/nginx-1.2.4.tar.gz'

tar -xzvf nginx-1.2.4.tar.gz

cd nginx-1.2.4/

export LUAJIT_LIB=/usr/local/lj2/lib/

export LUAJIT_INC=/usr/local/lj2/include/luajit-2.0/

./configure --user=daemon --group=daemon --prefix=/usr/local/nginx/ --with-http_stub_status_module --with-http_sub_module --with-http_gzip_static_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module  --add-module=../ngx_devel_kit-0.2.17rc2/ --add-module=../lua-nginx-module-0.7.4/

Make -j8

Make install

Modify nginx configuration file nginx.conf (the following is just a principle demonstration)

Http {

include       mime.types;

default_type  application/octet-stream;

#Add the following five lines

lua_need_request_body on;

lua_package_path "/usr/local/nginx/conf/waf/?.lua";

lua_shared_dict limit 10m;

init_by_lua_file  /usr/local/nginx/conf/waf/init.lua;

access_by_lua_file /usr/local/nginx/conf/waf/waf.lua;

HelloWorld rule

[[email protected] waf]# cat waf.lua

if ngx.var.query_string and ngx.re.match( ngx.var.query_string, "passwd" ) then

-- ngx.say("You are blocked by douge!");

return ngx.exit(ngx.HTTP_FORBIDDEN);


[[email protected] waf]#

Access blocked

configuration management

Configuration management is actually the most easily ignored part of the whole system, but in fact, it is a very important part.

Configure synchronization model

There are two most common ways to configure synchronization: push and pull. Push mode is the most convenient, but it can not guarantee the timeliness in large-scale distributed system; pull mode performance is poor. Can we balance the two ways?


Zookeeper is a distributed, open-source distributed application coordination service, which is an important component of Hadoop and HBase. It is a software that provides consistency services for distributed applications. Its functions include configuration maintenance, domain name service, distributed synchronization, group service, etc. Zookeeper is widely used in configuration management of distributed system. Zookeeper supports publish and subscribe mode, which can be regarded as one to many relationship: multiple subscriber objects listen to a subject object at the same time. When the subject object changes its status, it will notify all subscriber objects, so that they can automatically update their status. Publish and subscribe mode enables publishers and subscribers to independently encapsulate and change independently. When an object changes, it needs to change other objects at the same time, and it does not know how many objects need to change, it can use publish and subscribe mode.

In the most simplified design, we can make the configuration of each WAF exactly the same. This advantage is that you can add servers at any time, any server downtime will not affect the WAF service, and the architecture of the whole system will be very simple. Each WAF only needs to use Python script to subscribe to the configuration in ZK. Once the configuration changes, get the latest configuration, update the relevant configuration file, and send a signal to nginx to hot load the new configuration file (nginx - s reload).

ZK is very simple to install and use

Download it and unzip it.

Edit the configuration file conf / zoo.cfg


Startup service

bin/zkServer.sh start

It is recommended to use at least three ZK instances, which are distributed on different servers, and can be mixed with WAF deployment, because the performance consumption is very small.


The Core Python pseudo code is:

#Create zookeeper node > > > zookeeper. Create (zk, "/ zk_waf_rule")

#Get node information > > > zookeeper. Get (zk, "/ zk_waf_rule")

#Define a listener, subscription configuration > > > def mywatch (zk, type, state, path):... Print "ZK:", str (type), str (state), str (path)... Zookeeper. Get (zk, path, mywatch)... > > data = zookeeper. Get (zk, "/ zk_waf_rule", mywatch)

Log processing

Log processing is also based on popular storm for real-time streaming processing, spark for off-line analysis, HDFS for off-line storage, if there is a need for real-time retrieval of the original log, ES can be introduced. For details, please refer to my previous article "building an open source Siem platform for enterprise security construction (I)". The most basic functions of this part are:

Statistics real-time interception Report

Statistics real-time access PV / UV

PV / UV of off-line analysis hour / day / week / month

Offline analysis of interception report of hour / day / week / month

Save the original log offline

The bonus function is:

Missing report of running HMM analysis

Missing of running semantic analysis


In the production environment, it is far more complex than this. Self research needs to be cautious. The data of BAT3 and the traditional top 4 WAF team members are not single digits:

Nginx layer needs a lot of optimization to improve performance

WAF rules need to follow up the latest loopholes, with a large amount of operation and maintenance

The stable operation of open-source big data framework storm, HDFS and spark is not easy and needs special maintenance

For the performance of nginx + Lua architecture, it is difficult to choose between false positive and false negative. For more optimization points, please refer to my last article