Hacking Book | Free Online Hacking Learning

 Home

using open source software to build large-scale waf cluster for enterprise security construction (including open class video)

Posted by verstraete at 2020-02-24
all

*Original author: brother Dou, this article belongs to the freebuf original award program, and can't be reproduced without permission

Preface

This article and freebuf open class summary, review the actual situation of open class, please click me

Most Internet companies can't develop their business without web services, and web protection has become a problem that many security teams of party a need to face. In this paper, combined with practical experience, how to use open-source software to build a large-scale WAF cluster is described. In order to be more general, some self-developed modules use open source components instead. The development and maintenance of self built large-scale WAF cluster needs considerable manpower. You need to be cautious. Both BAT3 and the traditional top 4 have solutions. You can really consider a mature business plan.

WAF cluster architecture

The main components are: four layers of load balancing, WAF service, WAF log processing, WAF configuration management, which support cluster deployment and rapid horizontal expansion.

Four layer load balancing

From the perspective of basic network security, we should strictly limit the direct use of public IP to provide external services, so that the attack surface will be very large and uncontrollable. All tested services should use load balancing to publish web services to the outside world, and other services are not directly published to the outside world, which can strictly control the attack surface of hackers. The main functions of the four layer load are:

The common open source solution is LVS. LVS is the abbreviation of Linux virtual server, which means Linux virtual server. It is a virtual server cluster system, which supports three modes: vs-nat, vs-tun and vs-dr

Vs-nat: it is to change the destination address of the IP header of the data packet sent from the client to the IP address of one of the RS's on the load balancer, and then process it to rs. after RS processing, the data will be handed over to the load balancer, and then the load balancer will change the original IP address of the data packet to its own IP address, and change the destination address to the client's IP address. During this period, regardless of the incoming traffic, Or the flow out must go through the load balancer

VS-TUN: encapsulate a new IP header (only for the purpose of IP) from the client and send it to RS. When RS receives the packet, it unlocks the header of the packet and restores the packet. After processing, it directly returns to the client without passing through the load balancer. Note that since RS needs to restore the packet sent by the load balancer, it must support the IPTUNNEL protocol, In the RS kernel, you must compile and support the iptunnel option.

Vs-dr: both the load balancer and RS use the same IP for external service. Because the load balancer needs to change the layer 2 packet header, the load balancer and RS must be in a broadcast domain, which can also be simply understood as on the same switch

Because we hope that only LVS will be exposed to the public network, the deployment of back-end WAF and web services will not be limited by too many physical cabinet network segments. It is recommended to use vs-nat mode. Although all requests and responses will pass LVS, LVS may become the bottleneck of the whole system, but the actual experience is that LVS performance is good enough, and the single machine performance bottleneck is often the network card, and the CPU has spare power when the 10 Gigabit network card is full 。

As an example, the LVS is configured as follows:

/sbin/ipvsadm -C /sbin/ipvsadm -a -t 201.114.5.36:80 -r 192.168.1.100:80 -m -w 1 /sbin/ipvsadm -a -t 201.114.5.36:80 -r 192.168.1.101:80 -m -w 1 /sbin/ipvsadm

Note: VRRP virtual VIP can be used between two LVS to achieve high availability. Please supplement more optimization of production environment.

WAF services

Common open source WAFS include mod_security and nginx + Lua. Due to the excellent performance of nginx and the programming level flexibility of lua, nginx + Lua has become the choice of more people. A code example is as follows:

location / {   default_type 'text/plain'; access_by_lua ' local waf = require "waf" ； waf.execute() '; }

All the real WAF work is done by the WAF module written by Lua. This part is written very clearly in loveshell (https://github.com/loveshell/ngx_lua_waf), and Chunge's is not bad (https://github.com/openresty)

I have slightly simplified the installation process:

Install luajit

wget http://luajit.org/download/LuaJIT-2.0.0.tar.gz

tar zxvf LuaJIT-2.0.0.tar.gz

cd LuaJIT-2.0.0

make

make install PREFIX=/usr/local/lj2

ln -s /usr/local/lj2/lib/libluajit-5.1.so.2 /lib64/

wget http://luajit.org/download/LuaJIT-2.0.0.tar.gz tar zxvf LuaJIT-2.0.0.tar.gz cd LuaJIT-2.0.0 make make install PREFIX=/usr/local/lj2 ln -s /usr/local/lj2/lib/libluajit-5.1.so.2 /lib64/ 安装devel_kit

wget https://github.com/simpl/ngx_devel_kit/archive/v0.2.17rc2.zip

unzip v0.2.17rc2

wget https://github.com/simpl/ngx_devel_kit/archive/v0.2.17rc2.zip unzip v0.2.17rc2 安装lua-nginx-module

wget https://github.com/chaoslawful/lua-nginx-module/archive/v0.7.4.zip

unzip v0.7.4

wget https://github.com/chaoslawful/lua-nginx-module/archive/v0.7.4.zip unzip v0.7.4 安装pcre

wget http://blog.s135.com/soft/linux/nginx_php/pcre/pcre-8.10.tar.gz

tar zxvf pcre-8.10.tar.gz

cd pcre-8.10/

./configure

make && make install

wget http://blog.s135.com/soft/linux/nginx php/pcre/pcre-8.10.tar.gz

take zxvf pcre-8.10.tar.gz

cd pcre-8.10/

./configure

make &make install

nginx

wget 'http://nginx.org/download/nginx-1.2.4.tar.gz'

tar -xzvf nginx-1.2.4.tar.gz

cd nginx-1.2.4/

export LUAJIT_LIB=/usr/local/lj2/lib/

export LUAJIT_INC=/usr/local/lj2/include/luajit-2.0/

./configure --user=daemon --group=daemon --prefix=/usr/local/nginx/ --with-http_stub_status_module --with-http_sub_module --with-http_gzip_static_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --add-module=../ngx_devel_kit-0.2.17rc2/ --add-module=../lua-nginx-module-0.7.4/

make -j8

make install

wget 'http://nginx.org/download/nginx-1.2.4.tar.gz'

tar -xzvf nginx-1.2.4.tar.gz

cd nginx-1.2.4/

export LUAJIT_LIB=/usr/local/lj2/lib/

export LUAJIT_INC=/usr/local/lj2/include/luajit-2.0/

./configure --user=daemon --group=daemon --prefix=/usr/local/nginx/ --with-http_stub_status_module --with-http_sub_module --with-http_gzip_static_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --add-module=../ngx_devel_kit-0.2.17rc2/ --add-module=../lua-nginx-module-0.7.4/

Make -j8

Make install

Modify nginx configuration file nginx.conf (the following is just a principle demonstration)

HelloWorld rule

Access blocked

configuration management

Configuration management is actually the most easily ignored part of the whole system, but in fact, it is a very important part.

Configure synchronization model

There are two most common ways to configure synchronization: push and pull. Push mode is the most convenient, but it can not guarantee the timeliness in large-scale distributed system; pull mode performance is poor. Can we balance the two ways?

ZK

Zookeeper is a distributed coordination service and an important component of Hadoop and HBase. It is a software that provides consistency services for distributed applications. Its functions include configuration maintenance, domain name service, distributed synchronization, group service, etc. Zookeeper is widely used in configuration management of distributed system. Zookeeper supports publish subscribe mode, which can be regarded as one to many relationship: multiple subscriber objects listen to a subject object at the same time. When the subject object changes its status, it will notify all subscriber objects, so that they can automatically update their status. Publish and subscribe mode enables publishers and subscribers to independently encapsulate and change independently. When an object changes, it needs to change other objects at the same time, and it does not know how many objects need to change, it can use publish and subscribe mode.

In the most simplified design, we can make the configuration of each WAF exactly the same. This advantage is that you can add servers at any time, any server downtime will not affect the WAF service, and the architecture of the whole system will be very simple. Each WAF only needs to use Python script to subscribe to the configuration in ZK. Once the configuration changes, get the latest configuration, update the relevant configuration file, and send a signal to nginx to hot load the new configuration file (nginx - s reload).

ZK is very simple to install and use

Download it and unzip it.

Edit the configuration file conf / zoo.cfg

tickTime=2000 

dataDir=/var/lib/zookeeper 

clientPort=2181

tickTime=2000

dataDir=/var/lib/zookeeper

clientPort=2181

"21551;" 26381;"21153rd;

bin/zkServer.sh start

It is recommended to use at least three ZK instances, which are distributed on different servers, and can be mixed with WAF deployment, because the performance consumption is very small.

tickTime=2000 dataDir=/var/lib/zookeeper clientPort=2181 initLimit=5 syncLimit=2 server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888

The Core Python pseudo code is:

Log processing

Log processing is also based on popular storm for real-time streaming processing, spark for off-line analysis, HDFS for off-line storage, if there is a need for real-time retrieval of the original log, ES can be introduced. For details, please refer to my previous article "building an open source Siem platform for enterprise security construction (I)". The most basic functions of this part are:

The bonus function is:

summary

In the production environment, it is far more complex than this. Self research needs to be cautious. The data of BAT3 and the traditional top 4 WAF team members are not single digits:

*Original author: brother Dou, this article belongs to the freebuf original award program, and can't be reproduced without permission

© 2023