Hacking Book | Free Online Hacking Learning


zhongtong tongan vulnerability management system

Posted by graebner at 2020-02-27

As a large-scale group company integrating express delivery, logistics, e-commerce, finance and other businesses, in recent years, with the explosive growth of business and rapid expansion of IT personnel, the relative whole security attack area is also expanding, and the security loopholes in the enterprise are also increasing. The traditional vulnerability management method uses Excel to record and track manually. With the increase of system and vulnerability, this method exposes the problems of statistical difficulties and tracking difficulties. There is an urgent need for a system to manage the entire life cycle.

Every company basically has a vulnerability management system, but the understanding and positioning of the system are not the same. Most companies only let the R & D department develop a system with basic functions such as vulnerability entry, tracking and statistics. A few companies, such as Yixin, integrate asset management, risk calculation, security knowledge base and other management modules to manage the whole vulnerability life cycle. Open source vulnerability management system in some details, such as unified access authentication, access permission system has no access, in order to customize some functions, it needs to carry out secondary development of open source system. At present, the open source system is not a skilled technical stack of the team, and the development cost will increase.

Synchronize the data to the security management operation platform to show the security vulnerabilities to people at different levels

Developed a vulnerability processing module, which can input, classify, retest, archive and count the generated vulnerabilities online

Different kinds of notification and reminder methods are used, such as email, nail, unified authentication app and SMS notification. The notification and reminder are integrated into the whole vulnerability management life cycle, so that developers can receive vulnerability information in time

Enrich the content of asset management, include domain name, component, middleware, web framework, application contact, etc. in asset management, when a new security vulnerability occurs, quickly locate the business with vulnerability and notify the business contact

Combined with the existing resources and the understanding of vulnerability management, the Tongan vulnerability management system is developed, which runs through the whole software development life cycle, and helps security operators deal with daily security vulnerabilities efficiently.

The system is developed by react + go + MySQL + etcd

Docking system: CMDB, quality control center, automatic release

Docking security tools: active vulnerability scanning, Hunter passive scanning

Docking security products: firewall, HIDS, jumperserver, SOC

The overall architecture is as follows:

Figure 1 overall architecture of vulnerability management system

We mainly developed the modules of asset management, security test application, vulnerability handling, notification and reminder. And based on the data generated, it shows the overview of security vulnerabilities in the whole security system for colleagues at different levels.

Asset management module is the main support module of vulnerability management system. A perfect asset management module can help us quickly contact the relevant person in charge when we find the vulnerability, and quickly locate the problem system when a new vulnerability occurs. Bucket theory is a classic theory of information security. Defense should be all-around. A random omission of a point may lead to a total collapse. Hackers often attack the neglected assets. If the essence of infiltration is information collection, the key point of Party A's defense is the management of its own assets.

So how to do the asset management module well? The first step, of course, is to determine which assets we want to collect. Based on our experience, we should determine the following assets: domain name, IP, external mapping port, account number, authorized keys, internal network port, process, software application, web framework, component, RPM, jar package, database, as well as the personnel and Department of the corresponding assets.

So how to collect these assets? The main methods in the industry are active detection, passive traffic monitoring, host agent, configuration management database CMDB. Based on the existing resources, we currently use active detection, host agent, CMDB, and asset application approval.

Figure 2 asset collection

Active detection, we mainly use the external security scanning system (asset discovery module) for the external network, and the internal network scanning integrates the traditional masscan + nmap tools to ensure the scanning speed and accurate judgment of port services.

The main functions in this area are HIDS, open-source OSS EC and wasuh based on OSS EC. If they are to be put into online large-scale use, they need to invest in human research and maintenance. Currently, there is no special person to do this. In order to make up for this gap, we purchased the commercial host protection, and looked at the function of its asset list in the purchase stage. The collected information includes account number, authorized, keys, port, process, software application, web framework, component, RPM, jar package. The above information can be obtained through the API interface, which can be well connected with the asset management module.

CMDB (configuration management database) obtains application basic data, including domain name application, IP, person in charge and other application basic data. In order to meet the needs of the security department, the operation and maintenance department has opened the API of domain name resolution record, which can obtain new domain names on the Internet in real time.

At present, the process approval includes port mapping application and third-party system procurement security risk assessment application.

Some third-party system developers do not pay attention to security, the purchased third-party system often appears security vulnerabilities. Based on the painful lessons in the past, our approach is to conduct security assessment on the third-party system in the procurement stage. If the system to be delivered belongs to "dice system" (with many loopholes) and there is no security related input, security will use one vote veto instead of cooperating with it.

Through the above methods, we have summarized the asset data we ultimately want. In addition to the daily management of asset information, we also need to do an important thing [fast security intervention of new assets], to know that the Internet is being watched in real time, now we need to compare the "attack and defense" technology, who finds the security problems of new assets faster.

Our approach is as follows:

New domain name and IP automatically add external network security scan

Security testing of new assets

Give priority to handling all kinds of security alarms (including missed scan, WAF, intrusion detection) of new assets

Figure 3 new domain name reminder

As Party B of Party A, the information security department provides an important security service for the group and business. With the intrusion of information security into business development process, business departments pay more and more attention to the security of business system. When the development, product and QA realize that some functions or systems have security risks, they can apply for security test through the same security vulnerability management system.

Figure 4 application for independent safety test

Figure 5 security test application list

The security test before the system goes online is a very important part of Party A's security. In order to make the security test cover more widely, the vulnerability management system is connected with the cloud automatic release platform and the quality control center, and the security test is embedded in the system online process. All system releases must pass the security test.

First, the Quality Engineer shall carry out the test in the quality control center, the relevant data shall be synchronized to the Tong'an vulnerability management system, and the Security Tester shall carry out the security test and vulnerability entry. When there are super critical and high-risk vulnerabilities, the system cannot be released online without repairing them.

Figure 6 safety test exit flow

At present, the main sources of the loopholes on Zhongtong online are rookie emergency response center, Zhongtong emergency response center, internal security testing, external network security scanning and security public testing. Involving many group businesses, in addition to receiving the security loopholes of the group's own products, there are loopholes in the whole Zhongtong ecosystem, such as Zhongtong yuncang, Zhongtong commerce, Zhongtong express, Dayu international, etc.

Figure 7 processing flow of online vulnerabilities

1. Synchronize vulnerability information from various channels

2. After the security personnel confirms the vulnerability, they will submit the vulnerability on the system and allocate the vulnerability

3. Develop and confirm the vulnerability to be repaired, and propose retest after the completion of the repair. The retest application must indicate the cause of the vulnerability and the person directly in charge

4. Security personnel receive retest application to verify the vulnerability. If the vulnerability is not repaired, it will be rejected and notified

According to the security risk handling process of China Telecom express, the repair period of super critical vulnerability is 1 day, and the repair period of high-risk vulnerability is 2 days. If the repair period is exceeded, the performance evaluation of the responsible department and individual will be affected. Every month, the security department will take the lead to pull together the Department Manager, development and SQA that generate the online vulnerability, and recover the causes of the security vulnerability to avoid the recurrence of such incidents.

The main difference between the vulnerability before the system goes online and the handling of the online vulnerability is that the super and high vulnerability repair period before the system goes online is before the system goes online, and it cannot go online without repair.

Figure 8 vulnerability handling before the system goes online

Figure 9 security test results must be passed to be published

In the security test before the system goes online, we think that all the test data is valuable, so we developed the burp plug-in, which saves all the request packages and response packages in the security test into es and associates them with the security test application form. The benefits of doing so are as follows:

The security test can be traced back. If a security tested system finds a high-risk vulnerability after it goes online, there will be some bullshit at this time. Is it because the security is not detected? Or is it because the business has done something? We can go back to the security test data package at that time to find out whether there is a problem with the security personnel's test method or whether the development made other changes after the security test. If there are problems in the testing methods or technologies of safety personnel, the performance evaluation of safety testers will be affected, and the safety department will take the main responsibility. If the developers fail to comply with the release specifications, the business department will take the main responsibility, and the safety department will take joint and several responsibility.

The collected request URL, parameters and response package content can be used as a "brute force guess" of black box. The security test of black box is mostly based on function test. Some hidden interfaces are not put into function and JS. In addition to our white box code audit and security gateway, the "violent guess" of black box is also a powerful tool for security test due to its simple and crude characteristics.

The same type of vulnerability has certain common characteristics. The problem interface of a system is queryxxbyxbid. For example, the common URL jump, SSRF, file reading, parameter name may be URI, URL, file, etc. in this case, you can directly query the historical data in ES and directly pull out the matching request package to assist in security testing.

When a vulnerability breaks out, if it has certain characteristics, it is also a part that can be located here. What can be done is as follows: if it is a request package, the response package has the direct matching characteristics of the characteristics (for example, a vulnerability exists in a specific directory)

The purpose of notification alert is to:

Alert development to fix vulnerability

Let relevant leaders know the loopholes and hazards, and coordinate resources to repair the loopholes

Relevant vulnerability handling process can be notified to relevant personnel by email

In order to improve the timeliness of notifications and reminders, we use a variety of ways to notify and remind:


Zhongtong unified certification app

Nail application notice

Nail robot notification

SMS notification

In order to ensure that the vulnerability can be fixed on time, during vulnerability entry, in addition to notifying the default configured contact, it will automatically notify the corresponding level of leaders according to the vulnerability level, high-risk will be copied to the manager level, and super critical will be copied to the director level.

In the whole process of security test and vulnerability handling, formal email notification will be sent. Those without formal notice will be reminded in the form of pin and unified certification app notice, such as: vulnerability application retest, vulnerability rejection.

Figure 10 safety test application

Figure 11 email of vulnerability handling before receiving the system online

Figure 12 Daily Online non fixed vulnerability notification

Figure 13 nail notice

Figure 14 design of vulnerability knowledge base

The components of vulnerability knowledge base include: vulnerability type, vulnerability name, cause, harm, repair suggestion, utilization mode, case, Zhongtong real case. Among them, "Zhongtong real case" is extracted by desensitization after the entered vulnerability is repaired.

The vulnerability repository has several uses:

Development, test and product can learn the content of vulnerability knowledge base, understand the principle and repair method of vulnerability

Relevant contents and cases will be extracted from the regular safety development technical training

Relevant contents and cases will be extracted from the regular safety test training for quality

When developing a function, you can see what vulnerabilities will occur. Such as: login, registration, password retrieval, etc

With the introduction of principles and real cases, development can more easily understand the security risks in their own code, so as to improve the security skills of development.

In the future, we will continue to improve the level of vulnerability management automation, support access to third-party vulnerability reporting platforms (such as SRC, mending days, vulnerability boxes), and support automatic vulnerability retest. At the same time, we also hope to open the vulnerability management system to the ecosystem, and output the security capabilities through the vulnerability management system, including a simple easy-to-use asset collection module, to help companies in the ecosystem collect their own asset information more quickly, and to help enterprises in the ecosystem improve the level of security knowledge by sharing the accumulated vulnerability knowledge. Take the vulnerability management system as the output point to enable the safety of the whole Zhongtong ecosystem.

Author brief introduction

Opop, information security engineer of China Telecom express, is mainly responsible for the analysis and emergency treatment of information security incidents of China Telecom express, participating in information security assessment and security reinforcement, carrying out daily risk assessment and penetration test.

Team Introduction

Zhongtong security team is a young, upward, down-to-earth family striving for dreams. Our goal is to build a fully automatic information security intelligent perception response system and management operation platform based on massive data. We are committed to supporting the safe development of the whole ecological chain business (express, express, e-commerce, media, finance, aviation, etc.) of China Express Group. Our technology stack follows the development of the industry, including react, Vue, golang, Hadoop, spark, tidb, AI, etc. The data scale of the largest express delivery company in the world will also be a very big challenge. Our focus is not only on the first-line Internet companies in China, but also on the practices of Google, Facebook, Amazon, etc. in terms of basic security and data security.

Join us

If you are interested in our team or what we do, and hope to make achievements in the field of engineering technology, you are welcome to join us. We need talents in information security, distributed platform development, big data, risk control, product, operation, etc. base Shanghai, working place: Hongqiao Vanke center and Zhongtong headquarters. Resume delivery address: [email protected]