51CTO Exclusive: In-depth analysis of the mystery of Ctrip’s database crash

Perhaps no one knows that Edison invented the battery 113 years ago today, but the news of the collapse of Ctrip.com on May 28, 2015 spread like a virus.

Positioning database

Ctrip’s official Weibo account announced that at 11:09 a.m., some of Ctrip’s servers were attacked by an unknown attacker, causing the official website and APP to be temporarily unavailable.

There are online rumors that all of Ctrip's server data was physically deleted during the outage and that backup data was unusable. Ctrip said that no data was lost and that booking data was intact.

A senior security consultant from Anhua Jinhe, a professional database security company in China, talked about the Ctrip security incident and said: "Judging from the time, the data has not been restored for such a long time. It is very likely that there is a problem with the database."

Internal or external

However, what exactly is the problem with the database? This is a question that everyone is concerned about. A Weibo post revealed that a root directory was accidentally deleted during server maintenance, resulting in data loss. An Hua Jinhe senior security consultant believes: "Ctrip uses MySQL database, and has a high availability mechanism through MySQL's own replication. If it is because a directory file was accidentally deleted, it can be directly restored through the backup device."

When deleting or modifying a piece of data, the condition is added incorrectly, causing the entire data to be deleted, and then because of the association problem, the data in the backup server is also deleted, which makes data recovery very, very difficult. We can even imagine that Ctrip does not even have internal audit equipment, but all actions in the database can be traced back.

An Hua Jinhe senior security consultant analyzed: If the attack comes from outside, it can be divided into three situations. First, according to the cross-site scripting vulnerability that Ctrip has reported before, hackers will use a reverse injection method to put the code in the background, which may cause damage to the background when it is called in the future.

Second, turn on the debugging mode on the application server and save the transaction card number and information. If it is not done by an internal staff, it means that hackers have captured this information on the application server, and then hackers may have planted a backdoor program on the application server. If a backdoor program is planted, it can directly connect to the backend server. However, this possibility is relatively small, because Ctrip should have already checked it after a similar problem occurred last year.

The third complex APT attack is that when operation and maintenance personnel log in to the external network, they download hacker applications through phishing. If they access the internal network while working, it will cause indirect attacks.

Common problems of Internet companies

The Internet industry emphasizes rapid response, so many Internet companies have adopted a strategy of integrated development and operation and maintenance like Ctrip. Although this has reduced development and response time to some extent, there are great risks in management.

A senior security consultant from Anhua Jinhe suggested: First of all, the management system should be standardized, and the integrated development and operation and maintenance model should be changed first. Development and operation and maintenance have their own working environments. Development work should be more carried out through the test system rather than directly contacting the production system. "

The author learned that some Internet companies' operation and maintenance equipment, development equipment, and equipment connecting to the external network are all available in one package. Although rapid response has been achieved to a certain extent, problems often also arise here.

Operation alarms are also essential. Then add some control and prevention equipment like bastion hosts. When performing operation and maintenance on the server, the operation and maintenance personnel will not be allowed to connect directly to the database. Finally, install a database firewall to prevent dangerous operations by hackers and internal personnel.

Sword of Damocles

In just one year from 2014 to now, Xiaomi, Ctrip, NetEase, and 12306 have stimulated our nerves time and again. Data is like a gold mine, with countless information distributed on the Internet every day, but all companies are wielding hoes but ignoring the sword of Damocles above their heads.