Unveiling the 12306 team: Fighting against ticket machines for a year

[[219796]]

On February 2, at the China Academy of Railway Sciences' passenger ticket system monitoring center, the three largest screens displayed the 2018 Spring Festival 12306 ticket sales, sub-control and queuing status. Image courtesy of Visual China

[[219797]]

In this hall, the curtains on half of the wall have been closed for years. There are three giant screens above the 106 screens of different sizes. The map of China is in the center, and the points representing cities are connected by dense arcs. Between the tables and chairs, black cables are winding, and server cabinets are standing in a circle against the wall. The red and green indicators of the data interface flash alternately, and the screen fluorescence flows on the white metal surface of the cabinet, and is also refracted by the glasses of the staff.

No one was talking, only the sound of keyboards, the hum of the machine's built-in fan, and the sound of hard disk reading and writing overlapped and resonated. The second hand of the wall clock clicked one step, and the last three digits of the eight-digit number on the bright red display on the big screen changed.

In this 1 second, the 12306 online ticketing platform sold hundreds of train tickets. The hall is the China Academy of Railway Sciences (hereinafter referred to as "RAS") passenger ticket system monitoring center, which monitors the health of China's railway ticketing system, which has the largest transaction volume in the world, day and night. The system sells 3 billion train tickets a year, and 12306 Internet ticket sales accounted for more than 80% at its peak.

The monitoring hall is broadcasting live the system’s biggest battle of the year: the Spring Festival travel rush.

Since the official sale of tickets for the Year of the Dog Spring Festival on January 3, 280 million tickets have been successfully sold through the Internet platform, accounting for 80% of all ticket sales through all channels. Across the country, people's requests and waiting, satisfaction or frustration when using 12306 will be captured by this platform.

It sold an average of 9.37 million train tickets per day and welcomed 124.04 million logins. On the day it reached its peak, it was refreshed 157.778 billion times, an average of 27 million times per second.

What determines whether you can buy a ticket?

Netizens often use "grabbed in seconds" or "sold out in seconds" to make fun of buying tickets, but few people know what actually happens in the period between clicking on the button to check the remaining tickets and the appearance of the final result.

In the eyes of the entire 12306 team, every second is extremely long. The unit of time they use to advance their work is "milliseconds".

"Beep", someone somewhere touched the screen, and a ticket check request signal was born. It started from the mobile phone, flew along the network, and arrived at the computer room of the Institute of Electronic Computing Technology of the China Academy of Railway Sciences. There stood rows of black servers, with densely packed chassis and circuits, and each area functioned in an orderly manner.

"Beep", the ticket check request signal travels in the circuit, like a billiard ball hitting several other balls, which in turn scatter and hit more balls. The ticket check request is sent to the passenger ticket transaction system of each station outside Beijing, which may be Tibet with its snowy mountains or Guangzhou with its fragrant flowers.

This step is more complicated than the flash sales of goods on the night of "Double 11". The inventory changes of goods are "simple addition and subtraction of numbers", and the less you bid for one, the less you have. The number of remaining tickets changes with the combination of departure and arrival places, and changes in real time. A set of algorithm models is needed to get an accurate answer. On October 12, 2017, the continuous transfer function of 12306 was launched, adding transfer stations to the existing change factors, and the requirements for the algorithm are more stringent.

"Beep", the calculation results are transmitted back to the calculation center. Meanwhile, people continue to buy tickets successfully. The storage of remaining ticket information is dynamic storage, which can be called "ever-changing". Shan Xinghua, deputy chief engineer of the Electronics Institute of the China Academy of Railway Sciences and director of the 12306 Technology Department, admitted that "there is a small probability" that even if the remaining tickets are found, they may be sold out in the next moment.

"Beep", the query signal gets a response from the system and is returned to the phone. The queryer selects the train number, clicks the buy button, and sends a purchase request signal. It joins similar signals rushing to the system, like a group of passengers rushing to the ticket hall. Speed ​​is everything, and it is affected by network speed and luck. If you are late, you will be locked out of the "hall".

To prevent the buyer from wasting their chance due to hesitation and regret, the “hall” will put a little more signal than the number of remaining tickets at one time. However, the number of buyers and remaining tickets will be basically the same within a certain period of time, “for fairness”.

Zhu Jiansheng, deputy director and researcher of the Institute of Electronics of the China Academy of Railway Sciences, said that if every "passenger" in this "hall" is required to buy a ticket as quickly as possible, there will be as many "ticket windows" as there are "passengers" - task threads to handle ticket purchase needs. This means huge computing costs.

At present, the 12306 online ticketing platform has several windows in the "hall" and long queues in front of the windows. Each signal is assigned to a window according to a certain algorithm and follows the queue to move forward in an orderly manner. At this time, the words "waiting in line" will be displayed on many people's mobile phones.

The core of this process is the "asynchronous transaction queuing" technology. The Railway Science Institute is "at the leading level in China" in this regard.

"Fairness" is the basic principle followed by technical developers. The order in which each ticket request signal enters will be recorded by the system, just like a popular restaurant issuing queue numbers to diners waiting for a table. Regardless of the speed of the subsequent process, the signal position is reserved by this "number plate" and first come first served.

"Ding", the ticket purchase signal is responded to and the order is generated. People click to pay and complete the transaction.

The large screen at the monitoring center showed that at 10:22 a.m. on February 2, 2018, the system's average response time to request signals was no more than 0.9 seconds, and the average waiting time for request signals was 1.8 seconds. Zhu Jiansheng told China Youth Daily and China Youth Online that after taking the average value, each user's situation is different. Some people may have to wait eight or nine seconds or even longer after clicking to buy, while others complete the entire transaction in less than one second, "without even realizing how complicated the process was."

The 40-day “Double 11” shopping festival

The technical team of 12306 has had contact with the Taobao team of Alibaba Research Institute, and both sides felt a little emotional, "These may be the two technical teams in China that understand each other best."

Shan Xinghua feels that her work experience during the Spring Festival travel rush is similar to that of Taobao engineers on the night of "Double 11". However, "they only have one day, but we have to face 40 days."

Train travel is the first choice for many people during the Spring Festival travel rush, and it is also the choice with the best overall cost-effectiveness. It is not easily affected by weather and is moderately priced. The railway system has discussed raising ticket prices significantly in the hope of using market leverage to reduce demand.

This motion was eventually abandoned. At the end of 2016, the National Development and Reform Commission issued the “Measures for the Supervision and Examination of the Pricing Cost of Ordinary Passenger Train Transport by Railway (Trial)”. “Fairness” was put first.

During the 2017 Spring Festival travel rush, the national railways transported a total of 357 million passengers, equivalent to moving the entire population of the United States and Canada. Online ticket purchases have gradually become a trend, and by 2017, the ticket sales volume of the 12306 platform reached 70% of all channels.

A 2018 Spring Festival online ticketing calendar hangs high in the monitoring hall, with each number the size of an A4 paper. The two days before New Year's Eve and the fifth to seventh days of the first lunar month are the peak times for returning home and returning home, and are specially marked in red.

Recently, Shan Xinghua gets up at 6:30 every morning, while her son is still asleep and the Beijing sky is almost completely dark. The monitoring center is brightly lit, and most staff members arrive before the online ticket sales start at 7 a.m. They get off work after 11 p.m., when the system is resting, and then they can rest.

The large and small screens in the monitoring center display the ticket purchase situation in real time. The most popular request queues will be paid special attention. Once the average waiting time is "abnormal", engineers will quickly start to find and solve the problem. The so-called "abnormal" sometimes means that the waiting time is only a few seconds or even a few tenths of a second different from the waiting time of the same popular queue.

Zhu Jiansheng introduced that this place is similar to a satellite launch center, and the flames of the "Spring Festival Travel Battle" are shining in the computing center.

China Railway Passenger Ticketing System has two computing centers, located at the two ends of a long corridor. The two centers have the same capabilities, regardless of master or slave, and each takes on 50% of the system tasks. If one crashes, the other will immediately take over its work to ensure that the service operation will not be interrupted by any accident.

Similarly, in-memory computing is "distributed." "That is, the same work is divided among many computers," said Shan Xinghua.

The traffic of 12306 is divided into peak and off-peak seasons. If the computing hardware and software resources are reserved based solely on the demand for computing power during the peak season, a large number of resources will be idle when the off-peak season arrives.

The elastic computing architecture was established, which can be expanded during the peak season to ensure sufficient usage, and compressed during normal times to avoid waste. The China Academy of Railway Sciences has developed its own cloud system, which is sufficient to meet the daily ticket purchase needs. However, they still rented Alibaba Cloud and Tencent Cloud to cope with the surge in computing power during peak seasons such as the Spring Festival.

"They only share the computing work, and the user information is still on our own 'cloud' to ensure (user information) security." Zhu Jiansheng said.

Shan Xinghua's most anxious period has passed. During the 2012 Spring Festival travel rush, the then Ministry of Railways decided to put about 20% of the tickets on the 12306 platform for Internet sales. "We could handle 1 million ticket purchases every day, and we thought it should be enough," said a former Ministry of Railways staff member in charge of this work in an interview with China Youth Daily in 2012.

The promoters of Internet ticket sales did not expect that at its peak, the 12306 platform sold 1.192 million train tickets a day, 20% more than the designed maximum daily ticket sales. Due to insufficient capacity, the system at the time could only successfully sell dozens of train tickets per second on average, and this number would drop below 10 when the computing pressure was high. That year, "12306" was listed on the "Annual Hot Words" list of many mainstream media, and "stuttering" and "failure" frequently appeared in reports about it.

During those days, Dan Xinghua "moved her bed to the laboratory". After the ticketing system was closed at 11 o'clock every night, the team optimized and tested the system overnight.

Six years have passed, and the original 400 servers of the 12306 online ticketing platform have increased to 2,000, and the system version has been upgraded six times. The regional network has been upgraded to a national network, and electronic payment has been introduced. Electronic tickets have been put into use, and ID cards can be swiped to enter the station.

The unwritten rules of the railway system are exposed on the Internet. Few people know that after the official issuance of train tickets for some trains, there will be a small-scale sale. The former are mostly seat resources at the departure and arrival stations of this train, while the latter come from a small amount of allocation from the railway ticketing system of each station along the line. This is called "reuse" of seats in the industry. After the 12306 website version 5.0 was updated, the railway system shared "reuse" resources with the online platform. 12306 users who encountered "sold out in seconds" during the pre-sale period may find a large number of remaining tickets when they check the website again when the departure date is approaching.

In Zhu Jiansheng's view, the year of 12306 is divided into two by the "Spring Festival Travel Rush". The preparation for this year-end "big test" will last for a year. The end of the "Spring Festival Travel Rush" is the beginning of preparations for the next year.

In the past year, the daily ticket sales capacity of the 12306 website has increased from 10 million to 15 million. The automatic seat selection function and the connecting ride function for EMUs were launched on October 12, 2017. Soon after the WeChat payment function was implemented, the ticket purchase notification SMS can be transferred to WeChat reading, and details from seat arrangements to the weather in both places can be displayed.

On July 17, 2017, the Internet meal ordering system was launched. After purchasing tickets, passengers can place orders through the 12306 system and select the products of a catering provider at a certain station along the way in advance. When the train stops at that station, passengers can sit in the car and wait for the takeaway to be delivered. Keeping the time accurate is the key to the entire system. "Usually, when we order takeaway through mobile phone apps, it takes more than ten minutes, and it's too late." Shan Xinghua said, "If the train delivers food more than ten minutes late, the train will leave."

12306 is 8 years old this year. It is developing from a new ticket purchasing channel to a more comprehensive "service platform".

Battle against ticket grabbing machines

When 12306 was quietly launched, it did not attract much attention. That was in 2010, when ".cn", representing the Chinese Internet, was officially written into the global Internet root domain name system. There are 457 million Internet users in China, of which 303 million are mobile Internet users. "Buying train tickets on the Internet is an inevitable step in an irreversible trend."

Zhu Jiansheng introduced that the development of mobile terminals has been the biggest trend in recent years, with mobile phone ticket purchases accounting for 70% of Internet ticket purchases.

12306 will continue to develop, but it will not be the only channel for railway ticket sales. According to Huang Min, deputy general manager of China Railway Corporation, this is the consensus of the entire system. The telephone booking system and offline windows will be retained. There are always people who lag behind in this digital age, and their needs are equally important.

On the other hand, the heaviest grains of sand will eventually be lifted up by the tide of the times.

Coordinating costs and demand has always been a major issue for the railway system. The data returned by 12306 will serve as an important reference to help the railway system make fine adjustments, saving costs while allowing everyone who is anxiously waiting to embark on their journey.

In 2009, the then Ministry of Railways Public Security Bureau arrested 4,069 ticket scalpers and eliminated 1,672 ticket scalping and counterfeit ticket production and sales gangs and dens. According to media reports, many of these gangs are from entire families in rural areas, and "ticket scalping makes more money than farming, and the sentence is worth it."

With the advent of online ticket sales, the education level of these ticket scalpers has not kept up. After the real-name system for train ticket purchases was implemented in 2014, railway police at many stations once noticed a decrease in ticket scalpers.

New times bring new problems.

"The ticket-swiping software has indeed put a lot of pressure on our system," said Shan Xinghua. The machine can click and send requests hundreds or even thousands of times per second, which greatly increases the workload of the system and also virtually increases the difficulty for other users who do not rely on the software to buy tickets.

In addition to ticket scalpers who hide in the dark and are familiar with technology, many technology companies, including well-known travel service platforms, have openly launched ticket grabbing software, which allows users to buy tickets faster and more steadily by paying a certain fee. In addition to a large number of criticisms, some comments also stated that this is the self-regulation of the train ticket market, and some people are willing to pay a higher price in exchange for more opportunities.

12306 tried to distinguish between humans and machines in the massive requests. In 2015, many users found that they had to pass a level consisting of 8 pictures to buy tickets on 12306. They had to accurately find designated items such as "peanuts" and "squirrels". In a later interview, the 12306 team regarded this as "an effective means to ensure fairness" because at that time, "identifying objects by looking at pictures" was a problem that only humans could solve.

They didn’t expect that this initiative would quickly become popular online. To this day, Zhu Jiansheng has to explain almost every time he is interviewed: “Many pictures on the Internet are photoshopped.” The pictures that have been forwarded tens of thousands of times by netizens include finding a certain Internet celebrity among a bunch of cone-faced people.

During the 2018 Spring Festival travel rush, no verification method that can replace the image verification code has yet appeared. 12306 said it would reduce the probability of their appearance, so that more than 80% of users would not have to see them.

The battle against "ticket grabbing machines" is also continuing. A large screen in the monitoring center displays the real-time risk control situation. On the pie chart of ticket purchase requests, red representing the "risk" of ticket grabbing machines and blue representing normal ticket purchase requests account for almost half each. The 11-digit IP numbers change and flash, and they are all dangerous elements discovered by the system.

Shan Xinghua said that within a certain period of time, if the number of requests from IP addresses or devices exceeds a certain range, the system will take them into account, restrict their access, or simply block them.

This also means that once you have a "black history" of using ticket grabbing software, your device and IP address are likely to be recognized by the system and restricted, making it even more difficult to buy tickets through normal channels.

"We can only try to limit it as much as possible." Shan Xinghua was cautious about this topic, but also said that "the ticket-grabbing software will not be so easy to use this year."

A huge amount of data flows to 12306, forming a "Spring Festival Population Migration Map" of China. "Beijing", "Shanghai" and "Guangzhou" are the three brightest points on this map, emitting tens of millions of tracks like light sources, pointing to a small site hidden on the map of China.

China Youth Daily·China Youth Online reporter Wang Mengying, Zhou Wei