Why does 12306 crash from time to time?

01

2019 is about to pass. Have you bought the train tickets to go home for the New Year?

I know many of you didn't get it, and I can feel the desperation in your hearts.

A few days ago, 12306 crashed. Many people cursed 12306. Many people also asked me in the background of my official account why 12306 always crashes and is as useless as my period.

It's obviously just a simple ticket selling software, how did it become like this? On QQ and WeChat, hundreds of millions of people are chatting online at the same time, interacting passionately, and watching all kinds of short videos.

On the other hand, there is no problem with hundreds of millions of people shopping crazily on Double Eleven, so why is it that as soon as 12306 makes a move, the user experience is so terrible?

Let the hardcore teacher Banfo give you some basic knowledge about how much business pressure and challenges 12306 is facing.

Of course they can't stand so many of you rushing in. Who can stand it?

Although this article will contain a lot of extremely hardcore technical terms, I will try to keep it as simple as possible. Everyone must listen carefully and take more notes. When you brag about them at the dinner table during the New Year, these will be the ultimate treasure. No need to thank me.

It doesn’t mean that you can buy tickets after reading this article. In fact, buying tickets is a mystery.

I just want to say that it would be better if you die with understanding.

02

Many people compare 12306 with Double Eleven, and think that 12306 can handle so many orders on Double Eleven, but 12306 cannot, which is obviously because of its inadequate technical level.

My friend, you took a detour at the beginning. The business model of 12306 is fundamentally different from that of Double Eleven.

This difference makes 12306 much more difficult than Double Eleven~

If the difficulty of Double Eleven is the human mode, then 12306 is roughly equivalent to the eighteenth level of hell, and the difficulty of digging another pit is required.

First, no matter how large the traffic is during Double Eleven, it is just a pure online business . What is a pure online business? All users place orders on the website or APP, and the entire data is actually a closed loop.

This means that Double Eleven is actually just a pure online traffic problem, which is relatively simple to solve, just like a simple child who is easy to bully.

But 12306 is not like that. 12306 is not just an APP and a website. All transactions generated by people in offline ticket offices and offline machines will also affect the entire 12306 data system.

In fact, in real life, many people who buy train tickets to return home, such as hard-working farmers and workers, do not know how to operate online. They can only queue up offline all night, which is very hard. Therefore, 12306 must also take care of their feelings and cannot cut off offline business.

This results in 12306 itself being a complex business that shares data offline and online, and its complexity is an order of magnitude higher than that of Double Eleven.

Compared with the purely simple Double Eleven, 12306 is like an old fox that has been ravaged by society. You never know when they will do something weird.

It's like a pure boy going on a date with his boyfriend for the first time. He is afraid that he won't come, but also afraid that he will do something else.

03

Second, putting aside the online and offline aspects, after all, this is bullying 12306, let’s talk about the planning and predictability of the business itself.

If you think about it carefully, you will find that Double Eleven is a business with clear plans and operation nodes, but 12306 is not.

The Double Eleven event is not only on November 11th. In fact, it started a month or even several months ago. A large number of users have already paid the deposit, and a large number of merchants have also locked in inventory and sales. It’s just that the final settlement is carried out on November 11th.

This means that Double Eleven is facing a highly certain task. As long as there is certainty, no matter how large the traffic is, it will not be a particularly difficult problem.

When you know when difficulties will occur, the difficulties are no longer difficult.

The real difficulty lies in the unknown.

What is unknowable? 12306 is unknowable.

Because you can never calculate how many people will go to which place on which day. For a person returning from Zhejiang to Shandong, the route and time he chooses in order to go home will be very strange and changeable.

He can buy tickets from Zhejiang to Shandong, from Zhejiang to Shanghai to Shandong, from Zhejiang to Nanjing to Shandong, from Qiandao Lake to Shandong, from Zhejiang to Beijing to Shandong, from Zhejiang to Heilongjiang to Shandong, and even from Zhejiang to Guangdong and then fly back to Shandong, as long as he can return to Shandong, oh my green onions.

He can accept departure No. 1, departure No. 2, departure No. 3, departure No. 5, departure No. 10086, as long as he can depart.

This means that no one knows how sudden the demand traffic will be, or how complex the purchased content will be.

Let me give you another example and you will understand. Isn’t Weibo powerful? Such a large amount of traffic and so much attention every day.

But why does Weibo often go down when a celebrity cheats, gets married, or gets negative news? Is it because of lack of technical capabilities or insufficient server resources?

Neither. It’s because this kind of traffic is sudden and no one knows that there will be such an explosive growth, so the server crashed.

This is just like during the Water Splashing Festival, when everyone is mentally prepared, but you throw boiling water on them.

Who can endure this?

Therefore, it is recommended that all celebrities report their affair on Weibo before cheating, so that they can rest assured that they are cheating and we can also feel at ease while watching the incident.

He is fine, and so am I.

04

Third, e-commerce business is not a business that requires all processes to be completed at one time, but 12306 must be completed at one time, which further increases the difficulty.

Think about it, e-commerce shopping is not actually a one-time thing.

A typical e-commerce shopping process is to browse, negotiate with the merchant about price, place an order, negotiate with the merchant about shipping and courier, logistics delivery, buyer receiving the goods, and negotiate with the merchant about after-sales service.

Even if we exclude things like quarrels, the e-commerce shopping process still has many nodes.

The entire process will take 24 hours for the fastest delivery, even the next day.

This means that the data pressure on e-commerce is not as great as imagined. It can be processed asynchronously. All payments can be completed first, and then the shipment can be processed slowly, and then the after-sales service can be handled slowly.

Just like you want to browse Bilibili and boil water at the same time, you can boil the water first and browse Bilibili while the water is boiling. This is the advantage of asynchrony, which allows multiple tasks to be run in parallel without affecting the user's core experience.

The goods will be shipped within 3 hours or 5 hours after you place the order, which will not affect the users and has a high fault tolerance rate.

But 12306 is different, selling tickets must be completed immediately.

From placing an order, to payment, to locking the ticket, everything is done in one go. There is no such thing as me placing an order for a ticket and then being told whether it is successful or failed 2 days later. It is really frustrating for consumers.

Therefore, 12306 does not have such a buffer period. It needs to face all the traffic directly and meet all the needs of everyone in the shortest time. All business must be processed and completed in a very short time, which means it will be filled up all of a sudden.

With the same 100 million users and 4 steps, e-commerce can handle it calmly in 4 steps, with each step handling 100 million traffic, and it can be handled in multiple hours or days.

12306 can only handle 400 million at a time, and there is no relief. You can imagine the pressure.

We all know that no matter how powerful something is, it will break if it is forced to be filled.

Don't let your imagination run wild. I'm talking about the highway.

Fourth, the inventory management of e-commerce business is relatively simple, while 12306 is extremely complicated, so complicated that you will get depressed if I give you a brief explanation.

Think about it, as an e-commerce platform, although it is difficult to manage goods, the overall planning itself is nothing more than adding, deleting, modifying and checking fixed products. There are as many as there are. If there is payment, it is reduced by one. If the quantity is increased or returned, it is increased by one. At most, there is a small probability that the last product will be bought by two people at the same time. These are all trivial matters.

12306 is a completely different level of difficulty, and the difference in difficulty between the two is roughly equivalent to the battle between Paramecium and Namek.

Let me give you an example. If you are a student in Beijing and your home is in the south of Beijing, you will go home for the Chinese New Year.

Just pick a train heading south from Beijing, the G65 high-speed train, starting in Beijing and ending in Zhuhai, with a total of 17 stations, and a total journey time of 10 hours and 55 minutes.

For this one route, there are 17 stations, and you can get on and off at any station. How many possibilities are there?

Because it is impossible to make a single-stop loop by train, that is, you cannot go from Beijing to Beijing.

So it is from 1 to 16, a total of 136 possibilities. Please note that the same scenario in e-commerce has only 4 possibilities: add, delete, modify and check.

It is very daunting to keep real-time inventory for such a route.

If someone buys a route from Beijing to Guangzhou, then all corresponding inventories will be reduced by 1, but the inventory from Guangzhou to Zhuhai will not be reduced.

If someone buys a ticket from Wuhan to Zhuhai, the inventory of all routes along the route from Wuhan to Zhuhai will be reduced by 1, but the inventory from Beijing to Wuhan will not be reduced.

If someone buys a ticket from Shijiazhuang to Luohe West, then there is no need to change tickets from Beijing to Baoding and from Beijing to Shijiazhuang, and there is no need to change tickets from Luohe West to Zhuhai.

All other possibilities will change because any route passing through these two stations will be affected.

In actual business, such changes will cause real-time changes in the entire inventory, and it is a P-level data change. If you have a little understanding of databases, you know how terrible this kind of data change is in terms of resource consumption. A 1GB spreadsheet running a query can crash many high-performance computers. Early spreadsheets were even limited to 65,000 rows to prevent the computer from crashing.

It would be against advertising laws to even mention the resources required to obtain data of this level.

So why does 12306 need maintenance from 11 pm to morning? If such a database were not maintained and protected on a daily basis, it would have been completely destroyed long ago.

Every day, 12306 is fighting for life.

So next time you buy tickets, be more pampering and gentle, because what you are buying is not a ticket, but the engineers’ hair.

05

Fifth, business deduplication requires a lot of judgment.

Strictly speaking, e-commerce business does not require real-name registration for users, nor does it require deduplication of user identities. As long as you pay, the goods will be shipped if they are available, unless there are purchase restrictions on goods, which will be intercepted slightly. However, when facing scalpers, it is just a symbolic struggle. After all, everyone is out there selling things and will not go against money.

But 12306 is different. The model of 12306 is that everyone is subject to purchase restrictions.

There are purchase restrictions for the same route and time.

Take the Beijing-Zhuhai route as an example. 12306 does not allow one person to buy a large number of tickets from Beijing to Zhuhai on the same day. This is unfair to others, so it is necessary to limit people's departure time and purchase routes.

So the question is, if you want to add purchase restrictions, then you have to cache the person's current purchase information and time, and every transaction of this person must be deduplicated and matched with his current itinerary.

This consumption of data resources is extremely horrible.

Moreover, there is another practical problem in duplicate checking, which is that 12306 itself allows people other than the original owner to buy tickets. That is, I can buy tickets for my parents, and my parents can buy tickets for me, as long as the passengers are added. This means that the information of the same person can be added by different buyers at different time points, which in turn brings huge computing pressure.

It's just like life, it's so difficult.

Sixth, no company is qualified to compete with 12306 in terms of traffic.

Many people really think that Double Eleven is the peak of traffic, but in fact it is not. 12306 is the most terrifying peak of traffic.

Why? Because the business model of e-commerce will not lead to repeated clicks by users, while 12306 is repeatedly clicked by all users all the time.

For example, when you buy something on Double Eleven, do you just buy it and leave? If you can’t buy it, you’ll just curse a few words and then leave.

A user has a limited number of clicks. Even if you have been single for 30 years and you are given the freedom to click, how fast can you click?

You should know that for the system, every click is a data exchange.

The click traffic faced by 12306 is much larger. When you buy a ticket, the operation will be constantly refreshed. When you check the remaining tickets, all databases must be run every time to help you synchronize the current remaining ticket information. The load and computing power are enormous.

Moreover, a lot of people are now using ticket-grabbing software.

The so-called ticket grabbing software is based on the principle of continuously using a machine to read the data interface of 12306. The speed of the machine is definitely dozens of times faster than the speed of your hands after being single for 30 years. It can swipe hundreds of times per second. One person using the ticket grabbing software can cause data pressure caused by thousands of people swiping together.

The major ticket-grabbing software companies have tens of millions of users in total, and 1.4 billion people are more than enough to generate traffic of tens of billions of people.

Do you know how heavy the traffic load of 12306 is?

On 12306, everyone is Naruto, with a clone every day.

06

Some people may ask, since the challenge is so great and 12306 is so difficult, why not learn from advanced technologies from abroad? Why not increase investment in servers? Why are you still making excuses?

Why doesn’t 12306 introduce advanced technology from abroad?

The answer is actually very simple. It cannot be covered even abroad.

As early as 2012, 12306 had an open tender with no budget limit. As long as it could solve the problem, the world's top institutions came to bid, but in the end they basically gave up.

Because of the technical environment at that time, no one could solve this problem.

Many foreign technologies are indeed advanced, but no country or company in history has ever accepted the challenge of tens of billions of traffic from 1.4 billion people. None of the world's top companies you can name can withstand such strong real-time transaction traffic.

Some of them have larger traffic, but just like e-commerce business, they can operate asynchronously, do not require identity uniqueness, and do not have such complicated route inventory calculations. You also know what kind of garbage level the high-speed rail and subways abroad are. They have never encountered the problems we encounter.

This is a very realistic thing.

When it comes to the travel needs of 1.4 billion people in China, everyone is the same.

Speaking of this, I thought of a joke in our industry. An early Facebook engineer returned to China to join Alibaba. Before leaving, he said he was going to save Alibaba's data system. But after he came back, he found that the data challenges he encountered at Facebook were at the kindergarten level compared to Alibaba.

In terms of data challenges, the data challenges we face are definitely the strongest in the world, and in many cases, there is no doubt about it.

So why not add a server? If technology is not enough, hardware can make up for it.

There are three core issues facing adding servers.

First, adding servers only increases storage capacity, but does not solve database problems . This is like a woman taking 10 months to give birth to a child, but that does not mean that you can find 10 women to give birth to a child within 1 month.

Second, how to drive these servers? At that time, Alibaba Cloud's world-leading technology was the breakthrough of driving 5,000 servers simultaneously, making it one of the world's three largest clouds.

You know, Alibaba Cloud is only facing Double Eleven, but the challenge of 12306 is even more terrifying, as it requires more servers to be driven at the same time, which also involves technical challenges.

In addition, Alibaba Cloud has indeed participated in the construction of 12306.

Third, the issue of cost.

12306 usually performs well throughout the year and only crashes occasionally during major holidays. In order to cope with the few major holidays in a year, you purchase so many servers with high depreciation rates that are never used during normal times. This is a waste of money.

China Railway itself is making huge losses and is continuously subsidized by the state. In this case, huge costs are invested to meet short-term needs. Needless to say, right?

Look at Weibo next door. They would rather be overwhelmed by traffic every time than maintain a large number of servers for a long time. According to the financial report, Weibo is profitable every year, and the amount is settled in billions. They all have this attitude. You know how difficult it is for 12306.

And this is all taxpayers' money.

In the end, the essence of the ticket buying problem is still the relationship between supply and demand.

With so many people in the country having to complete so many trips in such a short time, it far exceeds the carrying capacity of the railway itself. In this situation where supply is less than demand, no matter how resources are allocated, there is no way to solve the supply and demand problem.

There are only so many things and everyone wants them, so what can we do?

Add money and continue expansion?

You should know that many routes are only full during the Spring Festival, and are usually empty and loss-making. It is not cost-effective to waste a lot of money on routes that already have surplus traffic just to meet the short-term travel needs of a small number of people.

With this money, we should invest in more areas.

Infrastructure like 12306 is naturally criticized. If it is done well, no one will praise it. If it is done a little bit badly, it will be criticized to death. This is the tragedy of infrastructure. Everyone has unreasonable expectations.

What's more, it is already an amazing operation that 12306 has achieved this level with such a small budget.

It is not rational to criticize without considering the financial and technical costs.

Why don't you talk about the Indian trains selling tickets?

Author: Half Buddha Immortal

Official account: Xianren JUMP (ID: xrtiaotiao)