Next time your boss asks you to refactor your system, let him read this article

[[155440]]

Everyone is unhappy with the old legacy system and wants to start over. But how to start over? Let's hear Yu Sheng's opinion.

Last month, a former colleague asked me: "Why didn't you redo the original system when you were here? We obviously had the capability."

I said, “We have also done a lot of things, including system stability, security, increasing redundancy, clarifying the responsibilities of each module, establishing an API communication mechanism, and organizing internal layers.”

He said, "Yes, but I still want to know, why don't you redo the system?"

So I asked: "After I left, it seemed that a lot of people were hired to redo the system? What was the result?"

He said: "As a result, the result is that we have to operate three or four systems at the same time to do business..."

From what I have seen, the preference to “tear down and rebuild” the original system is not only shared by programmers, but also by users. Take the job I had a few years ago, for example. When I first joined the company, the senior managers came to discuss with me the plan to rebuild the system: how many people, how much money, how long it would take to tear down the original system and rebuild it. After all, everyone suffers from the pain every day: slow speed, frequent errors, insecurity, customer complaints, bad architecture... So everyone wants to show the spirit of “daring to change the sun and the moon” and come up with a clean and thorough solution.

This feeling is understandable, but "rebuilding the system" has never been on the agenda during my tenure. The entire technical team has been doing "improvement" work, and the content is just as I said above: system stability, security, increasing redundancy, clarifying the responsibilities of each module, establishing an API communication mechanism, and organizing internal layers. I am fully confident of this choice, and in my opinion, if I decisively "start over", I may not be able to do better than my successor, and may even do worse, because "starting over" is by no means that simple.

As we all know, one of the difficulties in software development is to control complexity. However, complexity manifests itself differently in different fields. For pure Internet business or IT infrastructure, the complexity lies in the software itself, the formulation of the architecture, the selection of class libraries, the quality of coding, etc. For other IT systems - especially IT systems with rapid growth and increasingly complex business - the complexity does not lie in the software itself. Security, performance, and load issues are all applied to ready-made IT solutions. The real complexity comes from the business carried by the system itself. For example, the simplest ones are: what documents are in the system, what information do various documents carry, what scenarios are used, how do these documents circulate, what kind of constraints exist between various documents, and how to deal with abnormal situations to ensure the consistency of business data... If there are no accurate and stable answers to these questions, no matter how hard IT tries, it will be in vain.

For businesses that can already run in a standardized manner offline, or for work that has classic solutions (such as finance and warehouse management), this knowledge is ready-made and can be used directly. However, for emerging fields and emerging businesses, there is often no "classic solution". In addition, many companies grow very fast and did not build a good IT foundation (actually the business architecture foundation) at the beginning. The typical situation is: the business concepts are unclear, the business logic layer is also messy, and many systems simply treat the database as the business logic layer (this is not a joke, because the database cannot shirk responsibility). As a result, the chaotic business logic is attached to the bad IT system, and the chaos is compounded and eventually becomes a mess. For IT, problems with existing businesses emerge in an endless stream, and every time a problem occurs, a lot of energy is spent on finding clues to "solve the case"; for business, new businesses often affect the original business, but no one knows whether it will affect it or how it will affect it. The other side of the increasingly large system is that the internal system is becoming increasingly disordered, and the complexity and maintenance costs are growing rapidly, far beyond the controllable range.

Paradoxically, many people do not address the root cause of the problem, assess business complexity, sort out business logic, and sort out business relationships. Instead, they think that starting over and building a new system will solve the problem. People who hold this view usually have a misunderstanding of the relationship between the system and the business.

For those who want to "start over", the relationship between system and business is a bit like the relationship between vehicle and personnel: after driving a car for a while and feeling that it is not good, I want to change to another car, which is very natural. However, today, when informatization has penetrated into every corner of work, the relationship between system and business is far from being as distant as "vehicle to personnel", but more like the relationship between "pacemaker to people" or "artificial bones and muscles". They are already entangled together, and the more and wider the system supports the business (regardless of quality for the time being), the more closely the two sides are entangled. The difficulty of replacing a pacemaker or artificial bone is much greater than that of changing a car, so it needs to be carefully considered, and it cannot be rashly decided to replace the pacemaker simply because it is "not so good". The same is true for the system.

If you want to completely transform a legacy system with a poor foundation, I have a few experiences to refer to:

***, you must have very excellent business personnel and developers.

For business personnel, they must not only be familiar with the operations at hand, but also understand the logic behind the operations, and need to go beyond their job duties and think about their business from a global perspective (sometimes even making their own operations more complex to improve system security and other benefits), so that they can truly grasp the complexity of the business. For developers, they must be able to fully understand domain knowledge and have superb programming skills to deal with legacy codes, dare to take action instead of shrinking back, and act cautiously instead of rashly - if the technical ability of the original system developer can be scored 30 points, and the technical requirement for the newly developed system is 60 points, then the technicians who want to successfully transform the legacy system often need to have a score of more than 80 to be competent.

Second, “starting from scratch” is often not as good as “gradual improvement”.

The so-called "gradual improvement" means that everyone first confirms the design blueprint of the future system through discussion, and then needs to develop a transitional interface layer. Therefore, the newly developed modules must be developed strictly in accordance with the new specifications (this is what I mean by "clarifying the responsibilities of each module, establishing the API communication mechanism, and organizing the internal layers"), and at the same time, they must be connected to the original system through the transitional interface layer. The original modules, while clarifying the business logic, cut out appropriate interfaces as needed, and migrate part by part after passing the test. In the end, the new system is slowly pieced together like a puzzle and takes shape on the last day, rather than being built with a flat floor. In this process, the most important thing is to find the right entry point and build a suitable interface or interface layer. These tasks are like scaffolding for building a house. Even if they will not be used later, they cannot be omitted in the middle and must be treated carefully. Of course, this is a challenging job - I once encountered a query of cross-database table connection in a database transaction. This bad design seriously hindered the progress of splitting a single database instance into multiple instances. Looking back, it was really like a nightmare.

If you have your own insights on transforming legacy systems, or have any interesting experiences in this process, please leave me a message.

***Recommend an interesting book. In fact, whether it is software development or social change, people often like to come up with a "straightforward" and "thorough" solution to the status quo they don't like, but these solutions are often not the ones that are truly successful. At the end of World War II, what exactly happened in the world, what problems did it encounter, and how did it rebuild the social order? The ninth volume of Guangxi Normal University's "Utopia" series, "Year Zero: 1945, the Moment of the Birth of the Modern World", uses detailed writing to comprehensively record the scenes after the "end of the war". Many of the scenes will surely surprise readers - in many cases, "civilization" can be said to have been beaten back to its original form, and the name "Year Zero" is truly worthy of its name.

Editor's note: Everyone is unhappy with the old legacy system and wants to start over. But how to start over? Listen to Yu Sheng's opinion. This article was first published on his WeChat public account Yu Sheng Yi (yurii-says)

Last month, a former colleague asked me: "Why didn't you redo the original system when you were here? We obviously had the capability."

I said, “We have also done a lot of things, including system stability, security, increasing redundancy, clarifying the responsibilities of each module, establishing an API communication mechanism, and organizing internal layers.”

He said, "Yes, but I still want to know, why don't you redo the system?"

So I asked: "After I left, it seemed that a lot of people were hired to redo the system? What was the result?"

He said: "As a result, the result is that we have to operate three or four systems at the same time to do business..."

From what I have seen, the preference to “tear down and rebuild” the original system is not only shared by programmers, but also by users. Take the job I had a few years ago, for example. When I first joined the company, the senior managers came to discuss with me the plan to rebuild the system: how many people, how much money, how long it would take to tear down the original system and rebuild it. After all, everyone suffers from the pain every day: slow speed, frequent errors, insecurity, customer complaints, bad architecture... So everyone wants to show the spirit of “daring to change the sun and the moon” and come up with a clean and thorough solution.

This feeling is understandable, but "rebuilding the system" has never been on the agenda during my tenure. The entire technical team has been doing "improvement" work, and the content is just as I said above: system stability, security, increasing redundancy, clarifying the responsibilities of each module, establishing an API communication mechanism, and organizing internal layers. I am fully confident of this choice, and in my opinion, if I decisively "start over", I may not be able to do better than my successor, and may even do worse, because "starting over" is by no means that simple.

As we all know, one of the difficulties in software development is to control complexity. However, complexity manifests itself differently in different fields. For pure Internet business or IT infrastructure, the complexity lies in the software itself, the formulation of the architecture, the selection of class libraries, the quality of coding, etc. For other IT systems - especially IT systems with rapid growth and increasingly complex business - the complexity does not lie in the software itself. Security, performance, and load issues are all applied to ready-made IT solutions. The real complexity comes from the business carried by the system itself. For example, the simplest ones are: what documents are in the system, what information do various documents carry, what scenarios are used, how do these documents circulate, what kind of constraints exist between various documents, and how to deal with abnormal situations to ensure the consistency of business data... If there are no accurate and stable answers to these questions, no matter how hard IT tries, it will be in vain.

For businesses that can already run in a standardized manner offline, or for work that has classic solutions (such as finance and warehouse management), this knowledge is ready-made and can be used directly. However, for emerging fields and emerging businesses, there is often no "classic solution". In addition, many companies grow very fast and did not build a good IT foundation (actually the business architecture foundation) at the beginning. The typical situation is: the business concepts are unclear, the business logic layer is also messy, and many systems simply treat the database as the business logic layer (this is not a joke, because the database cannot shirk responsibility). As a result, the chaotic business logic is attached to the bad IT system, and the chaos is compounded and eventually becomes a mess. For IT, problems with existing businesses emerge in an endless stream, and every time a problem occurs, a lot of energy is spent on finding clues to "solve the case"; for business, new businesses often affect the original business, but no one knows whether it will affect it or how it will affect it. The other side of the increasingly large system is that the internal system is becoming increasingly disordered, and the complexity and maintenance costs are growing rapidly, far beyond the controllable range.

Paradoxically, many people do not address the root cause of the problem, assess business complexity, sort out business logic, and sort out business relationships. Instead, they think that starting over and building a new system will solve the problem. People who hold this view usually have a misunderstanding of the relationship between the system and the business.

For those who want to "start over", the relationship between system and business is a bit like the relationship between vehicle and personnel: after driving a car for a while and feeling that it is not good, I want to change to another car, which is very natural. However, today, when informatization has penetrated into every corner of work, the relationship between system and business is far from being as distant as "vehicle to personnel", but more like the relationship between "pacemaker to people" or "artificial bones and muscles". They are already entangled together, and the more and wider the system supports the business (regardless of quality for the time being), the more closely the two sides are entangled. The difficulty of replacing a pacemaker or artificial bone is much greater than that of changing a car, so it needs to be carefully considered, and it cannot be rashly decided to replace the pacemaker simply because it is "not so good". The same is true for the system.

If you want to completely transform a legacy system with a poor foundation, I have a few experiences to refer to:

***, you must have very excellent business personnel and developers.

For business personnel, they must not only be familiar with the operations at hand, but also understand the logic behind the operations, and need to go beyond their job duties and think about their business from a global perspective (sometimes even making their own operations more complex to improve system security and other benefits), so that they can truly grasp the complexity of the business. For developers, they must be able to fully understand domain knowledge and have superb programming skills to deal with legacy codes, dare to take action instead of shrinking back, and act cautiously instead of rashly - if the technical ability of the original system developer can be scored 30 points, and the technical requirement for the newly developed system is 60 points, then the technicians who want to successfully transform the legacy system often need to have a score of more than 80 to be competent.

Second, “starting from scratch” is often not as good as “gradual improvement”.

The so-called "gradual improvement" means that everyone first confirms the design blueprint of the future system through discussion, and then needs to develop a transitional interface layer. Therefore, the newly developed modules must be developed strictly in accordance with the new specifications (this is what I mean by "clarifying the responsibilities of each module, establishing the API communication mechanism, and organizing the internal layers"), and at the same time, they must be connected to the original system through the transitional interface layer. The original modules, while clarifying the business logic, cut out appropriate interfaces as needed, and migrate part by part after passing the test. In the end, the new system is slowly pieced together like a puzzle and takes shape on the last day, rather than being built with a flat floor. In this process, the most important thing is to find the right entry point and build a suitable interface or interface layer. These tasks are like scaffolding for building a house. Even if they will not be used later, they cannot be omitted in the middle and must be treated carefully. Of course, this is a challenging job - I once encountered a query of cross-database table connection in a database transaction. This bad design seriously hindered the progress of splitting a single database instance into multiple instances. Looking back, it was really like a nightmare.

If you have your own insights on transforming legacy systems, or have any interesting experiences in this process, please leave me a message.