The evolution and thinking of Taobao Native R&D model

The origin and development of DX

DX was hatched from the homepage to solve the problem of consistency and dynamism at both ends of the business. After several upgrades and iterations, it gradually completed the multi-level jump from the homepage dynamic solution - the basic link dynamic solution - the group co-construction dynamic solution - the group mobile group standard Native development model. With the increasing number of connected businesses and developers and the increasingly complex and diverse usage scenarios, the technology scope has also expanded from the end-side SDK to the dynamic technology system and then to the Native development model.

Technology selection for Taobao

This may be the most complex app on the planet. We divide all of Taobao's businesses into three categories: core domain, shopping guide domain, and open domain. Different scenarios have different requirements for performance experience, stability, and delivery efficiency. There is no "technical silver bullet" here, only the best solution suitable for the target scenario.

What challenges have we faced in the past two years?

As a core domain standard technology selection company, what challenges has DX faced in the past few years?

First, let’s look at the big business background:

1. As business competition intensified, the group upgraded from the aircraft carrier strategy to the App matrix strategy, which placed higher requirements on delivery volume and efficiency;
2. With the development and popularization of live broadcast/short video/AR/3D technologies, mobile terminals are entering an immersive/rich interactive experience, and the advantages of the Native technology stack are becoming more obvious;
3. Taobao's shopping efficiency is getting higher and higher, the user's visit time/frequency is decreasing, and most purchasing decisions happen outside Taobao. Taobao has gradually become an ordering/price comparison tool. When the content field has a large amount of entry traffic and user time, it begins to close the loop of e-commerce transactions. The former traffic channels have become direct competitors. Taobao will also face the dilemma of "water without a source and a tree without roots". Therefore, Taobao has been building its own content/community in recent years to allow more consumer pre-purchase decisions to happen on Taobao.

These three business backgrounds profoundly affect every business connected to DX as well as potential new businesses. Therefore, over the past two years, our work has mainly focused on upgrading R&D models and experience.

First of all, there were two main problems in upgrading the R&D model:

1. DX's own technical system is incomplete. Before 2019, DX could only be described as a dynamic client solution, including an end-side SDK and a component platform. However, with the access of more and more new businesses, the bottleneck of DX's own incomplete technical system became more and more obvious. A large number of new businesses required repeated construction of end-side containers, platform construction and other infrastructure, resulting in high access costs and low returns. The containers and platforms built by each business were difficult to settle and reuse, further exacerbating the difficulty of subsequent technology integration.

2. The R&D model is too single. With the expansion of immersive and rich interactive experiences, many businesses have bid farewell to the past model of heavy display and light interaction, and have begun to upgrade their experience to moderate interaction. In particular, content-like fields represented by browsing and subscribing, which were split from Weitao, require a more immersive browsing/interactive experience to attract more users to stay and consume content. However, our past construction-based approach was not able to provide good R&D support for such scenarios.

Next is the experience upgrade. Although we cannot decide the visual interaction and experience design of specific business scenarios, we can do several things to achieve excellent experience:

1. Provide standardized components with excellent out-of-the-box experience, such as calendar/rich text/complex nested container/Page Indicator, etc., which greatly reduce the adaptation cost brought by refined experience, such as Dark Mode/silver version font automatic enlargement adaptation, etc.
2. With the rapid development of rich media/end intelligence and the increase in low-end devices in the sinking market, we continue to optimize the performance of the DX rendering pipeline to improve the basic browsing and interactive experience in a wide range of business scenarios;
3. Provide more complete performance tools to help developers detect, troubleshoot, and optimize relatively "black box" template performance issues early.

Thoughts on ProCode and LowCode

When we took on new scenarios such as new subscriptions and browsing, we found that in this type of content-based interactive scenario, there are rich interactions, linkages and transitions. DX's previous lack of dynamic logic expression and the module-based LowCode development model could not support its efficient development. The core contradictions are dynamic logic, complex containers and internal linkages, multi-person parallel development and module references. So after a series of discussions and thinking, we have the following picture:

Based on the problems faced in the new scenarios, we decided to stratify the R&D model. Business scenarios with strong customization/high dynamic logic requirements are developed in ProCode mode, and the powerful and professional DX IDE+Git is used for engineering multi-person collaborative development, which can debug UI and dynamic logic (event chain). Business scenarios with weak customization/high reusability are undertaken in LowCode mode, and the visual construction platform is used to drag and drop to generate views and layouts, and configure corresponding data sources and rules. At the same time, the component library/capability library is continuously abstracted and precipitated, so that more and more interactions/functions/layouts can be generated by dragging and dropping in a visual way.

The future we envision is that when business colleagues have a new business idea, they can build and configure it themselves through the LowCode mode, select a small number of users online for grayscale verification, and after verifying that the product direction is correct, invest more professional and technical personnel to do further product customization and upgrades to produce greater business results.

ProCode “Troika”

To make the ProCode model take shape, we must solve the problems of dynamic logic, complex containers and corresponding R&D support, so we have   The "three pillars" of ProCode: IDE, list container, and event chain.

IDE

The first is IDE. Here is some background information. Before making IDE, DX had a template platform, which was mainly used to carry out template development/debugging/releasing work. So why do we need to make another IDE?

First of all, strictly speaking, the original DX platform is not called IDE, but Editor. From a relationship point of view, IDE includes Editor. Let's use a picture of VSCode to illustrate the difference between the two:

Secondly, although the original Web Editor (Monaco) on the DX platform also supports most LSP APIs, it does not have engineering/project systems, file systems, code context understanding, debugging and other capabilities, and is limited to the browser environment. The IDE runs in the local environment, has rich extension capabilities and plug-ins, and can provide complete R&D life cycle support.

Therefore, in ProCode mode, our technology selection is based on Katian IDE (compatible with VSCode API) to obtain more view expansion capabilities. In the future, we can also achieve more efficient/fast R&D and collaboration through flexible deployment methods (remote development, cloud deployment).

Currently, DX IDE has been integrated into many important businesses of the group, including Taobao. After using the IDE, the R&D efficiency of these developers has been greatly improved, especially the efficiency of multi-person collaboration and debugging has been increased by an order of magnitude.

List Container

The second is the list container. In the past, the container based on the construction had a feature that the container/page layout was described by the data protocol, that is, the data layout was integrated. This worked well in the scenario of simple page structure (such as a single list page). With the construction, the floor could be configured quickly, and the end-side rendering was driven by the protocol. The server did not need to perceive the business structure too much (the default was a single list layout). However, in the case of complex page structure (such as multi-layer multi-tab nested containers), especially in the case of complex interactions, the entire data structure would become too complicated, and the payload of the data protocol would be increased, while the frequency of changes in the page layout was much less than that of the business data.

In this scenario, it is obviously inappropriate to couple the two together. Therefore, based on the principle of single responsibility and "separation of static and dynamic", we rethought and designed it, unified the description of layout (including container/page layout) in template XML, and let the data return to pure data structure. At the same time, we also built a new powerful list container in DX, which can be written as follows:

 < RecyclerLayout
 userId = "recycler"
 width = "match_parent"
 height = "match_parent"
 dataSource = "@data{itemList}"
 backgroundColor = "@triple{@getEngineStorage{backgroundColor}, @getEngineStorage{backgroundColor}, '#ffffff'}"
 columnCount = "2"
 columnGap = "4"
 isOpenPullToRefresh = "True"
 refreshPullText = "Pull down to refresh..."
 refreshReleaseText = "Release to refresh..."
 refreshLoadingText = "Loading..."
 loadMoreFailText = "Load failed"
 loadMoreLoadingText = "Loading..."
 loadMoreNoMoreDataText = "In the end"
 onPullToRefresh = "@dxEventHandler{'refreshTest'}"
 onEndReached = "@dxEventHandler{'loadMoreTest'}"
 >
 < Template
 if = "@subdata{span}"
 sticky = "True"
 colspan = "2"
 name = "aa"
 >
 < TextView
 width = "match_parent"
 height = "100"
 text = "@data{text}"
 textSize = "15"
 backgroundColor = "#FFFF55"
 borderColor = "#FF9911"
 borderWidth = "2"
 / >

 < /Template>
 < Template
 if = "@subdata{recyclerInsertItems}"
 name = "recycler_sub_template1"
 version = "2"
 / >
 </RecyclerLayout> ​

Currently, the list container has covered multiple important business scenarios within Taobao and the group, and supports capabilities such as nested containers/linked frameworks/dynamic templates/slots. We will also promote it to more businesses next year.

Event Chain

The third is the event chain. In the past, DX only supported UI dynamics, and limited logic control could only be achieved through expressions. Other logic changes depended on the release of the version. As the business has higher and higher requirements for the timeliness of release and launch, the demand for dynamic logic has become stronger. Therefore, in response to this demand, after repeated discussions and thinking, we comprehensively considered Apple compliance, performance, stability, and development efficiency, and provided an event chain solution.

In the event chain solution, developers use atomic capabilities and expressions in the eventchain.json corresponding to XML for logical assembly. During the R&D period, the IDE provides intelligent perception capabilities of the event chain JSON (including completion/prompt/jump/formatting/folding/highlighting, etc.). The event chain playback function can be used to graphically display the execution process of the event chain and the input and output parameters and context of each atomic capability to facilitate problem troubleshooting. After development is completed, in the compilation phase, the DX compilation service will compile the JSON into binary and send it down together with the template binary to ensure performance and consistency. On the end side, the event chain engine is responsible for binary parsing and atomic capability scheduling.

Thoughts and practices on performance optimization

In the past two years, DX has become more and more powerful and has more and more functions (animation dynamics, event chains, complex nested containers, live broadcast/video components, 3D components, etc.). Will the performance advantage of DX be affected or reduced? How can we keep adding new features while improving performance? Faced with increasingly complex templates, how can we help developers have a handle to troubleshoot and optimize performance issues instead of being at a loss?

This is about the rendering pipeline of DX. If Flutter is a revolutionary that takes over the rendering of the entire system, then DX is a reformer based on the native system rendering. The main design idea of ​​DX is to improve the performance bottlenecks and problems in the system's native rendering pipeline/mechanism based on Platform First, and unify both ends.

Judging from the time consumption distribution of each stage of the DX pipeline, the main bulk is in loading and rendering, so we mainly use asynchronous pipelines to solve the time consumption problem of loading, use asynchronous drawing and high-performance components to solve the time consumption problem of rendering, and use computing resource management to ensure the reasonable allocation of resources as much as possible.

Asynchronous pipeline and asynchronous drawing are mainly to enable more work to be executed in child threads or scheduled in parallel, reduce the burden on the main thread, and make rational use of multi-core computing resources. In the DX list container, we have built-in this capability, which can be used out of the box. In the business container, the business party needs to call the corresponding API according to our recommended timing/the timing deemed appropriate by the business to obtain asynchronous scheduling.

In terms of high-performance components, we mainly optimized the image components and developed our own rich text components. Especially for the rich text components, we implemented self-testing and self-drawing based on the system TextKit/TextLayout, and provided capabilities such as TextSpan/ImageSpan/expand and collapse. Compared with native components, memory usage is reduced (iOS 70% Android 8%) and performance is improved (iOS 20% Android 10%).

In terms of computing resource management and control, we mainly designed and implemented an off-screen computing resource management and control framework to implement video/live broadcast/animated image broadcast control issues, and provided rule configuration capabilities. In the case of multiple videos, the iOS CPU occupancy rate was reduced by 66%/Android by 25%. It has been launched in some important information flow scenarios of Taobao.

In terms of performance tools, in the past we mainly performed various low-level optimizations at runtime and performance warnings during compilation. When encountering template performance issues, the business side and DX colleagues usually need to log separately to see the time consumption and locate the problem, which is time-consuming and labor-intensive for both parties. This situation occurs partly because DX is highly integrated with Native. During a rendering process, a large amount of DX internal code and business side custom code are intertwined, making it impossible to quickly and accurately locate the problem; another part is that the DX rendering process is relatively black box for the business side, and the business side has no effective means for troubleshooting and optimization.

In order to get rid of this scenario, we need to white-box the entire template rendering process during the development phase, and use visual graphics to show the rendering time and attribution of each stage of the template (DX internal or business-customized part, specifically which Widget and which stage/DataParser/EventHandler). At the same time, through the accumulated common performance issues, we automatically analyze the template and provide certain optimization suggestions.

Technology

To sum up, this is the big picture of the current DX technology system. At the R&D support layer, we use IDE and platform to meet the demands of both ProCode and LowCode production modes. At the core runtime layer, we continue to enhance the view capabilities of containers/components/layouts while introducing certain dynamic logic capabilities. Through joint construction, we continue to precipitate the atomic capability library. At the business framework layer, we also cooperate with the Taobao team to introduce complex state management, process orchestration and other capabilities into the DX technology ecosystem, and businesses can be connected on demand.

Future Outlook

From the perspective of technological evolution, we believe that the next stage of DX will achieve three things:

• Standardization. Currently, DX's DSL tends to be a private standard, which has a high development threshold and is not conducive to expanding the developer group. Next, we need to align with the client industry standards to attract more developers.
• Modernization. Currently, DX describes UI through imperatives, while the industry has moved towards declaratives. From the perspective of technology evolution, we must also embrace declaratives to further improve development experience and efficiency. In addition, in terms of R&D support, we built DX IDE from 0 to 1 this fiscal year. Installing and running it locally has greatly improved the R&D efficiency of complex pages/templates, but we will not stop improving R&D efficiency. Next, we will try to achieve installation-free/out-of-the-box/workspace sharing through cloud deployment to further improve R&D and collaboration efficiency.
• Integration. Currently, we have achieved partial engineering integration through DX IDE. In the future, in order to further improve the efficiency of front-end and back-end collaboration and to make Serverless one of the basic capabilities of the client, we must achieve end-cloud integration. In an integrated R&D environment, with the same technology stack, we will develop end-side and cloud-side codes at the same time, simplify the development cost of end-cloud remote scheduling, and deploy to different targets through compilation technology during the release period, to improve the efficiency of front-end and back-end collaboration and the autonomy of end-side technical solutions.