Text layout performance improved by 60%, Inline Text technology principle and implementation

Alipay clients have a strong demand for dynamics. Regardless of the iOS or Android platform, redistributing software packages is difficult to meet the requirements of product operations in terms of time and efficiency, so client dynamic technology came into being.

Cube originated from the dynamic demands of Native pages, and its product form is represented by Cube cards. With the emergence of mini-programs, Cube has integrated the Alipay mini-program technology stack, and its product form is a lightweight Alipay mini-program solution (compared to the Web mini-program that uses browsing as the core). As a lightweight engine, Cube mini-programs are small in size, fast in startup, and low in memory usage. We use self-developed rendering technology and support CSS subsets to achieve these features.

Unlike browsers, the input of the Cube Mini Program Engine is a Mini Program DSL (which can be understood as the language of the Mini Program specification). The product after construction (the product mainly consists of a JS file and related resources) is output as a user interface and subsequent interactions (continuous user input and UI output). Cube Mini Programs continuously iterate to support style sheets. Inline Text can support a large number of CSS styles in a small package size (the main so is only 2.8 MB), and the layout drawing is almost completely consistent with that of a web browser.

Inline Text

What is Inline Text? This starts with the layout engine. The layout engine, also known as the typesetting engine, is a software component that is responsible for obtaining tags or elements combined with styles to produce corresponding layout results such as position, size, and hierarchy. This typesetting result can be used to output to a display or a printer. The Cube engine uses two layout engines. Cards use Yoga for high-performance layout scenarios, and mini-programs use Flow Layout. Among them, Flow Layout supports Flow layout (streaming layout), and the representative style is display:block, which is the default layout behavior of elements such as div in the Web.

An important task of the layout engine is text typesetting, and most non-Web rendering technology stacks (including the old version of Cube) use platform-layer text layout objects. In the layout stage, the platform-layer text objects are constructed according to the calculated style, and the width and height are calculated and then passed back to the layout engine to complete the layout; the created platform-layer objects are used for subsequent drawing.

Using platform layer objects will cause a certain degree of performance loss in the layout phase. This loss mainly includes two parts: one is the consumption of text layout calculation by the operating system API; the other is the performance loss caused by the interaction between the layout engine and the platform layer. This implementation results in text only existing in rectangles, making it difficult to flexibly mix text and graphics, and it is also difficult to add color and style to part of the text in a paragraph. This is also the meaning of styles related to Inline. Typical styles related to Inline are: display:inline, display:inline-block, display:inline-flex, float:left. The following two pictures are typical representatives of this capability.

In addition to the obvious capabilities, compliance with relevant Web standards has always been one of the technical goals of the Cube applet. Let's take an example to illustrate. Please see the following code (the HTML code applet can be replaced with view and image tags)

 < div style = "font-size: 60px;" > < img src = "http://xx" / > < /div>

<img> is a replaceable element. The default value of its display property is inline , but its default resolution is determined by the original width and height of the embedded image, making it like inline-block. Its placement is determined by the baseline of the text. Although there is no text in this example, img relies on the baseline of the text when it is laid out, so font-size will affect the position of img. Since font-size is inherited, if it is a dynamically issued element, accidentally setting a font size that does not meet the expected size in the ancestor will affect the layout and drawing results. This small example illustrates the impact of text on an engine during layout. It can be seen that text has many impacts on page layout. As a layout engine, Flow Layout needs to perfectly replicate these behaviors, and Cube has taken another step towards standards.

To summarize the above points, the Cube team decided to enhance the text-related capabilities on the Cube engine, including the measurement and calculation of text width and height, typesetting and layout, which can enhance CSS capabilities while improving layout performance. These text-related capabilities (or features) are collectively called Inline Text, which restores the old usage habits of front-end developers for text and other styles when using the Cube engine, greatly improving the development experience. As a result, the Cube applet can carry more applet, thereby achieving greater business value.

Implementation details

The implementation of Inline Text in the Cube engine is divided into two parts: one is to remove the docking text platform layer part in Flow Layout, and enhance it to directly measure the width and height of the text, which we call text layout. The other part is drawing. This implementation still uses the platform layer drawing, but uses a lower-level API to reduce the calculation of the operating system and directly call the drawing API as much as possible. We call it text drawing. In order to better understand the relationship between these two parts and the implementation details, let's first understand rasterization and text self-drawing.

Rasterization

We all know the concept of pixels. The computer world is discrete. Every line and every word needs to be drawn on a pixel point. This process from vector graphics to pixels is called rasterization, just like a grid. You can refer to the following picture:

Text self-drawing

Corresponding to the self-drawing of text, the existing rendering process uses the platform layer (OS) API for text layout and rendering, which is the logic in the platform module under the Cube engine. During layout, the corresponding platform layer object is created, the area of ​​the text rectangle is calculated through the platform layer, and the result is passed back to the layout engine, and the layout is completed after the result is obtained. During subsequent drawing, the created platform layer object is used for drawing.

Text self-drawing means that the width and height of the text are directly measured by the layout engine, and the text is placed in the appropriate area to complete the layout; subsequent drawing is also directly rasterized through the font information (some information describing the glyph).

In the process of technical iteration, Cube first performs the layout of the first half, and the second half is completed together with the self-drawing of the entire engine. Therefore, the current method is to use the platform layer API to draw after the layout engine layout is completed. The process can be referred to in the following picture:

Text layout

In order to correctly place the text on the screen or in the correct position of a rectangle, we divide it into two steps. The first step is to calculate the width and height of each character, and the second step is to perform layout calculations (place text and other things such as pictures). It should be noted that when the real program is running, it sometimes uses a more efficient calculation method and does not necessarily need to fully measure the width and height of each character, such as monospaced fonts.

1 Calculation of width and height of a single character

Common sans-serif, serif, and monospace are called universal font families. They are called universal font families, which means that they represent a series of font families (FontFamily), and the layout engine will find the appropriate font in the operating system. The font family is called a font family, which means that a font sometimes provides multiple versions to meet different style needs. After selecting one version of a font, we call it Typeface. Typeface contains a lot of text, and each single text (for the same Unicode character) is called Glyph. Each Glyph contains a single text information. For contemporary TrueType vector fonts, a single text information is composed of multiple Bezier curves, which we call vector graphics. According to the size of the font, a clear size of text can be obtained. After applying some styles and rasterizing, the width and height of a single character can be obtained.

Here is a small example of comparison. For the same font, the font author will generally provide two fonts, one with normal width and the other with bold. The bold is called font-weight. The bold and non-bold represent two Typefaces. According to the previous introduction, there will be two sets of Bezier curves to describe the same Unicode character. The following are the curves of the non-bold and bold character Y. You can imagine that if the font author provides a separate Typeface for italic, it will also have a separate curve description like bold.

The following is an introduction to TextStyle. According to the above description, in addition to Typeface, some text styles are needed to determine the width and height of a character in order to determine its size. We call these styles TextStyle. After understanding the concepts of Typeface, we will find that drawing a font requires other parameters besides Typeface and text size, such as: color, font-size, whether to apply FakeItalic, whether to use FakeBold (explained later), and many other parameters. This information other than Typeface is abstracted as TextStyle. After the Typeface is determined, the TextStyle information is needed and then rasterized to know the final rendered width and height of a glyph (due to the characteristics of vector Typeface, rasterization is not necessarily required to obtain the width and height, and it can also be calculated through a specific ratio).

According to the previous description, if the author of the font does not design a Typeface corresponding to font-weight for the font, nor does he design a Typeface for italic, what should be done when the user specifies the relevant style? At this time, Fake-related transformations come into play. It can directly transform the original font curve to achieve the effect of looking thicker and more inclined. At this time, FakeItalic and FakeBold will be applied to the font. Through the pre-made transformation function, TextStyle now includes FakeItalic and FakeBold, otherwise it does not. Sometimes whether to apply Fake-related transformations is decided by the layout engine itself.

In order to calculate text-related capabilities, we introduced some libraries, mainly Skia and Harfbuzz. Skia cuts off all drawing parts and only retains the relevant abstractions for text, while Harfbuzz is a library used to measure the width and height of text. Harfbuzz uses the Typeface and Glyph interfaces provided by Skia. The following briefly introduces the abstraction of SkTypeface for Typeface and the details of font initialization on various platforms.

In the Skia library, SkTypeface abstracts the detailed implementation libraries FreeType (Android) and CoreText (iOS) under Typeface, and provides the most basic functions, such as converting Unicode to the index of a specific glyph in the font (abstract for Glyph), for example: directly obtaining or calculating the abstraction of a single text outline through the index, reading the system font file through the FreeType library on Android, and reading the relevant table data (Typeface metadata) through the CoreTextAPI on IOS for direct construction.

The details of font initialization are consistent with the browser implementation logic. On Android, the font information in /etc/fonts.xml is used. Old versions of Android systems use /system/etc/fonts.xml and other similar locations. It defines the fonts provided by the operating system, as well as the corresponding font-family, font-weight, font-style, and language. After initialization, it is ready to find the appropriate Typeface for subsequent layout. On iOS, the CoreText API is used directly to obtain the relevant font list and information.

Now we have the ability to measure a single character, find the corresponding Typeface and TextStyle through the system and user input, and finally use Harfbuzz to calculate the width and height of the character. (This is for the sake of ease of understanding. In fact, the width and height of multiple characters can be measured at once)

2 Layout Calculation

First of all, the layout tree and style sheet, which will not be introduced in detail here. The layout tree is constructed by adding relevant elements through CSS styles. Before layout, each element has its own ComputedStyle calculated by the style sheet, and the block element and flex element start layout according to their default behavior. The inline element starts layout in its own LayoutBlockFlow, determines a FallbackList according to the font-family in ComputedStyle, and then measures the width of the text based on the font, text, white-space and other related information. ICU is introduced here. This library is used to segment different languages, determine whether to break sentences, whether punctuation marks should be placed in the next line or left in the previous line, and then determine the TextRun according to the interval where the Unicode characters are located, and finally form a LineBox for subsequent drawing. The description here is relatively concise, and there will be a separate article to expand on the layout process later.

Text drawing

1 Docking platform layer drawing

Due to the characteristics of Inline Text, the rendering link must be transformed. The entire rendering link is not introduced in detail (please refer to other related articles). The main structure of the Cube rendering process is the rendering tree (internally called RenderTree). This tree has many nodes, which are used to describe the parent-child relationship. Among them, the text node is a leaf node at this stage. When recursively drawing, the original text is a rectangle, so the entire Text uses the same border drawing process and background drawing process. Due to the inline characteristics, when drawing to the text node, the background and border drawing process needs to be drawn according to each line, adding a layer of loop. There are many details to deal with for the background and border line breaks. Here, the drawing effect of the Web needs to be aligned. If you want to support more complex text features in the future and make the text node a non-leaf node, you need to further enhance the drawing process.

After the text is laid out in the previous stage, three data structures are generated for each Text node (span) through the information on the LineBox for drawing:

 class TextStyle {
 int textColor ;

 float textSize ;

 int fontWeight ;

 int textDecoration ;

 int fontStyle ;

 TextShadow textShadow ; // Used to describe TextShadow related properties

 float alpha ;

 Padding padding ; // Contains left top right bottom

 } style ;




 class TextRun {

 int typefaceId ;

 int start ;

 int end ;

 float width ;

 };




 class TextLine {

 String text ;

 float originX ;

 float originY ;

 float ascent ;

 TextRun runs [];

 } lines ;

The above pseudo code is shown to better understand the details of the docking platform layer drawing. TextStyle is the other information required for drawing except Typeface introduced before. Due to the characteristics of Element, the styles under the same Element are consistent, so a Text node has a TextStyle, and then a bunch of TextLineInfo, each representing a line of data, each line has many TextRun, representing the corresponding Typeface of each segment and the beginning and end of the substring. The platform side Typeface has been created in advance during layout, and the corresponding object can be directly obtained through typefaceId during the drawing stage. Among them, the Android platform directly uses the Android API android.graphics.Typeface to create it during layout, and the iOS platform directly uses the wrapped CoreText related font object to draw because SkTypeface is directly abstracted for the CoreText object.

Package size

According to the positioning of the Cube engine and the requirements of the mini program for text rendering capabilities, Inline Text has deeply optimized and customized Skia, Harfbuzz, ICU, Freetype, etc. Cube has done a lot of work on the package size, and has done a lot of tailoring for the introduced libraries, such as the drawing part of Skia, removing some unnecessary logic in Harfbuzz, and customizing its own part for ICU. The implementation of Inline Text (including all dependent libraries) ultimately controls the increase in the size of the Cube package to around 170kb.

Experience and Application

Rich CSS styles and capabilities

After the Cube engine supports Inline Text, details such as style float, display:inline-block, and baseline alignment of multiple elements in flex layout have been improved, making the layout results almost consistent with those of the browser engine.

The following are some examples of Inline Text capabilities. The layout engine has recursive and nested features, which are also reflected in the examples:

• Parts of the text within a larger text block use different styles:

• float:left

• float:left "nesting doll"

• CJK text with word-break:keep-all , numerous storm breaks will not break:

• Font-family Fallback Mechanism

When we write the following code in CSS, it means:

 < style >
 div {

 font - family : alipay - number ; serif ;

 }

 </style> ​

 < body >

 < div > 123 China 456 < /div>

 </body> ​

Among them, 123 and 456 use alipay-number, but China uses the system's serif font. The reason is that alipay-number font does not provide the two Unicode glyphs of China.

Many non-Web rendering engines do not fully support this feature, which involves the selection rules and drawing of Typeface. Inline Text perfectly replicates the Fallback mechanism of the Web rendering engine.

• Font-face support

Third-party fonts are a common requirement. Many businesses cannot be satisfied with system fonts. Generally speaking, they can be built into the package or obtained through a URL. After the third-party font is ready, the browser's CSS re-triggers the matching rules through the CSS selector and re-selects the corresponding Typeface through FontSelector to complete the redrawing.

Unlike browsers, the style matching of the Cube engine is very streamlined. After downloading third-party fonts, the previous corresponding font-face text cache is cleared and the layout drawing is re-triggered to achieve the same effect. Thanks to the support of style sheets and the access of Inline Text to ICU, font-icon can be used directly using pseudo-elements (content) plus private Unicode, which is completely consistent with the Web experience.

Performance Improvements

The performance improvement of text layout is due to the use of text measurement and layout by the layout engine. The previous JNI interaction with the platform layer is gone, and the creation and layout logic of platform layer objects is gone. The performance of text layout has been greatly improved:

Application Scenario

Currently, 90% of the products generated by building the platform on Youku OTT have Inline Text enabled by default, using related capabilities to improve layout performance. Due to the requirements of the protocol page, developers no longer need to use Javascript to segment words and change colors, and can directly use the engine capabilities. You can refer to the following page. After applying Inline Text to this page, the page loading time is only 1/3 of the original:

Future and Outlook

The current Inline Text is implemented in version 1.0. The plan for version 2.0 is as follows:

• According to the current Cube drawing model, the support for textNest is enhanced (which can be understood as the nesting of span tags);
• Use a self-drawn backend (such as Skia) to achieve full platform consistency by directly rasterizing font information;
• Support vertical text layout;
• Support bidirectional text such as Arabic;
• Support more rich text features;
• Further optimize the calculation time of super long text layout.

Appendix: Inline Text supported styles

Legend: Supported Partially supported Not supported

​​