Alibaba.com's Thin Package is 40MB - The Industry's Most Comprehensive Summary of iOS Package Size Technology

Preface

Package size is an important indicator for measuring APP performance. It directly affects the user's download click rate (the package is too big and you don't want to download it), download and installation success rate (download slowly and don't use it), and APP uninstall rate (delete it first if it takes up too much space). The calculation logic of package size is very simple. It is the sum of the disk size occupied by various types of files. The technology of APP slimming is very complicated. The complexity of the code file and the compiler strategy determine the size of the executable file, and the business function and engineering architecture determine the complexity of the code file. For iOS APP slimming, the skills that need to be mastered include XCode build technology, LLVM compiler technology, CocoaPods build technology, image compression technology, and continuous integration technology. This article summarizes and refines the slimming technology and strategy of Alibaba.com App, and systematically introduces the business value, analysis technology, slimming technology, and anti-degradation mechanism of APP slimming, so that readers can systematically understand the technical system of APP slimming. Based on practical experience, this article also introduces the ROI of various slimming technologies, so that readers can avoid stepping on mines and wasting resources on technologies with poor results. I hope it will be helpful to you. 

Business Value

For every 6MB increase in package size, the app download conversion rate decreases by 1%.

At the 2019 Google Developer Conference, Google gave a very detailed data, every 6MB increase in package size, the app download conversion rate will drop by 1%. The conversion rates in different regions vary slightly. For every 10MB reduction in APK package size, the global average download conversion rate will increase by 1.75%, the download conversion rate of emerging countries such as India and Brazil will increase by more than 2.0%, and the download conversion rate of high-end markets such as the United States and Germany will increase by 1.5%.

Title of the above picture: APK reduced by 10MB, conversion rate increased in different countries.

Data source: Google Play internal data.

Detailed information: White Whale goes to sea: Highlights of the first day of the 2019 Google Developer Conference.

The above data survey and analysis report was before 2019 and is already somewhat lagging behind, so it is for reference only.

There are three possible reasons why package size affects download conversion rate:

• In a cellular network environment, users are reluctant to pay for data traffic. When the package size exceeds 200MB, the App Store will pop up a window to remind users that downloading may incur data traffic charges.
• The download time is too long and the user is unwilling to wait and cancels it.
• There was a network connection problem during the download.

Although Google Play does not provide data for different APP categories, based on the above three reasons, it is estimated that the impact of package sizes of different categories on download conversion rates is similar. The user population of the App Store is relatively high-end, and the data of the United States and Germany can be used as a reference.

20% of people uninstall apps due to limited storage space

CleverTap conducted a survey in 2021 in which they surveyed more than 2,000 mobile app users and asked them about the main reasons for uninstalling mobile apps. 20% of them uninstalled apps due to limited storage space.

The three main reasons are:

• They no longer use the app
• Limited storage space
• Too many ads.

Detailed material: clevertap: Why Users Uninstall Apps.

App Store Distribution and Download Restrictions

For iOS8-compatible apps, the text segment of the main binary file cannot exceed 60MB, otherwise it will not be submitted to the App Store. If the App Store download package exceeds 200MB, it cannot be downloaded and updated using cellular data.

Analytical techniques

The ultimate goal of APP slimming is to reduce the size of the App Store installation package and the download package size, but it is more convenient to compare the XCode build package size during the R&D phase, and the differences in caliber between them need to be clarified.

Result indicators: App Store installation package size and download package size

The viewing path is App Store Connect->TestFlight->Delivery version->Build version metadata->App Store file size.

Process Indicator: XCode Build Size

The size of the ipa package built by XCode and the installation size of the App Store are very different. The relationship between them can be obtained by simulating the process processed by Apple. The ipa package uploaded by the developer contains products of multiple architectures and multiple sets of image resources of different sizes. Apple will crop and redistribute them and convert them into ipa packages downloaded from the App Store.

The size of the iPad package of the built product: static library binary files (arm64, armv7), dynamic libraries (arm64, armv7), asset.car (@2x, @3x), and other resource files.

App Store installation size: static library binaries (single architecture), dynamic libraries (single architecture), asset.car (single size), other resource files.

Use the lipo tool to split a single architecture.

 lipo "originalExecutable" - thin arm64 - output "arm64Executable"

Use the assetutil tool to split assets

 xcrun --sdk iphoneos assetutil --scale 3 --output "$targetFolder/Assets3.car" "$sourceFolder/Assets.car"

Build package composition

Before learning how to cure diseases, we must first understand the composition of the human body. Similarly, we must first understand the iOS project structure and IPA package structure.

iOS project structure: iOS project consists of shell project and Pod module. There are two types of modules: static library and dynamic library. Shell project consists of main Target and Apple plug-in Target. The internal structure of the module includes source code files (OC, C, C++.h and.m), nib, bundle, xcassets, multi-language files, and various configuration files (plist, json). IPA package structure: iOS uploaded to App Store is an IPA package. After decompression, the IPA package is a folder, which consists of various types of files, mainly including MachO executable files, .framework (dynamic library), Assets.car, .appex (Apple plug-in), .strings (multi-language), .bundle, nib, json, png... From iOS project to IPA package: iOS project is built as IPA package. The core changes are compilation and file copying. The source code in the static library will be compiled into MachO executable files, and the xcassets folder will be converted into Assets.car. The others can be simply understood as file copies.

By analyzing the composition of the build package, you can determine where there is room for optimization. If the dynamic library accounts for too high a proportion, there may be a large amount of code that can be cut. It is recommended to divide it according to the sensitivity of resource size, such as: MachO executeable (including all static libraries), dynamic library, Apple Extension, assets.car, bundle, nib, audio, video.

Pod module size

APP slimming involves a large number of business modules and cannot be separated from the participation of the business team. Therefore, we need to analyze the size of each Pod module so that we can compare the package size ratio of each business team horizontally. The calculation principles of static libraries and dynamic libraries are different. For static libraries, first parse the linkmap data to calculate the code size of the Pod module, then parse the resource copy code of Pods-targetName-resource.sh to calculate the size of the resources copied to the Pod module. For dynamic libraries, first use lipo to split the binary files of the dynamic library, calculate the code size of a single architecture, and then calculate the resource files in the dynamic library framework to get the resource file size of the dynamic library.

Runtime Objc class coverage

If we know which classes have been used when the App is running, we can take down useless modules or code files. The Objc class coverage indicator can help us. When the APP is running, the number of classes loaded by a Pod module divided by the number of all classes can be called the Objc class coverage of this module. The core technology is to determine whether a class has been loaded. Here is a lightweight solution that has been verified online. The +initialize method is called when the ObjC class is used for the first time. After the class is loaded, cls->isInitialized will return True. The isInitialized method reads the flags in the data variable of metaClass and returns True if the 29th bit in flags is 1.

 // objc - class .mm
 Class class_initialize ( Class cls , id inst ) {
 if ( ! cls -> isInitialized ( ) ) {
 initializeNonMetaClass ( _class_getNonMetaClass ( cls , inst ) ) ;
 }
 return cls ;
 }


 // objc - runtime.h
 #define RW_INITIALIZED ( 1 << 29 )
 bool isInitialized ( ) {
 return getMeta ( ) -> data ( ) -> flags & RW_INITIALIZED ;
 }

Unused image resources

The technical principle is to first parse out all the resource files copied to the build product, and then parse out the resource files actually referenced in the code. The difference between the two is the useless resources. The first step is to obtain all resource files. In the Cocoapos project, the "Pods-targetName-resource.sh" script is responsible for copying the file resources in the Pod to the build product, including all file types bundle, xcassets, json, png. Parsing this script can get which image resources are copied by each Pod module.

 // Pods - targetName - resource .sh
 install_resource "${PODS_ROOT}/APodName/APodName.framework/APodName.bundle"
 install_resource "${PODS_ROOT}/BPodName/BPodName.framework/BPodName.xcassets"
 install_resource "${PODS_ROOT}/BPodName/BPodName.framework/xxx.png"

The second step is to get the resource files actually referenced in the code. The resource files referenced in the OC code are declared in the form of string literals and stored in the "__cstring" section of the Mach-O file after construction. By using strings to parse the binary file of the framework, you can get all the string declarations in the code, and then match "xxx", "[email protected]", "xxx.png", "xxx.bundle/xxx.png" based on various reference methods of the code.

 strings executable | grep 'xxx' > cstrings .txt

Classes not referenced at compile time

The product of iOS compilation is in Mach-o format. The __DATA __objc_classrefs segment in the file records the addresses of all referenced classes, and the __DATA __objc_classlist segment records the addresses of all classes. Differing the two can get the addresses of unreferenced classes. Then symbolize the addresses to get the information of unreferenced classes. Because Objc is a dynamic language, if a class is dynamically called using runtime, this situation cannot be scanned. (For example, Target Action and JS Core).

 otool - v - s __DATA __objc_classrefs xxxMainClient #Read the symbol with section __objc_classrefs in __DATA Segment
 otool - v - s __DATA __objc_classlist xxxMainClient #Read the symbol whose section is __objc_classlist in __DATA Segment
 nm - nm xxxMainClient

Open source tool to scan for unused classes.

Weight loss technology

Package size reduction can be achieved from a purely technical perspective or from a logical perspective.

From a purely technical perspective, there are two approaches. The first is to optimize the compilation logic. The second is to delete various other types of files (non-compilation products). Compilation optimization is non-invasive to business logic, with relatively low risks and costs, but the benefits are usually not high. Deleting files is more complicated. Deleting resource files has high benefits but high costs. Deleting source code files one by one has high risks and low benefits.

In comparison, the slimming effect will be more obvious from the perspective of logic. From the perspective of logic, the project is composed of many functional modules, which can be mainly divided into business functional modules (home page, search, cashier, order), basic functional modules (network library, image library, middleware, various third-party libraries...). Large projects usually consist of hundreds or even thousands of modules, small modules are tens of KB, and large modules are 1MB~10+MB. Functional modules have strong cohesion. When a functional module can be discarded or replaced, the overall risk and cost of offline are relatively low, and the ROI is very high. 

Slimming technology map

Based on project experience, I have sorted out the ROI of various optimization methods, which are introduced below.

Component Slimming

ROI of component slimming: Components with 0% class loading rate > useless functional components > duplicate functional components "Components with 0% class loading rate" is a complete Pod module, the module code and resources can be deleted as a whole, and the ROI is the highest. "Useless functional components" are usually a file directory within the module, which requires some coupling processing and resource ownership sorting, and the ROI is second. "Duplicate functional components" need to be properly refactored, with the highest migration cost and risk, and a relatively low ROI.

Components with 0% class loading rate

"Components with 0% class loading rate" means that no class has been loaded during operation. It is completely unused during operation and can be taken offline as a whole. When class loading rate statistics are generally collected, sampling is performed. Some low-frequency business components may be misreported. You need to confirm with the business owner before taking it offline. 

Useless functional components

"Useless functional components" are components that are no longer useful logically. They may still have coupling in the code, but the logic is no longer useful and can be taken offline through refactoring. Examples include business codes that have no effect in AB experiments, business codes from previous years' big promotions, old businesses left over after business revisions, and outdated third-party libraries.

Repeating functional components

For large-scale teams, different business lines may introduce "duplicate functional components", such as image selectors, cache libraries, and UI components. Duplicate functional components will bring unnecessary package size pressure and maintenance costs.

Resource reduction

Resource slimming ROI: large resources > lossy compression > duplicate resources > iconfont > multi-language text slimming > useless resources.

Great resources

Single-point optimization of large resources is very profitable, and resources larger than 100KB are analyzed first. For example, the audio file in our project is 900KB, and 700KB is removed after optimization.

Lossy Compression

When XCode is built, it will "compile asset catalog" and re-compress the images losslessly. Therefore, the effect of lossless compression using tools such as imageoptim is not obvious. There is no effect on compressing png images, but there is a certain effect on compressing jpg images. According to practical experience, lossy compression of icons does not affect the visual experience, and the compression rate can reach 70%~80%. For example, using the tinypng algorithm to compress a 425.9 KB png image, the compressed image is 79.8 KB, and the compression rate reaches 81%. There are many png compression tools in the industry, and we use tinypng, pngquant, pngcrush, optipng (lossless), and advpng. It is found that different tools have different compression effects for different images, so when compressing images, you can try each tool and then take the best one. It should be noted that compressed APNG images may become non-animated images. Try to avoid compressing APNG images and identify whether they are APNG images before compression.

 # Determine whether it is APNG format, APNG format is not compressed
 function isApng ( ) {
 ret = `grep "acTL" "$1" `
 lastWord = "${ret##* }"
 if [ [ $lastWord == "matches" ] ] ; then
 echo "YES"
 else
 echo "NO"
 fi
 }
 # Use various tools to compress png images
 #pngquant
 pngquant 256 - s5 --quality 70 --output "${tmpFileName}" -- "${tmpOrigName}"
 #pngcrush
 pngcrush - brute - rem alla - nofilecheck - bail - blacken - reduce - cc -- "$tmpOrigName" "$tmpFileName"
 #optipng
 optipng - o6 - i0 - out "$tmpFileName" -- "$tmpOrigName"
 #advpng
 cp "$tmpOrigName" "$tmpFileName"
 advpng - 4 - z -- "$tmpFileName"


 # Compare the compression effects one by one and keep the best compressed images
 sizeBefore = `stat - f % z "${tmpOrigName}" `
 sizeBefore = `expr $sizeBefore`
 sizeNow = `stat - f % z "${tmpFileName}" `
 sizeNow = `expr $sizeNow`


 if [ "$sizeNow" - lt "$sizeBefore" ] ; then
 # The size has been reduced
 echo "[advpng] $tmpOrigName"
 retImg = "$tmpFileName"
 fi
 # Record hash value
 img_sum = ` / usr / bin / shasum - a 256 "$retImg" | cut - d " " - f1`
 cache_file = "$cacheFolder/$img_sum.png"

Useless resources

Long-term business iteration will accumulate many useless resources. Through code static scanning, you can analyze the image resources that are not referenced. We have 12MB of useless resources in the project, of which 20 modules have useless resources exceeding 100KB.

Repeat Icon

Different business requirements will introduce duplicate resources, which will cause great waste in the long run. The solution is to calculate the hash value of the resource during the build, remove the same hash resources, and keep the mapping table of source file names and hash values. Hook the "imageNamed" method of resource loading at runtime and replace the resource name according to the mapping table.

ODR

iOS cannot update the Framework at runtime like Android. iOS's ODR technology (On Demand Resource) provides the ability to dynamically download resources at runtime. If resource files are not needed at startup, they can be processed through ODR.

Icon font

Iconfont supports scaling and color modification. It is small in size and suitable for icon scenes such as arrows and placeholders. Using iconfont can reduce the package size and improve the uniformity of the development visual experience. First, a complete set of iconfonts is developed based on the design specifications, the iconfont components are encapsulated, and an easy-to-use API is provided. Then, it is forbidden to add image resources for new business scenarios. After the team forms development habits, the existing images can be gradually replaced.

Multilingual copywriting

For international apps, multilingual copywriting is also a big part. Our app supports 20 languages, with more than 4,000 copywritings in each language, and the total copywriting size is 6MB, which was reduced by 2MB after slimming down.

Streamlined compilation product Oz: Optimization Level

There are multiple optimization levels. -Oz is about 10% smaller than -O3. After setting -Oz, XCode will optimize consecutive assembly instructions to reduce the binary size, but the side effect is that the execution speed will be slower. C++ projects are recommended to enable both.

 Main Project Release
 Optimization Level : - Oz


 Framework Project
 Optimization Level : - Oz

LTO (OC/C++)

LTO is to optimize cross-module calling code when LLVM is linked. According to everyone's experience, the optimization effect of different APP package sizes varies greatly, and specific testing is required. In addition, it also has some side effects, so it is recommended to only enable it in the Release package. LTO Introduction:

advantage:

• Inline some functions.
• Removed some unused code.
• It has a global optimization effect on the program.
• Reduce the compilation and linking speed. It is only recommended to enable it when building a formal package.
• Reduce link map readability (XX-lto.thin class appears) Main project settings.

 Main Project Release
 Link - Time Optimization set to Incremental


 Framework Project Release
 Link - Time Optimization set to Incremental

Dynamic library reuses main binary static library

C++ dynamic libraries often use some basic libraries such as openssl, libyuv, and libcurl, which are generally static libraries. If a dynamic library references a static library, all the symbols of the static library will be embedded by default when it is compiled. Although we can set the dynamic library to export only the static library symbols that need to be used, it is possible that multiple dynamic libraries use the same basic library, which will still cause redundancy of the basic library. For example, the size of openssl is 1MB. If the two dynamic libraries A and B depend on openssl, and the APP also references openssl, the final ipa package actually has 3 openssl, and 2MB of it is redundant. In this scenario, the best solution is to share the symbol table so that the dynamic library can call the basic library symbols of the main binary, so that the built-in static library can be removed. Just modify the Link configuration of XCode, no additional code development is required.

Dynamic library project:

• Set to dynamically search the app main binary symbol table when encountering undefined functions.
• Turn off bitcode.

 Other Linker Flags -> - undefined dynamic_lookup
 Enable Bitcode -> No

3 Export the external symbols that the dynamic library needs to call and write them to a file exported_symbols.

 nm - u xxx .framework / xxx > exported_symbols .txt

APP Engineering:

• The configuration needs to export all symbols in the exported_symbols file to avoid stripping out the symbols needed by the dynamic library during compilation.
• Turn off bitcode.

 // exported_symbols.txt is the path to the symbol file that needs to be exported
 EXPORTED_SYMBOLS_FILE -> exported_symbols .txt
 Enable Bitcode -> No

If a C++ dynamic library depends on an external static library, all symbols of the static library will be exported by default. In fact, the dynamic library only uses some of the symbols. You can set a whitelist of exported symbols to reduce the number of unused static library symbols exported.

Notes: 1. Both APP and framwork projects must turn off bitcode, otherwise they will not compile. 2. If multiple dynamic libraries use the same basic library, the exported symbol tables need to be combined. 3. When upgrading dynamic libraries, the symbol tables must be updated in time, and when upgrading basic libraries, the compatibility must be tested.

Linker product compression (black technology)

The product of iOS project construction is MachO file. The TEXT segment in MachO file stores various read-only data segments, the __cstring segment stores ordinary C string, and __objc_methtype and __objc_methname store Objc method signature and method name. For example, if @"Hello world" is declared in Objc code, a CFString will be generated at the bottom layer, which will be stored in __cstring after construction. These data take up a lot of space. Generally, the project will have at least 10MB, so the benefits of compression are considerable. After we went online, the size of the App Store installation package was optimized from 191MB to 174MB, a reduction of 16MB.

Technical principle:

When linking, the TEXT segment data is moved to the __DATA segment and compressed. When running, the decompression code is executed first, the decompressed TEXT segment data is stored in the custom segment, and the address of the string reference in the code is corrected to the decompressed custom segment.

Strip Linked Product

The Strip Linked Product setting will strip specific symbols. Do not set it to YES in the Debug environment, otherwise the symbols will not be visible during debugging.

 Main Project Release
 Deployment Postprocessing : YES
 Strip Linked Product : YES
 Strip Style : All Symbols (strip all symbol tables and redirection information)


 Framework Project
 Deployment Postprocessing : YES
 Strip Linked Product : YES
 Strip Style : Non - Global Symbols (strip non-global symbols including debug information, retain external symbols)

illustrate:

• Static libraries cannot set Strip Style to All Symbols, because a static library with all symbols stripped cannot be linked normally.
• Removing symbols does not affect the symbol information in the dSYM file. When viewing the crash log, you can find the corresponding symbols from the dSYM file.

Symbols Hidden by Default

Symbols Hidden by Default is used to set the default visibility of symbols. If it is set to YES, XCode will define all symbols as "private extern", and the package size will be slightly reduced. Set it to NO for dynamic libraries, otherwise there will be link errors.

 Main Project Release
 Symbols Hidden by Default: Yes

 Framework project static library / dynamic library
 Symbols Hidden by Default: NO

Remove unreferenced C/C++/Swift code: Dead Code Stripping

When Dead Code Stripping is turned on, unused code will be removed during linking. It is effective for static languages ​​​​C/C++/Swift, but not for dynamic languages ​​​​OC.

 Main Project
 Asset Catalog Compiler -> Optimization set to space

Asset Catalog Compiler

Optimization has three options: empty, time, and Space. Select Space to optimize the package size.

 Main Project
 Asset Catalog Compiler->Optimization set to space

C++ only exports necessary symbols: Symbol Visibility

Both C++ static and dynamic libraries export only necessary symbols. The default setting is to hide all symbols, and then use Visibility Attributes to individually control the symbols that need to be exported.

Hide all C++ symbols by default

 Other C ++ Flags -> Add -fvisibility = hidden

Set the symbols to be exported

 __attribute__ ( ( visibility ( "default" ) ) ) void MyFunction1 ( ) { }
 __attribute__ ( ( visibility ( "default" ) ) ) void MyFunction2 ( ) { }
 ....

Code offline

According to the results of the ObjC class coverage statistics, unused classes can be gradually offline. The code files are relatively small, and the offline code needs to be carefully confirmed to avoid functional problems or crashes, and the ROI is relatively low.

Flutter Project

Flutter is built separately, and the content introduced includes Flutter engine products and Flutter business code products. If an APP introduces Flutter, Flutter needs to be specially slimmed down.

Streamlined compilation product Oz: Optimization Level

Compared with Os, the -Oz option is estimated to have an 11% gain, but a 1% to 9% loss in first screen performance.

Dart symbol stripping and obfuscation

According to the official documentation, you can use the --dwarf-stack-trace option to remove Dart standard debugging symbols. With the --obfuscate option, you can replace longer symbols with shorter symbols, but the side effect is that the symbols will be obfuscated. Practical effect: In the Release environment, the size is reduced by 14% after the --dwarf-stack-trace and --obfuscate options are turned on.

Flutter icon tree shaking optimization

--tree-shake-icons practical effect: The size of Alibaba.com App is reduced by about 300KB after opening.

Remove NOTICES files

Practical effect: 700KB before compression, 80KB after compression.

Anti-degradation mechanism

Incremental standard and integrated bayonet

The technology of package size slimming is like various weight loss exercises, such as running, dancing, and fat burning yoga. However, if you want to achieve the goal of weight loss, exercise alone is not enough. You must also control your calorie intake. Only those who control their diet can finally lose weight successfully. Therefore, we also need "additional calorie specifications" (package size increment specifications) and supervisors (integration checkpoints). Add a checkpoint plug-in to the continuous integration, analyze the linkemap file of the build package to get the module size, and then compare it with the baseline data. If the package size increment standard is violated, integration is prohibited.

Packet size increment specification (reference):

• Baseline data: Based on a specific version, the baseline data of each module is calculated using the "calculate module size" method mentioned above.
• Existing modules: If the module increment exceeds the baseline data by 100KB, integration is prohibited. Set up a compensation mechanism. If the existing modules can be slimmed down, the size of the newly added modules can be offset. Special approval can be applied for special cases. Adding the approval process is to inspire everyone to reflect. Is it valuable to add so much? Are there any unnecessary resources brought in?
• New module: New business functions need to go through an approval process to evaluate whether the new business value and package size increase are reasonable.

Compare business health horizontally

When the package size has been optimized to the extreme from a technical perspective, if you want to further reduce it, you can only explore from the perspective of business value. If a module has a large disk size but few users use it, it can be considered that its value to the business is small. Therefore, we can define a technical indicator to measure the business contribution of the module unit size. Based on the floor area ratio indicator, we can compare various businesses horizontally and require businesses with low floor area ratios to reduce package size or remove less important business functions.

Floor area ratio = Business PV / Business size.

Summarize

Let's summarize the implementation path of package size reduction. The first step is to set goals and track the result indicators such as APP download conversion rate (App Store Connect Analytics), APP installation package size, XCode build package size, etc. The second step is to build an analysis system, including Pod module size analysis, Objc class coverage analysis, useless image resource analysis, etc. The third step is to use various slimming technologies according to ROI, component slimming > resource slimming > compilation optimization > code offline. The fourth step is to build an anti-degradation mechanism, including incremental standards, integrated card slot capabilities, and health analysis.

References:

White Whale goes to sea: Highlights of the first day of the 2019 Google Developer Conference: New changes in Google Play: http://www.baijingapp.com/article/24808

Open source tool for scanning unused classes: https://github.com/xuezhulian/classunref

googleplaydev：Shrinking APKs, growing installs

https://medium.com/googleplaydev/shrinking-apks-growing-installs-5d3fcba23ce2

clevertap:Why Users Uninstall Apps

https://clevertap.com/blog/uninstall-apps/

Pngcrush: https://pmt.sourceforge.io/pngcrush/

pngquant：https://pngquant.org/

OptiPNG: http://optipng.sourceforge.net/

advpng: http://www.advancemame.it/doc-advpng.html

LTO Introduction: https://zhuanlan.zhihu.com/p/384160632