Solve the iOS memory crash problem caused by Flutter

background

If your Flutter version number is less than or equal to 2.5.3 or greater than or equal to 3.0.5, the problem described below will not occur in your application, but I believe that most applications will hit this range.

This happened recently. The crash data of the newly launched iOS version of our app (Gao Ding Design) has soared. According to crash logs and user feedback, most of the new crashes come from the same reason: insufficient memory. Some directly become OOM, which is difficult to troubleshoot. Others fail to apply for memory, resulting in subsequent logical errors.

Considering the situation of "blooming everywhere and exploding at many points", it should be some kind of low-level memory management problem. This is a bit puzzling, because this version did not make any memory-related changes. So I took a dichotomy and spent two hours trying all the PRs in the version, and found that the culprit was the Flutter version upgrade: 2.5.3 → 2.10.

So the question becomes: what changes did Flutter make in 2.5.3 → 2.10. that caused the memory crash problem?

Analyze the problem

Based on user feedback, we found an operation path that will inevitably cause a memory crash, so I tried to test the memory situation in Flutter 2.5.3 and 2.10.5:

By comparing the memory situation, we can draw a conclusion: before the upgrade, the memory tolerance was higher, and there was no problem with a peak of 1.2G; after the upgrade, the memory tolerance was lower, and it would crash at a peak of 1.1G.

This reminds me of "compressed memory": when the memory is tight, the iOS system will compress some unused memory to free up memory space. When you need to read this compressed memory, you also need to decompress it before reading it.

Why does this mechanism, which sounds good, go wrong? Here is a classic example:

• SDWebImage ^[1] is a third-party image cache library commonly used in iOS development. It caches used images in memory for quick reuse later, and releases the cache when memory is tight. One detail is that SDWebImage used to store the cache in NSMutableDictionary, which caused some image caches to be compressed by the system after a period of non-use. When the memory peak comes, the system will send a memory warning, and SDWebImage will choose to release the cache when it receives the warning. Remember? Before releasing, you must decompress it. At the moment of decompression, the memory peak is pushed higher, so the system kills the process, creating a classic OOM. Later, SDWebImage used the NSCache provided by the system for caching. NSCache has been specially optimized for memory compression to solve this problem.

So, I followed the clues and searched for a few keywords in Flutter’s issues: iOS compress memory. The first post [2] confirmed my guess:

Several key points are mentioned in the article:

1. After version 2.5.3, memory crashes started to occur more frequently.
2. After version 2.5.3, Flutter did change its memory strategy and adopted a compressed memory approach (called compressed pointers in the post).
3. Some people experimentally turned off compressed memory and solved this problem.

Considering that we upgraded from 2.5.3 to 2.10.5, we can basically pinpoint the issue as compressed memory.

Two options

There are currently two solutions:

• Solution 1: Wait for Flutter to officially resolve the issue, and then we can upgrade the version.
• Solution 2: We modify the Flutter Engine source code ourselves and turn off memory compression.

The cost of modifying the Flutter Engine source code is actually very high. You need to understand the dependencies between Flutter Engine and Flutter, the construction method, the Flutter Engine code logic, and so on.

We originally wanted to wait for solution 1, but as more and more backend user feedback became available, solving the memory-related crash issue became urgent, so we decided to adopt solution 2.

Are customer complaints the primary productive force?

So I, who had never written a line of Flutter code, just took the plunge. After reading countless official/private documents, I spent three days and finally figured it out. I added custom printing to the Flutter Engine:

How the specific solution 2 solves the problem is described in detail below.

Coincidentally, just as we were solving the problem with Solution 2, Solution 1 also saw the light of day: Flutter urgently released version 3.0.5, in which Flutter Engine disabled memory compression. So we immediately upgraded and tried it, and it really didn’t crash. We made some adjustments and went online. According to online data feedback, the memory crash problem has been perfectly solved.

Flutter Engine customization and source code debugging

Next, we will introduce the operation process of Solution 2 in detail. First, let’s take a look at the flowchart:

Download source code

I checked the official Flutter documentation ^[3] and found that there is only one page of documentation when downloading the source code. You can imagine how deep this pit is.

Fork the Flutter Engine repository

Open github.com/flutter/eng…[4] and fork a copy to your own repository. For example, mine is github.com/JPlay/engin…[5].

The purpose of fork is to have a place to store the modified code after modifying the source code. (Here, fork is enough, no need to clone)

Install depot_tools

depot_tools[6] is a tool set provided by Google to manage project code. It contains many packages. Here are the ones we will use:

• gclient - Source code management tool that can help you pull project source code and dependencies.
• gn - creates compilation materials, especially suitable for cross-platform and multi-compilation target projects such as Flutter.
• ninja - compilation tool, responsible for compiling the compilation materials generated by gn.

Start installing depot_tools:

 $ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git ​

After the installation is complete, add the following command to ~/.bashrc or ~/.zshrc to set depot_tools as an environment variable for subsequent use:

 export PATH = / path / to / depot_tools : $ PATH

Pull source code

Unlike usual when we use git to pull directly, we must use gclient here because there are many dependencies that can only be pulled down by gclient.

We first create a new folder called engine (the name is arbitrary), and the subsequent source code will be placed here. Create a new configuration file in engine, the name must be .gclient, and use a text editor to add the following content:

 solutions = [
 {
 "managed" : False ,
 "name" : "src/flutter" ,
 "url" : "[email protected]:JPlay/engine.git@57d3bac3dd5cb5b0e464ab70e7bc8a0d8cf083ab" ,
 "custom_deps" : { } ,
 "deps_file" : "DEPS" ,
 "safesync_url" : "" ,
 } ,
 ]

Here is the value of url:

• [email protected] ^[7] :JPlay/engine.git is the git repository I just forked.
• 57d3bac3dd5cb5b0e464ab70e7bc8a0d8cf083ab is the commit id corresponding to our current Flutter version 2.10.5.

You can find it in /bin/internal/engine.version in your Flutter directory, for example, mine is:

 $ cat / Users / JPlay / development / flutter / bin / internal / engine.version

After completing the configuration, attach the agent and you can pull the code in the engine folder:

 $ gclient sync - - verbose

It must be emphasized here that the code here exceeds 10GB and the process is quite slow. If there are any errors or jams during the process, it is basically a network problem. It is recommended to carefully check the logs. Most of them are failures in cloning a repository or accessing an address. It is recommended to use git clone or curl to try whether the network is smooth.

PS: My first agent was able to pull most of the code, but a small part of the code could not be pulled down, which wasted most of my time. Later, I changed an agent and it was successfully pulled down.

After success, you will find that all the code is concentrated in the engine/src directory, similar to this:

If you want to switch the engine branch later, you can first enter /src/flutter and then execute:

 $ git reset - - hard < commit id >
 $ gclient sync - - with_branch_heads - - with_tags

Compile

Next is compilation, which we will do in two steps:

1. Use gn to create compilation material.
2. Compilation is performed using ninja.

For a brief introduction to gn and ninja, see here ^[8] , for a detailed introduction to gn, see here, for a detailed introduction to ninja, see here ^[9]

It is worth mentioning that since Flutter's compiled products are platform-specific, we currently mainly need iOS and Android, which can be done on macOS.

When compiling iOS/Android products, you also need to compile a host product. This is because we need to compile a Dark SDK corresponding to the current version.

Because the code version, target platform, and target architecture are not unique, the iOS arm64 target is used as an example. Please imitate other situations as appropriate.

Creating Compiled Materials

gn provides a bunch of parameters to help us create compilation materials:

 usage : gn [ - h ] [ - - unoptimized ] [ - - enable - unittests ]
 [ - - runtime - mode { debug , profile , release , jit_release } ] [ - - interpreter ]
 [ --dart - debug ] [ --no - dart - version - git - info ] [ --full - dart - debug ] ​ ​ ​ ​ ​ ​
 [ - - target - os { android , ios , mac , linux , fuchsia , win , winuwp } ] [ - - android ]
 [ - - android - cpu { arm , x64 , x86 , arm64 } ] [ - - ios ] [ - - ios - cpu { arm , arm64 } ]
 [ - - mac ] [ - - mac - cpu { x64 , arm64 } ] [ - - simulator ] [ - - linux ] [ - - fuchsia ]
 [ - - winuwp ] [ - - linux - cpu { x64 , x86 , arm64 , arm } ]
 [ - - fuchsia - cpu { x64 , arm64 } ] [ - - windows - cpu { x64 , arm64 } ]
 [ - - simulator - cpu { x64 , arm64 } ] [ - - arm - float - abi { hard , soft , softfp } ]
 [ --goma ] [ --no - goma ] [ --xcode - symlinks ] [ --no - xcode - symlinks ] ​ ​ ​ ​ ​ ​ ​ ​
 [ - - depot - tools DEPOT_TOOLS ] [ - - lto ] [ - - no - lto ] [ - - clang ]
 [ - - no - clang ] [ - - clang - static - analyzer ] [ - - no - clang - static - analyzer ]
 [ - - target - sysroot TARGET_SYSROOT ]
 [ - - target - toolchain TARGET_TOOLCHAIN ​​]
 [ - - target - triple TARGET_TRIPLE ]
 [ - - operator - new - alignment OPERATOR_NEW_ALIGNMENT ]
 [ - - macos - enable - metal ] [ - - enable - vulkan ] [ - - enable - fontconfig ]
 [ - - enable - vulkan - validation - layers ] [ - - enable - skshaper ]
 [ - - no - enable - skshaper ] [ - - always - use - skshaper ]
 [ - - embedder - for - target ] [ - - coverage ] [ - - out - dir OUT_DIR ]
 [ --full - dart - sdk ] [ --no - full - dart - sdk ] [ --ide IDE ] ​ ​ ​ ​ ​ ​
 [ - - disable - desktop - embeddings ] [ - - build - glfw - shell ]
 [ - - no - build - glfw - shell ] [ - - build - embedder - examples ]
 [ - - no - build - embedder - examples ] [ - - bitcode ] [ - - stripped ]
 [ - - no - stripped ] [ - - prebuilt - dart - sdk ] [ - - no - prebuilt - dart - sdk ]
 [ - - fuchsia - target - api - level FUCHSIA_TARGET_API_LEVEL ]
 [ - - use - mallinfo2 ] [ - - asan ] [ - - lsan ] [ - - msan ] [ - - tsan ] [ - - ubsan ]
 [ - - trace - gn ] [ - - verbose ]

Here we will use a few:

For detailed instructions, enter: /path/to/gn --help.

We create a compilation material for iOS debugging in the src/ directory:

 $ ./flutter/tools/gn--runtime-mode=debug--unoptimized ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​
 $ ./flutter/tools/gn--ios--runtime-mode=debug--unoptimized ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

The first line generates host materials, and the second line generates iOS materials (no input architecture, the default is arm64). So two new folders are added under src/out/ , which are the compilation materials:

Execute Compilation

Now that the materials are ready, we are going to start compiling. If you are using a Mac with Intel CPU (x64 architecture), everything will go smoothly. Just execute the command:

 $ ninja - C out / ios_debug_unopt & & ninja - C out / host_debug_unopt

However, if you have an M series Mac (arm64 architecture), you need to do some work (I guess everyone is ¬_¬):

1. Modify /src/flutter/sky/tools/create_macos_gen_snapshots.py.

2. Modify /src/flutter/lib/snapshot/BUILD.gn.

3. Modify /src/third_party/dart/runtime/BUILD.gn.

The above modifications are all to solve the problem that "the build script considers the compiled host machine to be x64 architecture by default", and the modifications we made are to adapt to the arm64 architecture.

Since the compilation script is frequently updated, the above modification scheme may only be effective for the current commit, but I have summarized some experience to facilitate everyone to modify the script:

1. Pay attention to your host and target systems and architectures. Generally, the points that need to be modified revolve around these parameters.
2. gen_snapshot is a Dart compilation product. Make sure it is placed in the correct folder and called correctly.
3. Make good use of the debugging printing method. You can use print to print parameters in any .gn or .py file that needs to be modified. If you are not familiar with it, you can quickly preview the syntax of gn ^[11] and Python ^[12] (I did this).
4. Read the error message carefully, it is very detailed and you can follow the clues to solve the problem.
5. The best way is to find a x64 Mac.

This modification plan is my personal temporary solution. There are also some other ideas from other experts in the issue, which can be referenced: github.com/flutter/fluff ^[13]

Modify source code

If everything goes well, we have passed the compilation stage. Now we can modify the source code. I will give an example here just to prove that we have successfully modified the source code:

Add a print message to the Run method of /src/flutter/shell/common/engine.cc . This will cause the engine to print this message when it starts.

Don't forget our original intention: turn off iOS memory compression in /src/flutter/tools/gn to solve memory problems:

After the modification, recompile: (this is an incremental update, very fast):

 $ ninja - C out / ios_debug_unopt & & ninja - C out / host_debug_unopt

Next, enter a Flutter project directory and execute:

 $ flutter run - - local - engine - src - path = / path / to / engine / src / - - local - engine = ios_debug_unopt

You can see the console output:

The application runs successfully and outputs our customized information. So far, we have achieved a phased success and have successfully run our modified code in the Flutter project.

Source code debugging

The Flutter official documentation ^[14] provides a very complete description of debugging. I will only give an example of Xcode source code debugging here.

We open a Flutter project, for example, Runner.xcworkspace, since we just ran:

 $ flutter run - - local - engine - src - path = / path / to / engine / src / - - local - engine = ios_debug_unopt

Therefore, the Generated.xcconfig file has already been set with the relevant parameters (if not, set them yourself):

Then drag /src/out/ios_debug_unopt/flutter_engine.xcodeproj to the Runner project:

Find a place where the program will run and set a breakpoint, such as the init method in FlutterAppDelegate.mm. Run the project:

The breakpoint is successful, and then you can debug happily.

Summarize

This troubleshooting is really like a case-solving process, finding clues bit by bit based on clues, and finally solving the problem. Although there were many pitfalls in the process, I still felt a sense of satisfaction in reasoning and solving the case all the way to the end. I would like to share this with you, hoping that it can help you solve the same memory problem.

References

[1]SDWebImage: https://github.com/SDWebImage/SDWebImage.

[2] First post: https://github.com/flutter/flutter/issues/105183.

[3] Flutter official documentation: https://github.com/flutter/flutter/wiki/Setting-up-the-Engine-development-environment.

[4]github.com/flutter/engine: https://github.com/flutter/engine.

[5]github.com/JPlay/engine: https://github.com/JPlay/engine.

[6]depot_tools: http://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html#_setting_up.

[7][email protected]: mailto:[email protected].

[8]gn and ninja: https://zhuanlan.zhihu.com/p/136954435.

[9]https://ninja-build.org/: https://ninja-build.org/.

[10]https://github.com/flutter/flutter/wiki/Flutter%27s-modes: https://link.juejin.cn?target=https%3A%2F%2Fgithub.com%2Fflutter%2Fflutter%2Fwiki%2FFlutter%2527s-modes.

[11]gn: https://chromium.googlesource.com/chromium/src/tools/gn/+/48062805e19b4697c5fbd926dc649c78b6aaa138/docs/language.md#GN-Language-and-Operation.

[12]Python: https://www.runoob.com/python/att-string-format.html.

[13] flutter/issues/96745: https://github.com/flutter/flutter/issues/96745.

[14]Official documentation: https://github.com/flutter/flutter/wiki/Debugging-the-engine.