How to hide your hot update bundle files?

[[379259]]

This article is reprinted from the WeChat public account "Ludan Lab", the author is Halogenated Hydrocarbons. Please contact the Ludan Lab public account for reprinting this article.

A while ago, a senior executive of our company got some gossip from some channels that a national-level APP was rejected for three months because Apple App Store reviewers detected React Native hot update content. Our hot update platform is similar to the APP in question, so there is also a risk of rejection. So we have to think of some ways to hide the hot update bundle so that it will not be discovered by the reviewers.

In fact, this problem is quite complicated, because it is not just a technical problem, but also involves various complex commercial interests. Under many restrictions, it is difficult to find an optimal solution. This problem is also quite sensitive. I can only talk about my ideas in general, and the specific code implementation will not be provided in this article.

Solemnly declare: If someone hides the hot update data according to the ideas in this article and causes the application to be rejected or removed from the shelves, I will not be responsible

1. Commercial interests

Apple has very strict control over the iPhone ecosystem: Apps must be put on the App Store, dynamic link libraries must be signed, and virtual machines with JIT functions cannot be used...

Regarding hot update technology, Apple banned the hot update framework JSPatch[1] in 2017, causing many apps to be rejected. The reasons given by Apple are mainly:

• The hot update code is not encrypted and verified, and may be cracked and hijacked by a third party
• JSPatch has too high permissions and may call private APIs to change the original APP functions
• For Apple officials, JSPatch has too much freedom and will bypass the App Store, the only traffic distribution platform on iOS, to update apps, affecting commercial interests.

As the saying goes, cutting off someone's financial resources is like killing their parents. No one can tolerate such things involving commercial interests. Moreover, many applications are not WeChat, which has a huge user base and can negotiate with Apple officially (the WeChat mini-program ecosystem was negotiated, but the mini-program payment permissions were not negotiated), so this issue is still very complicated.

In fact, Apple officials have always been neither in favor of nor against dynamic hot updates. Compared with JSPatch, React Native and game hot updates are still allowed, mainly in the following three aspects:

• Online games, which are heavy operations, still need hot updates to maintain the popularity of activities. There are new activities every week. It is unreasonable to ask users to actively go to the App Store to download the update package. The same is true for App activity operations.
• React Native/Lua and other hot update technologies are dynamic in a container, unlike JSPatch which has such broad modification permissions.
• Apple has reached a delicate balance between commercial interests and those of game makers and Internet giants.

To be honest, Apple's review process has always been confusing. Review rejection is sometimes like practicing Tai Chi, and the given specifications are interpreted differently. However, to be on the safe side, we still have to study the relevant platform specifications.

II. Interpretation of the Standards

In 2015, Apple issued an agreement, the Apple Developer Program License Agreement[2]. Section 3.3.2 of the agreement contains a paragraph about hot updates:

Except as set forth in the next paragraph, an Application may not download or install executable code. Interpreted code may only be used in an Application if all scripts, code and interpreters are packaged in the Application and not downloaded. advertised purpose of the Application as submitted to the App Store.

This paragraph roughly means that except for Webkit and JavascriptCore, which can dynamically execute the scripts and files sent, all other scripts/codes/interpreters must be packaged inside the APP. This sentence actually leaves a loophole for React Native: React Native uses JavascriptCore to execute JS script files, so dynamic sending is also reasonable.

Interpreted code may be downloaded to an Application but only so long as such code: (a) does not change the primary purpose of the Application by providing features or functionality that are inconsistent with the intended and advertised purpose of the Application as submitted to the App Store, (b) does not create a store or storefront for other code or applications, and (c) does not bypass signing, sandbox, or other security features of the OS.

This paragraph roughly means that I allow you to hot update, but you must follow my three rules:

• You cannot make major changes to the app’s functionality, which would cause the app’s actual functionality to be inconsistent with the APP Store’s promotion (this is where the rules are very strict, and the judging criteria are entirely based on the reviewer’s mood)
• Unable to dynamically create an app store (which probably means that IAP payment cannot be bypassed, otherwise how can Apple collect tax)
• Cannot bypass the security features of signatures/sandboxes/OS (this is understandable, to maintain system and ecosystem security)

Interpreted this way, it seems that the problem can be solved by just following the rules and being a good citizen. But to be honest, dynamic rules are more of a gentleman's agreement. If both parties are honest, then everyone will be happy. If someone jumps out and wants to break the rules, to be honest, everyone will be embarrassed. In the future, hot update technology will definitely exist in a delicate balance.

3. Technical Implementation

Whenever I design a project, I always try to find a theoretical answer first. Take the example of the hidden hot update bundle. We mainly want to find a breakthrough in information transmission. In fact, Shannon proposed the Shannon-Weaver Communication Model in 1949 [3]. This model divides communication into five parts: information source, transmitter, channel, receiver, information receiver, and noise.

Shannon-Weaver communication model

Then, combined with this communication model, the answer to hiding/encrypting communication information is obvious:

• Encryption of the source: Encryption when the information is sent by the sending and receiving terminal, and decryption when receiving the message
• Encrypt the channel: When the information is transmitted in the channel, it is encrypted when passing through the channel.

So let's expand and explore these two major directions below.

1. Encrypt/obfuscate the message itself

1.1 Steganography - the modern Trojan horse

Steganography is a very, very old technology. The key to this technology is to hide/disguise the data you want to transmit so that a third party cannot see the real data you want to transmit.

There are many examples of steganography, such as the Trojan Horse, which looks like a wooden horse from the outside, but when it is transported to the city, the soldiers will run out; in some movies and TV series we watch, there are also similar plots: the protagonist receives a blank letter, and when it is burned over a candle, the text appears. In today's digital age, we certainly don't use blank letter paper to secretly transmit messages. We must have some more cyber methods, such as image seeding technology - writing messages in image files.

If you have been on Tieba for a while, you will definitely be familiar with the image seeding technology. Some great masters will post a thread and hide the seed file in a picture. You download the picture, change the suffix of .jpg to .zip or .rar, and then decompress the file to get the hidden seed file, and then leave a reputation as "the good poster" in Tieba.

So what is the principle of this technology? In fact, it is very simple. It simply merges a jpg file and a rar file together, but the image viewer will ignore the attached rar file data. In this way, it looks like a picture, but from a binary point of view, there are some data hidden in this picture file.

Next, let’s look at the principle of the seed file.

First, I used an image editor to generate a 2x2 4 pixel image - RGBY.jpg. I matched the colors with the Google logo:

RGBY-image

Then we use a binary viewing tool (I use Hex Fiend software here) to view the encoding of this image. Because the image has only 4 pixels, the binary data will be relatively small. Pay attention to the binary data of this file. It starts with FF D8 and ends with FF D9.

When the image viewer loads an image file, it will do a check. If it starts with FF D8, it will think it is a jpg image, and then it will enter the branch of jpg image decoding. After loading binary data and encountering FF D9, it will think that the image has been loaded and will not care about the subsequent data.

RGBY-Binary-Code

Based on the fact that the image previewer will not load data after FF D9, we can append some hidden data to the end of the jpg file.

For the convenience of testing, I created a text.txt file with the content of hello word, and then used the cat command to merge RGBY.jpg and text.txt to generate an RGBY_text.jpg file:

 cat RGBY.jpg text.txt > RGBY_text.jpg

At this time, when you use the image browser to view the file, you can see that the file is still previewed normally:

RGBY_text-image

But if you use a binary viewer to view this image, you will find that there are 11 extra bytes at the end, which are exactly the content in text.txt - hello word:

RGBY_text-Binary-Code

In this way, we have achieved the purpose of steganography.

Don't think this solution is low. In fact, some of Alibaba's keys are written into a picture using a similar principle (of course, it is not as simple as the above case). When we transmit hot update bundle files, we can write the bundle files into a picture. In this way, when the auditors monitor the traffic, they will see a picture when they capture the packet. If they do not check the binary encoding of the picture, they will not find the data hidden in it.

For this solution, the changes on both the server and the client are relatively small. The server only needs to merge an image file before sending a bundle each time, and the client only needs to remove the redundant image data after reading the steganographic image.

Of course, there are many kinds of steganography, such as LSB-based image steganography, which writes data in the extended data field of jpg, png, and mp4. Since the principles are similar, I will not introduce them here. Students who are interested can search and learn on their own.

1.2 Symmetric Encryption

Symmetric encryption is also an encryption technology with a long history. It has also developed rapidly with the support of information technology. Let me give you an example of the simplest symmetric encryption algorithm - XOR encryption.

I think every programmer is familiar with the XOR operation. Let's agree that 0 is false and 1 is true. Then the truth table of the XOR operation is as follows:

The following operation rules can be easily derived from the truth table:

 // Encryption:
 // Original text Key Secret text
 01010111 ^ 11110011 = 10100100 
 
 // Decryption:
 // Secret text key original text
 10100100 ^ 11110011 = 01010111

As we all know, bit operations are very fast. If you want to simply obfuscate the bundle, directly using XOR encryption will basically not affect the performance.

Although the XOR operation is simple, there is a first rule in cryptography: never implement an encryption algorithm yourself. We can use very mature symmetric encryption algorithms (such as AES and DES) to encrypt the bundle: high performance, good security, and most importantly, the open source community has ready-made libraries, which can be directly swapped.

Therefore, if a symmetric encryption solution is used, as long as the server and the client agree on a key, and then the server encrypts the bundle with the key and the client decrypts it with the same key, the App Store's abnormal traffic detection can be bypassed to a certain extent.

1.3 Asymmetric encryption

Asymmetric encryption belongs to the content of modern cryptography. It is very new, but also very reliable. The specific principles are too complicated to explain it clearly in a few words, so I will not introduce it.

In the scenario of encrypting hot update bundles, the effect is actually similar to symmetric encryption, except that private key encryption and public key decryption are used instead.

1.4 Summary

Generally speaking, bundle encryption does not use only one technology. For example, we may use hybrid encryption to encrypt the bundle itself, use message authentication codes (such as HMAC) to prevent tampering, add timestamp random numbers to prevent replay, and finally steganographically write the encrypted data... There are so many combinations here. I personally think that you can refer to some classic encryption combinations for business practice.

2. Encrypt the channel

Channel encryption is also relatively straightforward in the context of this article, which is to use the HTTPS protocol to prevent auditors from capturing our hot update traffic by capturing packets. Of course, HTTPS also has many interesting knowledge points, which I will briefly introduce below.

2.1 Use HTTPS

It's 2021 now, and I think there is basically no exposed HTTP plaintext traffic on the Internet... In the past few years, some companies may still consider the server costs brought by HTTPS encryption, but under the requirements of major platforms (iOS/Android/Chrome), except for a few unmaintained websites, basically all sites are on HTTPS. After all, the value of data is now much higher than the electricity bill of the server. After HTTPS, at least the probability of being hijacked by a man-in-the-middle attack will be reduced a lot.

Does HTTPS mean you can rest easy? Of course not. Last year, I wrote an article about Charles packet capture, which spent a lot of space to introduce HTTPS packet capture. Since an APP developer can use the tools on the market to capture packets, auditors can do it even better. With the packet capture tools, most HTTPS data can be captured and hijacked. Let's talk about some of the more advanced content in the HTTPS protocol.

2.2 HTTPS Certificate Pinning

HTTPS certificate pinning, also known as HTTPS certificate locking, means that we only accept certificates for specified domain names in the APP, and do not accept any certificates corresponding to the CA root certificates built into the operating system or browser.

Through this authorization method, we can ensure the uniqueness and security of the communication between the APP and the server. If the packet capture software is turned on and the fixed certificate is not actively imported, the packet capture cannot be effective (for the specific principle, please refer to my blog post: Charles packet capture principle). I think the auditors do not have the energy to destroy your APP to obtain your certificate, so you can hide your hot update bundle in this way.

Of course, certificate pinning also has a certain price. Certificates issued by CAs have validity periods, so the disadvantage is that the certificate needs to be re-embedded into the APP after the certificate is renewed.

2.3 HTTPS two-way authentication

When we usually use HTTPS, we usually only do one-way authentication, that is, the client authenticates the authenticity of the server. In fact, HTTPS supports two-way authentication, that is, the server authenticates the authenticity of the client (see the * part of the figure below for the specific process).

TLS 1.2 handshake flow chart

Generally speaking, apps that enable HTTPS two-way authentication are those that require extremely high security, such as financial apps and anonymous social apps. In addition, if you want to implement two-way authentication, you must build a public key certificate and private key into the client, but there is a risk of the app being damaged, so you have to find a way to encrypt and hide these two things (it's like a Russian doll).

Overall, the cost of implementing HTTPS two-way authentication is still very high, but once it is implemented, the security factor is still very high. It not only bypasses the traffic detection of auditors, but also greatly protects the network security of the entire APP.

IV. Conclusion

Regarding hot updates, according to Apple's application specifications, hot updates based on JavaScriptCore are completely feasible, but the premise is that you must abide by the rules and cannot get out of Apple's control; however, the App Store's review rules are extremely opaque. Although we are good citizens, we still have to hide the hot update bundle to a certain extent to avoid unnecessary trouble; we can think about hiding the hot update bundle from two perspectives: source encryption and channel encryption. Overall, it is to flexibly use cryptographic knowledge to encrypt network data to prevent abnormal traffic from being detected. While hiding the bundle, it also protects user data security and reduces the possibility of being attacked.

5. Reference reading 🍶 Why does your Charles fail to capture packets?