Detailed explanation of JSPatch implementation principle: Let JS call/replace any OC method

JSPatch is a small-sized JavaScript library that allows JS to call/replace any OC method, allowing iOS apps to be hot-updated. In the process of implementing JSPatch, I encountered many difficulties and pitfalls, some of which are worth sharing. This article introduces the implementation principle of the entire JSPatch from the three aspects of basic principles, method calling, and method replacement, and records the ideas and pitfalls encountered during the implementation as much as possible.

Basic principles

The most fundamental reason why we can call and rewrite OC methods through JS is that Objective-C is a dynamic language. All method calls/class generation on OC are performed at runtime through Objective-C Runtime. We can get the corresponding class and method through class name/method name reflection:

 Class class = NSClassFromString( "UIViewController" );
 id viewController = [[ class alloc] init];
 SEL selector = NSSelectorFromString( "viewDidLoad" );
 [viewController performSelector:selector];

You can also replace a class method with a new implementation:

 static   void newViewDidLoad(id slf, SEL sel) {}
 class_replaceMethod( class , selector, newViewDidLoad, @ "" );

You can also register a new class and add methods to the class:

 Class cls = objc_allocateClassPair(superCls, "JPObject" , 0 );
 objc_registerClassPair(cls);
 class_addMethod(cls, selector, implement, typedesc);

There are many articles that explain the Objective-C object model and the principle of dynamic message sending in detail, such as this one, so I won't elaborate on it here. In theory, you can call any OC method by class name/method name at runtime, replace the implementation of any class, and add any class. So the principle of JSPatch is: JS passes a string to OC, and OC calls and replaces OC methods through the Runtime interface. This is the most basic principle. There are still many things to do in the actual implementation process. Let's see how it is implemented.

Method Call

 require( 'UIView' )
 var view = UIView.alloc().init()
 view.setBackgroundColor(require( 'UIColor' ).grayColor())
 view.setAlpha( 0.5 )

After introducing JSPatch, you can use the above JS code to create a UIView instance and set the background color and transparency. It covers five aspects: require class introduction, JS call interface, message passing, object holding and conversion, and parameter conversion. Let's take a look at the specific implementation one by one.

1.require

After calling require('UIView'), you can directly use the UIView variable to call the corresponding class method. What require does is very simple. It creates a variable with the same name in the JS global scope. The variable points to an object. The object attribute __isCls indicates that this is a Class, and __clsName saves the class name. These two attributes will be used when calling the method.

 var _require = function(clsName) {
 if (!global[clsName]) {
 global[clsName] = {
 __isCls: 1 ,
 __clsName: clsName
 }
 }
 return global[clsName]
 }

So after calling require('UIView'), the UIView variable is generated in the global scope, pointing to an object like this:

 {
 __isCls: 1 ,
 __clsName: "UIView"  
 }

2.JS interface

Next, let’s look at how UIView.alloc() is called.

Old implementation

Regarding the implementation of this call, my initial thought was that according to the characteristics of JS, the only way to make the call of UIView.alloc() correct is to add the alloc method to the UIView object, otherwise the call will not be successful. JS will only throw an exception immediately when calling undefined properties/variables, unlike OC/Lua/ruby which has a forwarding mechanism. So I did a complicated thing, that is, when require generates a class object, pass the class name to OC, OC finds all the methods of this class through the Runtime method and returns them to JS. The JS class object generates a function for each method name, and the function content is to use the method name to call the corresponding method in OC. The generated UIView object is roughly like this:

 {
 __isCls: 1 ,
 __clsName: "UIView" ,
 alloc: function() {…},
 beginAnimations_context: function() {…},
 setAnimationsEnabled: function(){…},
 ...
 }

In fact, not only do we need to traverse all methods of the current class, but we also need to loop to find methods of the parent class until the top level. All methods on the entire inheritance chain must be added to the JS object. A class has hundreds of methods. Adding all methods to the JS object in this way will cause a serious problem. When a few classes are introduced, the memory will explode and become unusable. Later, in order to optimize the memory problem, we also set up an inheritance relationship in JS. We do not add all methods on the inheritance chain to a JS object, and avoid repeatedly adding hundreds of methods of the base class NSObject to each JS object. Each method only exists in one copy. The JS object copies the inheritance relationship of the OC object, and searches up the inheritance chain when looking for methods. As a result, the memory consumption is slightly smaller, but it is still too large to be acceptable.

New Implementation

At that time, I continued to look for a solution. If I followed the JS syntax, this was the only way, but what if I didn't follow the JS syntax? Suddenly, my mind was opened. CoffieScript/JSX can use JS to implement an interpreter to implement its own syntax. I can also do it in a similar way. Then I thought further that the effect I wanted was actually very simple, that is, when calling a non-existent method, it can be forwarded to a specified function for execution, which can solve all problems. This can actually be done with a simple string replacement to replace all method calls in the JS script. The best solution is to use regular expressions to change all method calls to calls to the __c() function before OC executes the JS script, and then execute the JS script, so as to achieve a message forwarding mechanism similar to OC/Lua/Ruby, etc.:

 UIView.alloc().init()
 ->
 UIView.__c( 'alloc' )().__c( 'init' )()

Add a __c member to the prototype of the JS object base class Object, so that all objects can call __c and perform different operations based on the current object type:

 Object.prototype.__c = function(methodName) {
 if (! this .__obj && ! this .__clsName) return   this [methodName].bind( this );
 var self = this  
 return function(){
 var args = Array.prototype.slice.call(arguments)
 return _methodFunc(self.__obj, self.__clsName, methodName, args, self.__isSuper)
 }
 }

_methodFunc() passes the relevant information to OC, OC uses the Runtime interface to call the corresponding method, returns the result value, and the call is completed.

This way, there is no need to traverse the object methods in OC, and there is no need to save these methods in JS objects. Memory consumption drops by 99%. This step is the most satisfying part of this project. A very simple method is used to solve a serious problem, replacing the previous complex and poor implementation.

3. Messaging

Now that we have solved the JS interface problem, let's see how JS and OC communicate with each other. Here we use the JavaScriptCore interface. When starting the JSPatch engine, OC will create a JSContext instance. JSContext is the execution environment of JS code. You can add methods to JSContext, and JS can directly call this method:

 JSContext *context = [[JSContext alloc] init];
 context[@ "hello" ] = ^(NSString *msg) {
 NSLog(@ "hello %@" , msg);
 };
[_context evaluateScript:@"hello('word')"]; //output hello word

JS passes data to OC by calling the method defined by JSContext, and OC passes it back to JS through the return value. When calling this method, its parameters/return values ​​will be automatically converted by JavaScriptCore, and NSArray, NSDictionary, NSString, NSNumber, NSBlock in OC will be converted to array/object/string/number/function type on the JS side respectively. The above _methodFunc() method passes the class name and method name to be called to OC in this way.

4. Object holding/conversion

After passing the above message, UIView.alloc() will execute [UIView alloc] in OC and return a UIView instance object to JS. How is this OC instance object represented in JS? How can JS directly call its instance method (UIView.alloc().init()) after getting this instance object?

For a custom id object, JavaScriptCore will pass the pointer of this custom object to JS. This object cannot be used in JS, but OC can find this object when it is passed back to OC. As for the management of this object life cycle, according to my understanding, if JS has a variable reference, the reference count of this OC object will increase by 1, and it will decrease by 1 when the reference of the JS variable is released. If there is no other holder on OC, the life cycle of this OC object follows JS and will be released when JS performs garbage collection.

The variable passed back to JS is the pointer of this OC object. Without any processing, it is impossible to call instance methods through this variable. Therefore, when returning the object, JSPatch will encapsulate the object.

First, tell JS that this is an OC object:

 static NSDictionary *toJSObj(id obj)
 {
 if (!obj) return nil;
 return @{@ "__isObj" : @(YES), @ "cls" : NSStringFromClass([obj class ]), @ "obj" : obj};
 }

__isObj is used to indicate that this is an OC object, and the object pointer is also returned. Then, on the JS side, this object will be converted into a JSClass instance:

 var JSClass
 var _getJSClass = function(className) {
 if (!JSClass) {
 JSClass = function(obj, className, isSuper) {
 this .__obj = obj
 this .__isSuper = isSuper
 this .__clsName = className
 }
 }
 return JSClass
 }
 var _toJSObj = function(meta) {
 var JSClass = _getJSClass()
 return new JSClass(meta["obj"], meta["cls"])
 }

If the JS side finds that the return is an OC object, it will pass it to _toJSObj() to generate a JSClass instance, which saves the OC object pointer, class name, etc. This instance is the JS object corresponding to the OC object in JSPatch, and the life cycle is the same.

Back to the JS interface we mentioned in the second point, this JSClass instance object also has a __c function. When calling the method of this object, it also goes to the __c function. The __c function will return the OC object pointer in the JSClass instance object and the method name and parameters to be called to OC, so that OC can call the instance method of this object.

Next, let's see how the object is passed back to OC. In the above example, view.setBackgroundColor(require('UIColor').grayColor()), a UIColor instance object is generated here and passed back to OC as a parameter. As mentioned above, the representation of this UIColor instance in JS is a JSClass instance, so it cannot be directly passed back to OC. The parameter here will actually be processed in the __c function, and the object's .__obj original pointer will be passed back to OC.

***One point, OC objects may exist in containers such as NSDictionary / NSArray, so it is necessary to traverse the container to pick out OC objects for formatting. OC needs to replace all objects with the format recognized by JS. JS needs to convert objects into JSClass instances. When JS instances are passed back to OC, they need to be converted into OC object pointers. Therefore, when OC flows out, it will be processed by the formatOCObj() method. When JS gets data from OC, it will be processed by _formatOCToJS(). When JS passes parameters to OC, it will be processed by _formatJSToOC(). The figure is as follows:

5. Type conversion

After JS passes the class name/method name/object to be called to OC, OC calls the corresponding method of the class/object through NSInvocation. To successfully call the method and obtain the return value, two things need to be done:

1. Get the parameter types of the OC method to be called, and convert the object sent by JS into the required type for calling.

2. Take out the return value according to the return value type, package it into an object and pass it back to JS.

For example, in the example view.setAlpha(0.5) at the beginning, JS passes an NSNumber to OC. OC needs to know from the NSMethodSignature of the OC method to be called that the parameter here is a float type value, so it converts the NSNumber to a float value and then uses it as a parameter to call the OC method. Here, we mainly deal with numeric types such as int/float/bool, and perform special conversion processing on types such as CGRect/CGRange. The rest are implementation details.

Method Replacement

JSPatch can use defineClass interface to replace any method of a class. The implementation process of method replacement is also quite tortuous. At first, the va_list method was used to obtain parameters, but it was found that it was not available under arm64, so we had to use another hack method to implement it. In addition, it took some effort to add methods to the class, implement property, and support self/super keywords. The following is an explanation of each one.

Basic principles

In OC, each class is a structure like this:

 struct objc_class {
 struct objc_class * isa;
 const   char *name;
 ….
 struct objc_method_list **methodLists; /*method list*/  
 };

The methodList method list stores the Method type:

 typedef struct objc_method *Method;
 typedef struct objc_ method {
 SEL method_name;
 char *method_types;
 IMP method_imp;
 };

Method stores all the information of a method, including the SEL method name, type parameters and return value type, and IMP the function pointer that specifically implements the method.

When a method is called through a Selector, the corresponding Method is found in the methodList list for calling. The elements on this methodList can be replaced dynamically. You can replace the function pointer IMP corresponding to a certain Selector with a new one, or you can get the function pointer IMP corresponding to an existing Selector and let another Selector correspond to it. Runtime provides some interfaces to do these things. Take replacing the -viewDidLoad: method of UIViewController as an example:

 static   void viewDidLoadIMP (id slf, SEL sel) {
 JSValue *jsFunction = ...;
 [jsFunction callWithArguments:nil];
 } 
 
 Class cls = NSClassFromString(@ "UIViewController" );
 SEL selector = @selector (viewDidLoad);
 Method method = class_getInstanceMethod(cls, selector); 
 
 //Get the function pointer of the viewDidLoad method  
 IMP imp = method_getImplementation(method) 
 
 //Get the parameter type of the viewDidLoad method  
 char *typeDescription = ( char *)method_getTypeEncoding(method); 
 
 //Add a new ORIGViewDidLoad method to point to the original viewDidLoad implementation  
 class_addMethod(cls, @selector (ORIGViewDidLoad), imp, typeDescription); 
 
 //Point viewDidLoad IMP to the custom new implementation  
 class_replaceMethod(cls, selector, viewDidLoadIMP, typeDescription);

In this way, the -viewDidLoad method of UIViewController is replaced with our custom method. The viewDidLoad method called by UIViewController in APP will go to the above viewDidLoadIMP function. In this new IMP function, the method passed in by JS is called to replace the -viewDidLoad method with the implementation in JS code. At the same time, a new method -ORIGViewDidLoad is added to UIViewController to point to the original viewDidLoad IMP. JS can call the original implementation through this method.

Method replacement is implemented very simply, but the premise of this simplicity is that this method has no parameters. If this method has parameters, how to pass the parameter value to our new IMP function? For example, in the -viewDidAppear: method of UIViewController, the caller will pass a Bool value. We need to get this value in the IMP we implement (viewDidLoadIMP mentioned above). How can we get it? If you only write an IMP for a method, you can directly get the parameter value:

 static   void viewDidAppear (id slf, SEL sel, BOOL animated) {
 [function callWithArguments:@(animated)];
 }

But what we want is to implement a universal IMP. Any method and any parameter can be transferred through this IMP to get all the parameters of the method and call back the JS implementation.

va_list implementation (32-bit)

Initially I implemented it using variable parameter va_list:

 static   void commonIMP(id slf, ...)
 va_list args;
 va_start(args, slf);
 NSMutableArray *list = [[NSMutableArray alloc] init];
 NSMethodSignature *methodSignature = [cls instanceMethodSignatureForSelector:selector];
 NSUInteger numberOfArguments = methodSignature.numberOfArguments;
 id obj;
 for (NSUInteger i = 2 ; i < numberOfArguments; i++) {
 const   char *argumentType = [methodSignature getArgumentTypeAtIndex:i];
 switch (argumentType[ 0 ]) {
 case   'i' :
 obj = @(va_arg(args, int ));
 break ;
 case   'B' :
 obj = @(va_arg(args, BOOL));
 break ;
 case   'f' :
 case   'd' :
 obj = @(va_arg(args, double ));
 break ;
 ... //Other numerical types  
 default : {
 obj = va_arg(args, id);
 break ;
 }
 }
 [list addObject:obj];
 }
 va_end(args);
 [function callWithArguments:list];
 }

In this way, no matter what the method parameters are or how many there are, they can be taken out one by one through a set of methods of va_list, and form NSArray to be passed back when calling the JS method. This solved the parameter problem very well, and it worked normally until I ran it on an arm64 machine and it crashed as soon as it was called. After checking the information, I found that the structure of va_list under arm64 has changed, which makes it impossible to take parameters in the above way. See this article for details.

ForwardInvocation implementation (64-bit)

Later, I found another very hacky method to solve the problem of parameter acquisition, which utilized the OC message forwarding mechanism.

When calling a method that does not exist on an NSObject, an exception will not be thrown immediately. Instead, the method will be forwarded through multiple layers, calling the object's -resolveInstanceMethod:, -forwardingTargetForSelector:, -methodSignatureForSelector:, -forwardInvocation: and other methods layer by layer. This article explains it clearly. *** -forwardInvocation: will have an NSInvocation object. This NSInvocation object stores all the information about the method call, including the Selector name, parameters and return value type. The most important thing is all the parameter values. All the parameter values ​​of the call can be obtained from this NSInvocation object. We can find a way to make each method call that needs to be replaced by JS be transferred to -forwardInvocation:, which can solve the problem of not being able to get the parameter values.

For the specific implementation, let's take replacing the -viewWillAppear: method of UIViewController as an example:

Point the -viewWillAppear: method of UIViewController to a non-existent IMP through the class_replaceMethod() interface: class_getMethodImplementation(cls, @selector(__JPNONImplementSelector)), so that when this method is called, it will go to -forwardInvocation:.

Add two methods, -ORIGviewWillAppear: and -_JPviewWillAppear:, to UIViewController. The former points to the original IMP implementation, and the latter is the new implementation, in which the JS function will be called back later.

Rewrite the -forwardInvocation: method of UIViewController to a custom implementation. Once the -viewWillAppear: method of UIViewController is called in OC, the call will be forwarded to -forwardInvocation: after the above processing. At this time, an NSInvocation has been assembled, including the parameters of this call. Here, the parameters are decompressed from NSInvocation, and the newly added method -JPviewWillAppear: is called with the parameters. In this new method, the parameters are obtained and passed to JS, and the JS implementation function is called. The entire calling process is over, and the entire process is illustrated as follows:

***One question, we replaced the implementation of the -forwardInvocation: method of UIViewController. If this method is really used in the program to forward messages, what should we do with the original logic? First, we will create a new method -ORIGforwardInvocation: before replacing the -forwardInvocation: method, save the original implementation IMP, and make a judgment in the new -forwardInvocation: implementation. If the forwarding method is what we want to rewrite, we will follow our logic. If not, we will call -ORIGforwardInvocation: and follow the original process.

There is another pitfall in the implementation process. When getting the parameter value from the NSInvocation object in -forwardInvocation:, if the parameter value is of type id, we will get it like this:

 id arg;
 [invocation getArgument:&arg atIndex:i];

But sometimes this will cause an inexplicable crash, and the crash is not in this place. It seems that the pointer here is wrong, which leads to subsequent memory disorder and crashes in various places. I searched for this bug for a long time before locating it here. I still don't know why. Later, I solved it in this way:

 void *arg;
 [invocation getArgument:&arg atIndex:i];
 id a = (__bridge id)arg;

The rest are implementation details, such as the need to generate different IMPs based on different return value types, and to handle parameter conversions in various places.

New methods

When JSPatch was first open sourced, it did not support adding new methods to a class, because it was considered sufficient to replace native methods. New methods were simply added to JS objects and only ran on the JS side. In addition, OC needed to know the types of each parameter and return value when adding a method to a class, and it was necessary to define a way in JS to pass these types to OC in order to complete the addition of the method, which was quite troublesome. Later, many people paid attention to this issue. The inability to add new methods made the action-target mode unusable, and I began to wonder if there was a better way to add methods. At first, I thought that since all the newly added methods were used by JS, it would be better to unify the return values ​​and parameters of the newly added methods into id types, so that the type would not need to be passed, but the number of parameters still needed to be known. Later, when chatting with Lancy, I found a solution. JS can obtain the number of function parameters, and it is directly encapsulated and the number of parameters can be passed to OC.

Now the method defined by defineClass will be wrapped by JS and turned into an array containing the number of parameters and method entity and passed to OC. OC will determine if the method already exists and perform the replacement operation. If it does not exist, it will call class_addMethod() to add a new method, generate a new Method based on the number of parameters and method entity passed in, and set the parameters and return value type of the Method to id.

There is a problem here. If a class implements a protocol, there are optional methods in the protocol method, and its parameters are not all of type id, such as a method of UITableViewDataSource:

 - (NSInteger)tableView:(UITableView *)tableView sectionForSectionIndexTitle:(NSString *)title atIndex:(NSInteger)index;

If the original class does not implement this method, but implements it in JS, it will go to the logic of the newly added method, and each parameter type will become id, which does not match the protocol method, resulting in an error. The protocol problem will be handled later. If the newly added method is a method implemented by the protocol, the NSMethodSignature of this method will be taken to obtain the correct parameter type for addition.

Property Implementation

 defineClass( 'JPTableViewController : UITableViewController' , {
 dataSource: function() {
 var data = self.getProp( 'data' )
 if (data) return data;
 data = [ 1 , 2 , 3 ]
 self.setProp_forKey(data, 'data' )
 return data;
 }
 }

JSPatch can add member variables to objects through the two methods -getProp: and -setProp:forKey:. The implementation uses the runtime association interface objc_getAssociatedObject() and objc_setAssociatedObject() to simulate, which is equivalent to associating an object with the current object self. Later, the object can be found through the current object self, which has the same effect as a member, except that it must be an id object type.

Originally, OC has class_addIvar() which can add members to a class, but it must be added before the class is registered. It cannot be added after the registration is completed. This means that members can be added to classes newly added in JS, but not to classes already existing in OC, so it can only be simulated using the above method.

The self keyword

 defineClass( "JPViewController: UIViewController" , {
 viewDidLoad: function() {
 var view = self.view()
 ...
 },
 }

JSPatch supports using the self keyword directly in the instance method in defineClass. Just like OC, self refers to the current object. How is this self keyword implemented? In fact, this self is a global variable. The instance method is packaged in defineClass. Before calling the instance method, the global variable self is set to the current object. After the call, it is set back to null. Then, the self variable can be used in the process of executing the instance method. This is a little trick.

super keyword

 defineClass( "JPViewController: UIViewController" , {
 viewDidLoad: function() {
 self.super.viewDidLoad ()
 },
 }

In OC, super is a keyword and cannot be obtained through dynamic methods. So how is super implemented in JSPatch? In fact, when calling the super method, OC calls a method of the parent class and passes the current object as self to the parent class method. We just need to simulate this process.

First, the JS side needs to tell OC that it wants to call the super method of the current object. The way to do this is to call self.super, which will return a new JSClass instance. This instance also saves the reference to the OC object and is marked with __isSuper=1.

 JSClass = function(obj, className, isSuper) {
 this .__obj = obj
 this .__isSuper = isSuper
 this .__clsName = className
 }
 JSClass.prototype.__defineGetter__( 'super' , function(){
 if (! this .__super) {
 this .__super = new JSClass( this .__obj, this .__clsName, 1 )
 }
 return   this .__super
 })

When calling a method, __isSuper will be passed to OC to tell OC to call the super method. What OC does is, if it is calling the super method, it will find the IMP implementation of the superClass method, and add a method to the current class pointing to the super IMP implementation. Then calling the new method of this class is equivalent to calling the super method. Replace the method to be called with this new method, and the super method is called.

 static id callSelector(NSString *className, NSString *selectorName, NSArray *arguments, id instance, BOOL isSuper) {
 ...
 if (isSuper) {
 NSString *superSelectorName = [NSString stringWithFormat:@ "SUPER_%@" , selectorName];
 SEL superSelector = NSSelectorFromString(superSelectorName); 
 
 Class superCls = [cls superclass];
 Method superMethod = class_getInstanceMethod(superCls, selector);
 IMP superIMP = method_getImplementation(superMethod); 
 
 class_addMethod(cls, superSelector, superIMP, method_getTypeEncoding(superMethod));
 selector = superSelector;
 }
 ...
 }

Summarize

The entire JSPatch implementation principle has been roughly described. The remaining small points, such as the GCD interface, block implementation, method name underline processing, etc., will not be described in detail. You can directly look at the code. JSPatch is still under continuous improvement and hopes to become the best solution for dynamic updates on the iOS platform. Welcome everyone to build this project together. The github address is: https://github.com/bang590/JSPatc