HandyJSON: Swift language JSON to Model tool library

background

JSON is a commonly used application layer data exchange protocol for mobile development. The most common scenario is that the client initiates a network request to the server, the server returns a JSON text, and then the client parses the JSON text and displays the corresponding data on the page.

But when programming, processing JSON is a hassle. Without introducing any wheels, we usually need to convert JSON to Dictionary first, and then remember the key corresponding to each data, and use this key to retrieve the corresponding value in the Dictionary for use. In this process, we will make various mistakes:

• Key is misspelled;
• The path is written wrong;
• The type is wrong;
• I was confused because I didn’t get the value;
• One day, a field agreed upon with the server was changed, and all places where it was used were not updated;
• ...

In order to solve these problems, many open source libraries for processing JSON have emerged. In Swift, these open source libraries mainly work in two directions:

• Keep JSON semantics and parse JSON directly, but make the calling method more elegant and safer through encapsulation;
• Predefine Model classes, deserialize JSON into class instances, and then use these instances;

For 1, the most widely used and highly rated library is SwiftyJSON, which represents the core of this direction. It is still based on the JSON structure to get the value, which is easy and clear to use. But because of this, this approach cannot properly solve the above problems, because the key, path, and type still need to be specified by the developer;

For option 2, I personally think this is a more reasonable approach. Due to the existence of the Model class, the parsing and use of JSON are subject to the constraints of the definition. As long as the client and the server agree on the Model class, after the client defines it, when using the data in the business, you can enjoy the benefits of syntax checking, property preview, property completion, etc., and once the data definition changes, the compiler will force all used places to change before the compilation passes, which is very safe. In this direction, the work done by open source libraries is mainly to deserialize JSON text into the Model class. This type of JSON library includes ObjectMapper, JSONNeverDie, HandyJSON, etc. HandyJSON is the most comfortable library to use. This article will introduce the use of HandyJSON to convert between Model and JSON.

Project address: https://github.com/alibaba/handyjson

Why use HandyJSON

Before HandyJSON appeared, there were two main ways to deserialize JSON to Model class in Swift:

• Let the Model class inherit from NSObject, then use the class_copyPropertyList() method to get the property name as the key, get the value from JSON, and then assign the class property value through the KVC mechanism supported by the Objective-C runtime; such as JSONNeverDie;
• Supports pure Swift classes, but requires developers to implement Mapping functions and use overloaded operators for assignment, such as ObjectMapper;

Both of them have obvious disadvantages. The former requires Model to inherit from NSObject, which is very inelegant and directly negates the way of defining Model with struct; the latter's Mapping function requires developers to customize it and specify the JSON field name corresponding to each attribute, which is highly invasive and still prone to spelling errors and maintenance difficulties.

HandyJSON takes a different approach and uses Swift reflection + memory assignment to construct Model instances, avoiding the problems encountered by the above two solutions.

Convert JSON to Model

Simple Types

If a Model class wants to support deserialization through HandyJSON, it only needs to implement the HandyJSON protocol when defining it. This protocol only requires the implementation of an empty init() function.

For example, we have agreed with the server on an Animal data with name/id/num fields, so we define the Animal class like this:

 class Animal: HandyJSON {
 var name : String?
 var id: String?
 var num: Int ? 
 
 required init() {}
 }

Then suppose we get the following JSON text from the server:

 let jsonString = "{\"name\":\"cat\",\"id\":\"12345\",\"num\":180}"  

After introducing HandyJSON, we can do deserialization like this:

 if let animal = JSONDeserializer<Animal>.deserializeFrom(json: jsonString) {
 print(animal. name )
 print(animal.id)
 print(animal.num)
 }

Simple, huh?

Support Struct

If Model is defined as a struct, since Swift provides a default constructor for struct, there is no need to implement an empty init() function. However, please note that if you specify another constructor for strcut, you need to keep an empty implementation.

 struct Animal: HandyJSON {
 var name : String?
 var id: String?
 var num: Int ?
 } 
 
 let jsonString = "{\"name\":\"cat\",\"id\":\"12345\",\"num\":180}"   
 
 if let animal = JSONDeserializer<Animal>.deserializeFrom(json: jsonString) {
 print(animal)
 }

More complex types

HandyJSON supports various forms of basic properties in class definitions, including optional (?), implicitly unwrapped optional (!), arrays, dictionaries, Objective-C basic types (NSString, NSNumber), various types of nesting ([Int]?, [String]?, [Int]!, ...), etc. For example, the following type looks more complicated:

 class Cat: HandyJSON {
 var id: Int64!
 var name : String!
 var friend: [String]?
 var weight: Double ?
 var alive: Bool = true  
 var color: NSString? 
 
 required init() {}
 }

Just as easy to convert:

 let jsonString = "{\"id\":1234567,\"name\":\"Kitty\",\"friend\":[\"Tom\",\"Jack\",\"Lily\",\"Black\"],\"weight\":15.34,\"alive\":false,\"color\":\"white\"}"   
 
 if let cat = JSONDeserializer<Cat>.deserializeFrom(json: jsonString) {
 print(cat.xxx)
 }

Nested Model Class

If an attribute in the Model class is another custom Model class, then as long as that Model class also implements the HandyJSON protocol, it can be converted:

 class Component: HandyJSON {
 var aInt: Int ?
 var aString: String? 
 
 required init() {}
 } 
 
 class Composition: HandyJSON {
 var aInt: Int ?
 var comp1: Component?
 var comp2: Component? 
 
 required init() {}
 } 
 
 let jsonString = "{\"num\":12345,\"comp1\":{\"aInt\":1,\"aString\":\"aaaaa\"},\"comp2\":{\"aInt\":2,\"aString\":\"bbbbb\"}}"   
 
 if let composition = JSONDeserializer<Composition>.deserializeFrom(json: jsonString) {
 print(composition)
 }

Specify a node in the deserialized JSON

Sometimes the JSON text returned to us by the server contains a lot of status information that has nothing to do with the Model, such as statusCode, debugMessage, etc., or the useful data is below a certain node, then we can specify which node to deserialize:

 class Cat: HandyJSON {
 var id: Int64!
 var name : String! 
 
 required init() {}
 } 
 
 
 // The server returns this JSON, we only want to parse the cat in data
 let jsonString = "{\"code\":200,\"msg\":\"success\",\"data\":{\"cat\":{\"id\":12345,\"name\":\"Kitty\"}}}"   
 
 // Then, we specify to parse "data.cat" and express the path through dots
 if let cat = JSONDeserializer<Cat>.deserializeFrom(json: jsonString, designatedPath: "data.cat" ) {
 print(cat. name )
 }

Model class with inheritance relationship

If a Model class inherits from another Model class, you only need to make the parent Model class implement the HandyJSON protocol:

 class Animal: HandyJSON {
 var id: Int ?
 var color: String? 
 
 required init() {}
 } 
 
 
 class Cat: Animal {
 var name : String? 
 
 required init() {}
 } 
 
 let jsonString = "{\"id\":12345,\"color\":\"black\",\"name\":\"cat\"}"   
 
 if let cat = JSONDeserializer<Cat>.deserializeFrom(json: jsonString) {
 print(cat)
 }

Custom parsing method

HandyJSON also provides an extension capability, which allows you to define the parsing key and parsing method of a field in the Model class. We often have such requirements:

• In a certain model, we don't want to use the key agreed on by the server as the attribute name, we want to set one ourselves;
• Some types such as enum and tuple cannot be parsed directly from JSON, but we have such attributes in the Model class;

The HandyJSON protocol provides an optional mapping() function, in which we can specify the key of a field or the method to parse its value from JSON. For example, we have a Model class and a JSON string returned by the server:

 class Cat: HandyJSON {
 var id: Int64!
 var name : String!
    var parent: (String, String)? 
 
 required init() {}
 } 
 
 let jsonString = "{\"cat_id\":12345,\"name\":\"Kitty\",\"parent\":\"Tom/Lily\"}"  

As you can see, the id attribute of the Cat class does not correspond to the Key in the JSON text; and as for the parent attribute, it is a tuple and cannot be parsed from "Tom/Lily" in JSON. So we need to define a Mapping function to support these two things:

 class Cat: HandyJSON {
 var id: Int64!
 var name : String!
    var parent: (String, String)? 
 
 required init() {} 
 
    func mapping(mapper: HelpingMapper) {
 //Specify the id field to be parsed using "cat_id"
        mapper.specify(property: &id, name : "cat_id" ) 
 
 //Specify the parent field to be parsed using this method
        mapper.specify(property: &parent) { (rawString) -> (String, String) in  
            let parentNames = rawString.characters.split{$0 == "/" }.map(String.init)
 return (parentNames[0], parentNames[1])
 }
 }
 }

In this way, HandyJSON helps us convert JSON to Model class perfectly. It is so convenient that this is the reason why it is named Handy.

Convert Model to JSON text

HandyJSON also provides the ability to serialize Model classes into JSON text, which is simply ruthless.

Basic Types

If you only need to serialize, you don't need to make any special changes when defining the Model class. For any class instance, you can directly call HandyJSON's serialization method to serialize and get a JSON string.

 class Animal {
 var name : String?
 var height: Int ? 
 
    init( name : String, height: Int ) {
 self.name = name  
 self.height = height
 }
 } 
 
 let cat = Animal( name : "cat" , height: 30) 
 
 //Serialize to simple JSON text
 if let jsonStr = JSONSerializer.serialize(model: cat).toJSON() {
    print( "simple json string: " , jsonStr)
 } 
 
 //Serialize to formatted JSON text
 if let prettifyJSON = JSONSerializer.serialize(model: cat).toPrettifyJSON() {
    print( "prettify json string: " , prettifyJSON)
 } 
 
 //Serialize to a simple dictionary
 if let dict = JSONSerializer.serialize(model: cat).toSimpleDictionary() {
    print( "dictionary: " , dict)
 }

Complex Types

Even if there are other Model classes in the Model class, they are all supported.

 enum Gender {
 case Male
 case Female
 } 
 
 struct Subject {
 var id: Int64?
 var name : String? 
 
    init(id: Int64, name : String) {
 self.id = id
 self.name = name  
 }
 } 
 
 class Student {
 var name : String?
 var gender: Gender?
 var subjects: [Subject]?
 } 
 
 let student = Student()
 student.name = "Jack"  
 student.gender = .Female
 student.subjects = [Subject(id: 1, name : "math" ), Subject(id: 2, name : "English" ), Subject(id: 3, name : "Philosophy" )] 
 
 if let jsonStr = JSONSerializer.serialize(model: student).toJSON() {
    print( "simple json string: " , jsonStr)
 }
 if let prettifyJSON = JSONSerializer.serialize(model: student).toPrettifyJSON() {
    print( "prettify json string: " , prettifyJSON)
 }
 if let dict = JSONSerializer.serialize(model: student).toSimpleDictionary() {
    print( "dictionary: " , dict)
 }

Summarize

With the support of HandyJSON, we can now happily use JSON in Swift. This library supports Swift 2.2+, Swift 3.0+. If you have any requirements or suggestions, you can go to https://github.com/alibaba/handyjson to raise an issue.

Why HandyJSON was developed?

My iOS team switched to Swift last November. Our server and client data exchange format has always been JSON, and at that time, the only well-known library for processing JSON in Swift seemed to be SwiftyJSON. After the project switched to Swift, we also used this library. After using it, the requirements were met, but for some complex models, the code looked very bad, because each value needs to be in the form of json["akey"]["bkey"]["ckey"].value. When writing, I didn't think there was any problem with the document, but later, when it was separated from the document, the whole article was full of keys expressed in strings, and it was difficult to feel the model structure from the code. So we would write a section of sample data in the comments. But it is still quite messy, and it is also difficult to debug if the key is written incorrectly. Sometimes it takes half a day to debug a case problem.

So we evolved a bit, writing the Model first, then writing the convert function in the Model class, and also using KVC to traverse the key assignment. It was much more comfortable to write, but still troublesome, and required every class to inherit from NSObject. Not long after, we got to know the ObjectMapper library, and without hesitation, we switched to it. The world suddenly became much cleaner.

But it still feels a little bit off, because ObjectMapper needs to specify the mapping relationship by itself. Usually the key in JSON and the field name in Model are the same, and I have to write a bunch of extra things every time, which always feels redundant, and it is also difficult to change the field. New colleagues who have just come into contact with Swift also expressed that they are not very comfortable, because the JSON deserialization library they used before, whether in Java or Objective-C, naturally uses the Model field name to get the value.

So I thought about studying whether this effect can be achieved in Swift.

Design ideas of HandyJSON

Limitations in Swift

Whether it is a JSON deserialization library in Java or Objective-C, it usually obtains the field name set of the Model at runtime, traverses the set, takes the key to get the value in JSON and completes the assignment. These steps can be achieved by Java relying on the reflection mechanism, and Objective-C can also easily achieve them through the class_copyPropertyList method plus the KVC mechanism. However, Swift will get stuck at the last step: unable to assign a value.

Swift's reflection is read-only, that is, we can get all the fields and field values ​​of a Model instance at runtime, but we cannot assign values ​​to it. In fact, the value we get is a read-only copy of the original value, and even if we get the address of this copy and write a new value, it is invalid.

 class Animal {
 var name : String?
 } 
 
 Mirror(reflecting: Animal()).children.forEach { (child) in  
    print(child.label ?? "" , child.value) // working correctly
 child.value = "cat" // error, cannot assign value directly
 }

Furthermore, Apple's official website still describes the Mirror class that implements the reflection mechanism as follows: Mirrors are used by playgrounds and the debugger. The attitude is very vague and seems not very encouraging, but many libraries in production use it. It can only be said that Apple will not easily remove this capability, but it is unlikely to expect it to make improvements to this capability (such as supporting runtime assignment).

How to bypass restrictions

The simplest way is to inherit NSObject when defining Model in Swift, so that the instance of this Model exists in the objc runtime, and the above-mentioned class_copyPropertyList method and KVC can be used. All the JSON libraries that do not require the mapping relationship to be specified in Swift that we have seen so far use this method.

Then there are libraries represented by ObjectMapper, which complete the assignment when specifying the mapping relationship through operator overloading. There are also many libraries that implement this type.

But what I want to do is to support pure Swift classes running in the Swift runtime without explicitly specifying the mapping relationship of each field. So, if reflection assignment is not possible, then write it directly to memory.

Specific implementation

In Swift, the memory layout of a class instance is regular:

• On 32-bit machines, there are 4+8 bytes in front of the class to store meta information, and on 64-bit machines, there are 8+8 bytes;
• In memory, fields are arranged in order from front to back;
• If the class inherits from a certain class, the fields of the parent class come first;
• Optional will add one byte to store .None/.Some information;
• Each field needs to consider memory alignment;

There is basically no official reference in this regard. Some of the above rules are collected from the summaries of other experts on the Internet, some are mined from some Clang documentation, and some are tested in playgrounds. I was not sure at first, but after implementing HandyJSON for so long, there has been no problem, so I think it is reliable.

Now that we have a way to calculate the memory layout, the rest is pretty easy. For an example:

• Get its starting pointer and move it to the valid starting point;
• Get the field name and field type of each field through Mirror;
• Get the value in JSON according to the field name, convert it to the same type as the field, and write it through the pointer;
• Calculate the alignment starting point of the next field based on the placeholder size of this field type and the next field type;
• Move the pointer and continue processing;

Get the starting pointer of the class instance

In Swift, the methods for getting the starting pointer of a struct instance and the starting pointer of a class instance are different, and are also related to the language version. In Swift3:

 // Get the starting pointer of the struct instance
 mutating func headPointerOfStruct() -> UnsafeMutablePointer<Byte> { 
 
 return withUnsafeMutablePointer( to : &self) {
 return UnsafeMutableRawPointer($0).bindMemory( to : Byte.self, capacity: MemoryLayout<Self>.stride)
 }
 } 
 
 // Get the starting pointer of the class instance
 mutating func headPointerOfClass() -> UnsafeMutablePointer<Byte> { 
 
    let opaquePointer = Unmanaged.passUnretained(self as AnyObject).toOpaque()
    let mutableTypedPointer = opaquePointer.bindMemory( to : Byte.self, capacity: MemoryLayout<Self>.stride)
 return UnsafeMutablePointer<Byte>(mutableTypedPointer)
 }

Get field name and type through Mirror

 Mirror(reflecting: Animal()).children.forEach { (child) in  
 print(child.label ?? "" ) // Get the field name
 print(type( of : child.value)) // Get the field type
 }

Calculate the size of each attribute field in the Model

Swift 3 exposes two interfaces for calculating the size of a type: MemoryLayout.size(ofValue: T) and MemoryLayout.size. Neither of them can be used directly because:

• For each property, we currently only hold its starting pointer, not its instance, so the first interface is not used;
• For each property, we get its type at runtime, and there is no way to instantiate the generic type MemoryLayout<T> to calculate the size. Therefore, I introduced the HandyJSON class and implemented the function in the extension:

 protocol HandyJSON {
 } 
 
 extension HandyJSON {
 static func size () -> Int {
 return MemoryLayout<Self>. size  
 }
 }

Therefore, for each Model class T that implements the HandyJSON protocol, you can directly call T.size() to get the size of T.

Impact of memory alignment

The attributes of class instances are not directly arranged in order according to their respective space sizes, otherwise things would be simple. Like C/C++, the instance memory layout in Swift also takes memory alignment into consideration. After reading Swift's docs and some LLVM materials, MemoryLayout provides an interface: MemoryLayout.alignment. The alignment rule is that the starting address of each field must be an integer multiple of the alignment value. I have forgotten the source of the details. After conducting some complex type tests at the time, it was determined that it was true. So the function in HandyJSON to calculate the starting address of the next field is:

 // Returns the offset to the next   integer that is greater than
 // or equal to Value and   is a multiple of Align. Align must be
 // non-zero.
 static func offsetToAlignment(value: Int , align: Int ) -> Int {
 let m = value % align
 return m == 0 ? 0 : (align - m)
 }

Other situations

The basic types can be processed in the above way. There are also optional types, array types, and dictionary types. The processing methods are similar through traversal, recursive parsing, etc. For example, arrays:

 extension Array: ArrayTypeProtocol {
 static func getWrappedType() -> Any .Type {
 return Element.self
 } 
 
 static func castArrayType(arr: [ Any ]) -> Array<Element> {
 return arr.map({ (p) -> Element in  
 return p as ! Element
 })
 }
 }

Get the Array generic parameter type, then construct an array of that type and complete the assignment.

Conclusion

The main process is like this, and it is relatively simple. The remaining handling of inheritance, combination, etc. is just an implementation issue, so I will not go into details. I always feel that my understanding of the Swift pointer facility is not very good. Maybe there are better uses, for example, there is no need for an empty init() function to initialize an instance of a class. If any students have a deeper understanding in this regard, and have any opinions or suggestions, please feel free to communicate~

refer to

• https://appventure.me/2015/10/24/swift-reflection-api-what-you-can-do/
• https://www.raywenderlich.com/119881/enums-structs-and-classes-in-swift
• http://vizlabxt.github.io/blog/2014/12/23/Swift-Memory/
• https://realm.io/news/russ-bishop-unsafe-swift/
• http://sketchytech.blogspot.jp/2014/10/becoming-less-afraid-of-unsafe-mutable.html
• http://southpeak.github.io/blog/2014/07/06/ios-swift-cpointer-2/
• https://onevcat.com/2015/01/swift-pointer/
• https://appventure.me/2015/10/17/advanced-practical-enum-examples/
• https://github.com/Hearst-DD/ObjectMapper