Deep learning has become a magic weapon for front-end development: automatically generating code based on UI design drawings

A neural network may be needed between UI designers and front-end engineers.

Recently, Uizard Technologies, a startup company in Copenhagen, developed a system called "pix2code". Using deep learning, this system can directly output the corresponding code based on the input graphical user interface (GUI) screenshot, thus eliminating the process of manual coding on the front end.

Currently, the accuracy of the codes generated by pix2code has reached 77%, and these codes are compatible with three platforms: Android, iOS and Web interface.

Just watch the video below to see how amazing it is.

Tony Beltramelli, founder of UIzard, said that in order to come up with this neural network, the R&D team had to overcome three major difficulties:

• The first is the computer vision level - computers cannot automatically recognize and understand given objects and scenes, nor can they determine their locations and features;

• The second is the language level — they need to teach the neural network to understand text so that it can create accurate samples;

• Finally, the neural network needs to understand the relationship between the code, text, and the corresponding image.

Beltramelli also said that in the future, they may use generative adversarial networks (GANs) to further improve pix2code. GANs have proven their advantages in generating sequences and images, but because research in this field is still in a relatively preliminary stage, it still takes a lot of time to train the neural network.

Attached paper address:

https://arxiv.org/abs/1705.07962

GitHub address:

https://github.com/tonybeltramelli/pix2code

In addition, on GitHub, Beltramelli answered some common questions about pix2code. QuantumBit compiled this part as follows:

Q: When will the dataset be made available?

A: We submitted the paper to NIPS this year, and the dataset will be open to the public after they give the acceptance or rejection result, which will be around September. At that time, we will provide the GUI screenshots included in the dataset, the related DSL code, and three target codes for iOS, Android, and Web interfaces.

Q: When will the source code be released?

A: As written in the paper, we had no plan to open source the source code. But we didn’t expect this project to attract so much attention, so we decided to open source the pix2code implementation code and dataset mentioned in the paper.

Q: Will pix2code support other target platforms/languages?

A: No, pix2code is just a research project and it will remain as described in the paper. This project is really just a small demonstration of the work we do at Uizard Technologies. Of course, you are welcome to fork it and experiment on other target platforms/languages ​​yourself.

Q: Can I use pix2code in my front-end projects?

A: No, pix2code is just an experimental project and you can't use it for your specific case yet. But we are working hard to make it commercially available.

Q: How is the performance of the model measured?

A: The accurate or incorrect results reported in the paper are obtained at the DSL level by comparing the generated tokens with the expected tokens. If there is any difference in length between the two, it will also be considered an error.

Q: How long does it take to train this model?

A: On an NVIDIA Tesla K80 GPU, it takes less than 5 hours to optimize the 109 * 10^6 parameters in a dataset. So if you want to train this model on three target platforms, it will take about 15 hours.

Q: I am a front-end developer. Will I be unemployed soon? (I have asked this question many times with great sincerity...)

A: AI will not replace front-end engineers so quickly.

Even assuming that there was a mature version of pix2code that generated code 100% accurately on every different platform/language, a good front end would still need logic, interactivity, advanced graphics and animations, and all the other things that users like.

Our goal in making this thing is to fill the gap between UI/UX designers and front-end developers, not to replace them. We hope to enable designers to create better, while allowing developers to spend more time on those core functions.

We believe that in the future AI will collaborate with humans rather than replace them.

--------------------------------------------------

Compiled by Tang Xu

Quantum Bit Report | Public Account QbitAI