Aiti Tribe Stories (34): Self-cultivation of a full-stack engineer

[51CTO.com original article] Shi Yafeng is a full-stack engineer with 5 years of development experience. He has accumulated many ideas, ranging from traditional Java Web to Golang, SVM, hadoop, and UI Automation. He has his own thoughts on everything and takes a different approach. In addition, Yafeng is a part-time magician who has won many awards in provincial and national magic competitions. He also often participates in performances after work. Perhaps magic has inspired Yafeng, such as FoolQQ mentioned below, which makes many developers exclaim that it is magical after reading the source code.

[[214217]]

Self-study programming from scratch and build your own website

Yafeng started working in July 2013, and it has been almost five years. He first came into contact with programming in 2010, just after the first semester of his freshman year. During the Spring Festival, everyone was busy visiting relatives and friends. His uncle, who started a business in Shanghai, came back to celebrate the New Year. After the New Year's Eve dinner, there was an in-depth family conversation as usual. After the greetings, his uncle proposed to build a website. At first, Yafeng refused. After all, he was only a freshman and had only learned C language and had never been exposed to the Web. At that time, he had never even heard of the word "Web". But seeing his uncle's anxious look, Yafeng still bit the bullet and agreed. He is a radical in doing things, and he is still like this until now. He always wants to implement things that others have told him to do as soon as possible. At that time, he had no Internet at home, so he went to a nearby Internet cafe the next day to search for how to build a website. Fortunately, Yafeng found a set of videos that taught the basics of HTML/CSS/JS. Now it seems that the path at that time was correct. For beginners, it is easy to succeed and build confidence step by step by watching videos and following them. After a week of study, I created several main pages of the website by following the instructions, which gave me a great sense of accomplishment. Of course, it took me some time to fine-tune the requirements, and I also learned about the compatibility issues that could not be avoided.

But the next question is, what about the background program? Coincidentally, the video was followed by Java Web development, which led him to the path of becoming a Java engineer. Now he thinks back that it is really amazing. If the video was followed by PHP, he would probably be a PHP engineer now. Yafeng, who had never touched Java, had to learn from basic Java syntax to JSP, and then SQL. The period was not short, and it was not as smooth as before. His uncle was also urging him. To be honest, he didn't understand many things. Yafeng only studied for more than 20 days, but still managed to make a background program. Another semester passed, and during the summer vacation, he went directly to his uncle to do some secondary development of software based on UFIDA's ERP and CRM. In this process, he became familiar with the Java system, learned Spring, MyBatis, Maven, and systematically learned the front end again, and then jQuery, etc. In the following winter and summer vacations, he would go to his uncle to help, constantly improving himself in the process, and also accumulated practical experience for future employment.

Massive large message data persistence

After graduation, Yafeng came to an information technology research company. The first project he came into contact with was a persistence problem for large message data under high concurrency. It was based on TT/TC, but the stress test performance could not be improved. The customer required a single-node 18KB message persistence of 5000TPS. After optimizing the database and code, he only achieved 2100TPS for a single node. However, things soon took a turn for the better. In an accidental experiment, Yafeng stress-tested TC alone and found that writing multiple TC files concurrently without TT could achieve 5000TPS. The previous way of accessing TC using TT could only be based on one TC file. Combined with the application's "more writes and less reads" characteristics and considering the distributed architecture, the following architecture can be designed:

As shown in the figure, each server has a socket-based read server for Web reading, and the write part is written concurrently to multiple TC files. What needs to be solved now is that you need to know where a certain message exists, that is, in which file on which server. Obviously, you only need to save the IP information of the server and the name of the file in the database. This database is like the namenode in hdfs. The data table should contain at least three fields, namely IP, TC file name, and message key. He still remembers this case very clearly. After all, it took a long time to do experiments at that time. This is a good way without changing the original system architecture. In addition, TC is really powerful and stable.

Verification code recognition based on SVM

In 2014, Yafeng went on a business trip to Beijing for more than half a year. He was working on an internal portal project for China Telecom Group. At that time, he wanted to build a mobile portal. He was negotiating interfaces with multiple external systems, and the progress was slow. So he proposed to try crawling. In just one week, he completed many interfaces and accumulated a lot of crawling experience. However, many systems used verification codes, so this was a hurdle that could not be circumvented. Yafeng made recognition based on pixel comparison and SVM, and the effect was good. It was very good to be exposed to these before artificial intelligence became popular in China. In order to give readers a perceptual understanding, for example, you often see various verification codes. They have different colors and can even be severely distorted. Take the number "8" as an example

[[214218]]

How does the computer tell that this is 8 and not 6 or 7?

If we enlarge the "8", we can see that the image is composed of color blocks, each of which has a color value. Then a threshold is set, and the color blocks greater than the threshold are set to black, otherwise they are set to white, and a black and white number "8" is obtained. This is called binarization. Next, from left to right and from top to bottom, white is marked as 0 and black is marked as 1, and a binary array is obtained. This is the input of SVM. SVM will automatically construct a multidimensional space based on these data and determine the number based on the distance. These numbers form the coordinates of the multidimensional space, which can be regarded as a point. Through the training of many "8" pictures, a point group composed of "8" will be formed, which looks like the Milky Way. Obviously, to judge whether a planet belongs to the Milky Way, it must be within the range of the Milky Way, otherwise it may be another galaxy.

Fela, a custom Ajax framework based on singleton objects

The days passed peacefully in 2015. The UI of each product of Yafeng Company also began to be designed uniformly based on UED, abandoning the previous heavy Extjs. This brought a problem. The most used paging table page needs to be re-implemented in combination with the CSS provided by UED. If it is left alone, everyone will have their own implementation. Because the style is very different from that on the Internet, it is impossible to directly apply a certain Ajax framework, so it can only be customized. A lightweight Ajax framework is needed to meet the needs of paging, query, various event support, background data parsing, form data packaging, and as simple as possible. Fela came into being. Fela is based on singleton objects. Unlike traditional Ajax frameworks, the use of singleton objects makes Fela more lightweight. Singletons have many advantages in JavaScript, such as smaller memory overhead, creation upon loading, easy overwriting and rewriting, clear object structure, and easy conversion to JSON.

But doing so also brings some problems. For example, there is only one object in a page. For this reason, Yafeng uses clone instead of new to generate new objects, avoiding coupling in the construction method when initializing large objects, and allowing users to use large objects on demand and write clearly structured code. He designed aliases and method chains, striving to make the development of a traditional table data page, query box, etc. as simple as possible. It took only three days to write Fela. The code is small and exquisite with less than 500 lines. Several products of the company are developed based on this. Basically, a page front end only requires simple configuration, which shortens the development cycle. Most importantly, Fela constrains the code through a limited API, so that the code written by everyone looks similar. Its source code is here >>.

Ruyi, a form tool equipped with high technology

Since 2016, Yafeng has started to perfect his theory. He squeezed out time to read a lot of books after work, such as JVM principles, concurrent programming, TCP/IP principles, etc. He also did some study on the hadoop technology stack. He knew a lot of things before, but most of them were too fragmented and unsystematic. Reading helped him sort out their relationship. In the IT industry, the more you know, the more you feel that you are still very insignificant. On the contrary, when you read less, you are more likely to be complacent. According to the project requirements, he spent a quarter of his time this year to independently build a form platform - Ruyi. Ruyi is a big production compared to Yafeng's previous works, and he also shot a promotional film for it. Ruyi has many features, including full control support, the first visual business logic design, multiple data sources, multiple tenants, online JS programming, etc. In October 2017, Yafeng used this project to participate in the 51CTO Developer Competition. Fortunately, Ruyi has been shortlisted. I hope it can achieve good results in the future. This year, he was appointed as the executive director of the company's software laboratory, responsible for the research and development of innovative projects and the solution of difficult problems.

Let the computer recognize itself!

Yafeng has participated in many emerging projects, such as GIS, voiceprint duration recognition, question-answering robots, and so on. During the development of the GIS project, he proposed a methodology based on allowing the computer to operate its own screen through image recognition, which is like putting a mirror in front of the computer so that it can see itself and operate itself. This saves a lot of manual work for the project and is mainly used for automated batch geometric drawing. Later, it was applied to the operation of QQ. Since the current robots that read and write QQ are based on crawler analysis of SmartQQ messages, and SmartQQ Tencent is no longer maintained, its HTTP server is often unavailable and often needs to be restarted at irregular intervals. Therefore, based on image recognition, Yafeng open-sourced FoolQQ. Since the output image of the screenshot taken by the computer is high-definition, it means that developers can directly implement image comparison based on pixel comparison. Determining whether an image contains a sub-image is as simple as this:

 public static boolean isEqual(int x, int y, BufferedImage image, BufferedImage point) {
 int pointW = point.getWidth();
 int pointY = point.getHeight();
 for (int m = 0; m < pointW; m++)
 for (int n = 0; n < pointY; n++) {
 if (image.getRGB(x + m, y + n) != point.getRGB(m, n)) {
 return false;
 }
 }
 return true;
 }

Fortunately, this project received donations from some individual developers, which gave Yafeng the motivation to continue maintaining it.

【Written at the end】

2017 is coming to an end very quickly. The above are some representative R&D works of Yafeng in recent years. Due to limited space, there are many examples that I will not mention. The workplace of programmers is very fair. Your salary is almost proportional to your knowledge, provided that you really master it. Finally, I will summarize some R&D experiences:

1. Architecture is always the first priority. If your performance is far from the target, please do not try to optimize the details.

2. Don’t reinvent the wheel. Pay attention to the latest developments in the IT field and frequently visit websites like 51CTO. The breadth of knowledge can save you a lot of trouble.

3. If you have to reinvent the wheel, design is particularly important. You must look at the problem from the user's perspective and know who your users are;

4. Personal technical improvement and daily work sometimes conflict with each other. It is important to find a win-win situation. Excellent employees should have a collaborator mentality.

Yafeng and his friends often say that entering this circle is a practice with no end in sight. The best result is that you are always on the road to progress and getting closer and closer to the end, and that is enough.

If you are also willing to share your story, please join the 51CTO developer QQ exchange group 669593076 and contact the group owner Xiaoguan. We look forward to your wonderful story!

[51CTO original article, please indicate the original author and source as 51CTO.com when reprinting on partner sites]