As a programmer, you must know these things about computers

[[130302]]

Storage - block devices, file systems, cluster file systems, distributed file systems, fiber SCSI, iSCSI, RAID, etc.

Network - Ethernet, fiber optic network, cellular network, WIFI, VLAN, etc.

Computer architecture is mainly the CPU instruction set, such as x86, ARM, etc.

USB protocol. Need to know URB packets.

PCI protocol, PCI-E protocol. Modern computer peripherals are all PCI protocol and PCI-E protocol. Graphics cards are now all connected to computers via PCI-E protocol. Relatively speaking, there is a lot less knowledge to learn. To engage in virtualization, you need to have a deep understanding of the PCI protocol.

Image processing – image compression, real-time video encoding, etc.

3D Games

Relational Database

NoSQL Databases

operating system

Distributed operating system

Compilation Principles

Machine learning – now big data is needed!

Understanding these domain knowledge also includes understanding the existing commercial hardware, commercial software, and open source software in the field. In many cases, there are already ready-made tools for the work you want to complete. You can complete the task by using the ready-made tools without developing. Sometimes, you only need to combine existing tools and write some scripts to complete the task.

For example, I wanted to implement a two-way synchronization task. I found an excellent open source software Unison and completed the task successfully by writing a configuration file. No code needed.

Another time, we needed to achieve high availability, and we easily implemented it by calling several open source software using Python.

Write an installer, customize the operating system, and knowing the domain knowledge of the operating system, you can easily get it done by writing a few lines of scripts.

People who do not have domain knowledge may have to carry out a lot of unnecessary development, and may even find out after a long period of development that this is a dead end.

In addition, solid domain knowledge can greatly improve your programming debugging and error-checking abilities. Knowing how compilers and programming language runtimes work will allow you to quickly modify your code based on compilation errors and warning messages.

Knowing the underlying operating mechanism of the operating system, you can quickly find the root cause of runtime errors. For example, I once wrote a Windows upgrade service program. It is a Windows service that needs to execute a DOS script, which will replace the Windows service itself. I found that sometimes the script execution is invalid. After checking for a night, I found that when the Windows service is installed, there will be a permission problem when the script is executed for the first time. The logs are correct, but the actual execution of the script has no effect. But once the Windows service program is started once, it is ok. This must be a problem with the underlying security mechanism of the Windows operating system. Because I don’t know much about the Windows kernel, it took me a long time to find this problem, and I didn’t know the root cause of the problem.

Stage 0 - Newbie in Domain Knowledge

I don't have much knowledge of the field. I use search engines to find some introductory articles about software and hardware in the field, and follow the instructions in the articles to configure and use the software. I can barely use the existing software and hardware.

1st stage - expert in field knowledge

Understand the commonly used hardware in the field, and have an in-depth understanding of the configuration and usage skills of commonly used software in the field. Be able to skillfully build solutions using existing software and hardware, and be able to solve various problems encountered in actual work.

Section 2 - Domain Knowledge Expert

When you not only master the software and tools in this field and know how to use them, but also understand their principles, "knowing why they are so", you will become a knowledge expert in this field.

If you know the principles of network protocols, you can know where the problem may be when there is a problem with the network. Is it a MAC conflict, an IP conflict, or a network loop?

Only when you understand the principles of storage can you understand why one storage method is not suitable for virtualization, another storage method is suitable for virtualization, and another method is suitable for data backup.

Only if you know the PCI protocol, you can know how to virtualize a hardware device.

Only if you know the network card hardware protocol can you simulate a virtual network card that can be used normally by a virtual machine.

Only when you know the video encoding formats and principles can you know which video format occupies the least bandwidth and which video format occupies the least CPU.

Only when you understand the IntelVT/Amd V instruction set can you know how virtualization is implemented.

You understand that a workflow is actually a state machine. When you encounter a complex workflow, you will know how to design a workflow engine that meets the requirements.

3rd paragraph - Scientist

You are an expert in domain knowledge, but your knowledge comes from books and other people.

If you are satisfied with being an expert in the field, you can only copy others and never think of surpassing others. Others may not be willing to tell you their research results. When others tell you, they may have discovered a newer theory and a new generation of products may be released soon.

Scientists are people who explore the unknown, have the courage to innovate, and promote the progress of human society.

Legend has it that a senior executive of Cisco once said half-jokingly: "If Cisco stops developing new technologies, Huawei will lose its direction." This is a mockery of Huawei's only being an expert in domain knowledge, and can only copy but not surpass. I don't know the actual situation of Huawei, but I hope that Huawei has now become a leader.

Irving Jacobs discovered the principle of CDMA code division multiple access and found it to be very promising in communications, so he founded Qualcomm. Qualcomm mainly makes a living from patent licensing fees, and it employs a large number of scientists to conduct research in the field of communications. Some people say that Qualcomm is a patent troll. These people do not understand the value of knowledge. In their eyes, the reasonable price of Windows should be 5 yuan, the price of a CD. The price of an iPhone should be more than 1,000 yuan for a bare phone. Qualcomm is a patent troll, so why don't you troll CDMA and LTE for me to see!

The X86 chip was not designed with virtualization in mind. Therefore, the so-called "virtualization vulnerability" occurs. That is, when some CPU privileged instructions are executed, no exception is thrown in the virtual machine environment, so it is impossible to switch to the host. In this way, virtual machines cannot be run on X86 chips.

VmWare was founded by several American scientists in 1998. They found that they could use binary translation technology to run virtual machines on X86 computers.

Xen virtualization software was also invented by several scientists. They found that as long as the kernel of the virtual machine operating system and the host operating system were modified, the host function could be directly called when the "virtualization vulnerability" instruction needed to be executed, virtualization could be achieved, and the operating performance of the virtual machine could be greatly improved.

Later, Intel added the IntelVT instruction set to its chips, and AMD added the AmdV instruction set to its chips, filling the "virtualization loophole". So there was KVM virtual machine software, which directly used CPU hardware instructions to achieve virtualization.

When KVM executes CPU instructions, it runs directly on the physical CPU, so it is very efficient. However, when the virtual machine runs virtual peripherals, it must use software simulation, so the IO access speed of the virtual machine is very slow.

IBM scientist Rusty Russell created the VirtIO technology by drawing on the R&D experience of Xen. This is to write a set of PCI virtual devices and drivers in the virtual machine. This set of virtual PCI devices has a virtual device memory. The host can access this virtual device memory, and the virtual machine can also access it through the VirtIO driver. In other words, a piece of memory is shared between the virtual machine and the host, which solves the IO performance problem of the virtual machine.

Here’s another search engine story:

A long time ago, I wanted to add a search function to a program. I first used SQL query to implement it, but found it was too slow. Later, I found the open source Lucene project. It uses inverted index technology to create inverted indexes in files, which greatly improves the search speed.

The two founders of Google discovered the secret of links in HTML. They found that they could set the weight of each HTML page through the link relationship of HTML pages. That is the PageRank algorithm. As a result, Google's automatic search engine defeated Yahoo's manually classified search engine.

OK, using reverse index technology and PageRank, as well as a simple HTML crawler robot, we can create a search engine. However, the Internet is huge, and a large number of new web pages are generated every day. It is difficult to build a reverse index for the entire Internet.

Several years later, Google published three more papers: Googlefs, Mapreduce, and Bigtable. So the developers of the Lucene project developed the Hadoop project based on Google's Mapreduce paper. MapReduce uses a large number of computers to store data and calculate, and finally summarize the results. Using Hadoop + reverse index + PageRank, you can create a search engine. Companies such as Yahoo and Baidu have developed their own search engines based on Hadoop.

However, the search engines of other companies are still not as effective as Google. We programmers know this best. For example, I always go out to search just to Google.

Google Blackboard published some articles by Dr. Wu Jun, which introduced a lot of knowledge about machine learning. From the article, we can know that Google actually uses machine learning to analyze the collected pages. Google obviously will not make this formula public. Even if Google really makes this formula public one day, it can be imagined that Google must have developed a more powerful secret formula, and the effect of the knockoff search engine is still not as good as Google's.

Copying is the only way to innovation. Before becoming the leader and leader in a field, one must go through the stage of learning and imitating. But to become the boss of the industry, to become a champion, one must have the courage to overtake on the curve, bravely embark on the road of innovation, and become a real scientist, a real expert!

Summarize

Programming ability can be divided into two dimensions: one is the level of programming skills, and the other is the level of domain knowledge.

Some programmers may spend all their energy on improving their programming skills, but have little knowledge of domain knowledge, which is actually extremely harmful in their daily work. Some needs may already have ready-made, open source and free solutions, or they only need to combine several existing software to quickly solve them, but they have to spend a lot of time developing them themselves. In addition, the lack of domain knowledge makes it difficult to quickly locate the root cause of the problem and solve the bug when the program encounters unexpected situations.