The 4 most important performance indicators for enterprise-level Java applications

Application Performance Management (APM) is a systematic solution for real-time monitoring to achieve application performance management and fault management. Currently, it mainly refers to monitoring and optimizing the key business applications of enterprises, ultimately achieving the goal of improving the reliability and quality of enterprise applications, ensuring that users receive good services, and reducing IT costs.

Although many people have predicted that Java will never recover, it is undeniable that Java still plays an extremely important role in many important projects, especially some large-scale projects of banks and governments. Tao Bingzhe, a Java engineer at OneAPM, a leading domestic APM company, has participated in performance optimization work for large enterprises such as banks and operators many times. He summarized the four performance indicators that enterprise-level Java applications should pay most attention to, mainly including: business affairs, external services, garbage collection and application layout. The following will elaborate on each one:

1. Commercial affairs

Business transactions are a direct reflection of real user experience: they capture real-time performance data that users experience when they interact with applications. To measure the performance of business transactions, you need to capture the overall response time of a business transaction and the response time of each of its components. These response times are then compared with a benchmark that meets business requirements to determine whether the application is functioning properly.

If you are only going to measure one aspect of your application, this article recommends measuring the performance of business transactions. Although container metrics can help you decide when to adjust the size of your cluster, business transactions determine the performance of your application itself. You don't need to ask about the use of the application server thread pool, but rather whether users can complete their business transactions quickly and whether these transactions are performing properly.

A little background: Business transactions are identified by their entry points, which are the entry points where users interact with your business. Such interactions include: a web page request, a web service call, or a message in a message queue. Of course, you can also define multiple entry points for the same web page request based on a URL parameter, or define multiple entry points based on the content of a service call. The key is: business transactions must be associated with your business processes. For example, China Mobile's over-the-air payment service corresponds to multiple atomic services in the system. We should aggregate these atomic services into an over-the-air payment service through corresponding associations for monitoring.

Once a business transaction is identified, its performance is measured across the entire application ecosystem. The performance of each business transaction is compared to its baseline to determine whether it is normal. For example, if the response time of a business transaction is greater than the threshold you set, we determine that it is not functioning properly.
In summary, business transactions best reflect user experience, so they are also the most important dimensions to capture.

2. External Services

External services come in many forms: dependent web services, legacy systems, or databases. External services are systems that interact with applications. The code running in external service systems is often out of our control, but we can control the configuration of these systems, so it is important to understand whether they are running properly and when they fail. In addition, we must be able to distinguish whether the problem is in our own application or in these external service systems.

From a business transaction perspective, we can identify and measure these external services in our own applications. Sometimes, we need to configure monitoring methods to identify the methods that wrap external service calls. But for common protocols, such as HTTP and JDBC, external services can be automatically detected.

Business transactions give you a global view of your application's performance and help you triage performance issues, but external services can greatly affect your application's operation in unexpected ways, so you must monitor them.

3. Garbage Collection

Garbage collection is a core feature that has been retained since the earliest release of Java. It is a love-hate relationship. Garbage collection eliminates the need for manual memory management: when we are done with an object, we just need to delete its reference, and the garbage collector will automatically release it. If you have used languages ​​that require manual memory management, such as C or C++, you will be grateful for this. Garbage collection reduces the tedious steps of allocating and freeing memory space for programmers.

Furthermore, because the garbage collector automatically frees memory that has no references, it reduces the traditional content leak situation where memory is allocated and references to that memory are deleted before the memory is released. Sounds like a panacea, doesn't it?

Although garbage collection achieves the goal of eliminating the need for manual memory management and preventing traditional memory leaks, the price is that the garbage collection process is sometimes quite clumsy. Depending on the JVM, the garbage collection strategy will also be different. In-depth discussion of these strategies is beyond the scope of this article. However, the reader should understand that it is important to understand how the garbage collection period works and the best configuration scheme.

The biggest enemy of garbage collection is the legendary major or full garbage collection. All JVMs have this problem except Azul JVM. Generally, garbage collection can be roughly divided into two categories:

Secondary

main

Minor garbage collections occur relatively frequently to free up short-lived objects. They do not block threads while they are running and have less impact.

However, major garbage collections are sometimes also called "Stop The World (STW)" garbage collections because they block all threads in the JVM while they run.

When the garbage collector runs, it runs a reachability test, as shown in Figure 4. It creates a root set of objects that are directly visible to each running thread. Next, it explores other objects that are referenced by the objects in the root set, and then explores the objects that are referenced by those objects, until all objects are referenced. In the process, it records (marks) the memory addresses of currently active objects, and then sweeps away all addresses that are not in use. More precisely, it releases all memory that is not referenced by any root set object. Finally, it compacts and organizes the memory so that new objects can be allocated memory.

Depending on the JVM, the methods of minor and major collection will be different. Figure 5 and Figure 6 show the operation methods of minor and major collection in Sun JVM.

In a minor collection, memory is primarily allocated to the Eden space until it is full. Then, the copy collector copies the live objects in Eden to the two survivor spaces (to space and from space). Objects left in Eden are removed. If the survivor space is full but there are still extra live objects, these objects are moved to the tenured space. Only a major collection can free memory in the tenured space.

Eventually, the tenured space will fill up and a major collection will be performed. It will not copy live objects that cannot fit in the survivor spaces into the tenured space. At this point, the JVM will block all threads, run reachability tests, clear young data (Eden and both survivor spaces), and compact the tenured space. We call this a major collection.

You might think that the larger the heap, the less often major collections will run. But when they do run, they take longer than on a smaller heap. Therefore, tuning the heap size and garbage collection strategy is also important for the performance of your application.

4. Application layout

The last performance indicator to explore is application layout. With the advent of the cloud, applications are now more flexible: the application environment can be resized according to user needs. Therefore, it is very important to test the application layout to determine whether the number of instances is appropriate. If you have too many instances, your cloud hosting costs will increase. But if you don't have enough instances, business transactions will be affected.

During the evaluation process, the following two indicators are particularly important:

Business transaction throughput

Container performance

Your business transactions should be benchmarked, and you should know the number of instances you need to meet the benchmark at a given time. If the throughput of your business transactions increases suddenly, you need to increase the instances to meet the users.

Another thing to monitor is container performance. Specifically, you want to determine if any instances in your application are under load, and if so, you may want to add instances in that application. It is important to view the instance status from the application's perspective, because a single instance may be under load due to factors such as garbage collection, but if most instances in an application are under load, then the application may no longer be able to support the traffic it is receiving.

Because instances in an application can be scaled individually, it is important to analyze the performance of each instance and adjust the application layout accordingly.