Server performance issues that PHPers should pay attention to – Tingyun Server trial notes

I have been using NewRelic's APM product to monitor the performance of my website's PHP application for a long time. Unfortunately, it is too slow to access foreign services from China. Although New Relic has been launched, the slow access problem has not improved, but has become more and more serious. It may be due to the occasional GFW failure. Sometimes you have to climb over the wall to access New Relic's report. Although climbing over the wall is a necessary skill for programmers, it is too troublesome to climb over the wall just to view a report or check a fault. Recently, I was surprised to find that there are also domestic vendors that provide services similar to New Relic. Tingyun (http://www.tingyun.com/), an APM Saas service provided by Keynote Network, a long-established domestic network performance monitoring vendor, also began public testing of their performance management product Tingyun Server for PHP at the end of 2014. I was very lucky to get a trial account for Tingyun Server. I tested it in my own test environment this week and it feels pretty good. Although it is not yet as mature as New Relic abroad, it is basically usable. I took some time these two days to summarize the test process and usage experience. In addition, I saw the QQ exchange group of Tingyun Server on OSChina: 332097173. Let's get started without further ado. For the test of Tingyun's PHP APM product, I was mainly concerned about the function, performance and stability, so I did a total of 3 tests.

Functional testing mainly tests the functions of system reports and the supporting framework and backend services.

Performance testing focuses on the impact of deploying and not deploying APM probes on application performance.

Stability mainly depends on whether the probe will introduce unstable factors to the application and the availability of data collection reports.

1. Functional testing

The testing at this stage is mainly to test what functional modules are provided by Tingyun Server's PHP probe, as well as the degree of support for PHP applications and their backend architecture.

Test environment:

In order to comprehensively test the functions of Tingyun Server APM products, I built a typical PHP application WordPress and installed some third-party plug-ins and backend services to simulate some other services I use in the production environment. The application architecture topology diagram and component list of the test environment are as follows:

Test environment configuration table and topology diagram

In the test environment, the application server and MySQL database service were installed on the same server, and the other three NoSQL services were installed on other machines in the intranet test environment. A separate Linux server was used to simulate client access to websites and applications. In subsequent performance tests, in order to reduce the impact of the network environment and other services on data accuracy, two network cards were added to the application server and the test machine and directly connected using network cables, using independent IP address segments 192.168.4.x.

Testing process:

1. Install the Tingyun Server PHP probe on the application server, and use the wget crawler mode on the client test machine to simulate user click access to the entire website. After the client maintains access for 1 hour, compare the data and functions from their respective reporting platforms.

2. After installing the Tingyun PHP probe, restart the database and httpd service before testing. Run the following script on the client test machine to simulate access:

3. Since the first test is a functional test and does not consider the impact of network performance, the 192.168.2.x network segment address is used for simulated access. During the test, in order to simulate the performance problems of the application, some artificial processing was also done on the MySQL database to increase its query time, thereby reducing application performance.

Test result output: 1. Functional module The report function module provided by Tingyun Server is similar to New Relic's APM. For old users of New Relic, it is very easy to use.

2. Performance indicators The performance indicators provided by Tingyun reports mainly include the following items:

Among the performance indicators, Tingyun provides the important performance indicator Apdex index. Students who want to know more about this index can go to the official website of the Apdex organization (http://apdex.org/) for in-depth understanding. We know that the most important thing about this index is the setting of the T value and the T value must be displayed at the same time when the index data is displayed. From the chart, the default application T value of Tingyun is 0.5 seconds.

Tingyun Apdex Index Chart

3. Application performance breakdown

Tingyun uses a stacked area chart to decompose and display application performance in the application overview, and Tingyun's response time decomposition is relatively complete, basically covering all items that the foreign New Relic APM can provide: application layer time, blocking time, database call time, Redis response time, Memcached response time, MongoDB response time and external service time, a total of 7 performance indicators.

Listen Cloud response time breakdown chart

4. Topology map function

Tingyun can automatically identify the application architecture and draw the logical topology of the application. However, Tingyun Server does not support dragging and clicking to drill down. The performance trend chart of each service node can only be displayed by hovering the mouse. However, it is relatively comprehensive in terms of service identification and support. Multiple service nodes including Memcached, MongoDB, Redis, SQL database and external service hosts can be identified and displayed normally. However, there is a bug in the display of external services: the external service host name is displayed incorrectly and there are garbled characters.

Tingyun application topology diagram

5. Transaction performance analysis function For the performance analysis of each transaction in the application, Tingyun Server provides a transaction list and a series of performance charts.

Web application process performance analysis report of Tingyun Server

The performance decomposition of the Web application process of Tingyun Server can be broken down into the performance time of the database, code module, external service and NoSQL service.

Tingyun's Web application process decomposition can see code segments, databases, NoSQL and external service calls

Tingyun Server provides a trace function for Web transaction analysis, which retains detailed trace data for particularly slow Web transactions. The data includes time statistics of code modules, the flow and time consumption of code execution, HTTP parameters, and SQL statements called during the request process, etc.

The web process tracking analysis of Tingyun Server shows very detailed code running process data. For example, when the performance of the code segment accessing the database is slow, the relevant SQL statements and the code call stack information detailed to the code line will be displayed on the corresponding code segment at the same time, but the display of HTTP request parameters is not seen in the tracking details for the time being.

Listen to the Web process tracking details of Cloud Server to display the slow code call stack and SQL query statements

6.SQL performance analysis function

In terms of SQL performance analysis, Tingyun Server provides a SQL performance list and a series of SQL performance and throughput charts to show the performance problems of SQL statements. In terms of sorting the list, Tingyun Server adds sorting by total time consumption and Web process combination on the basis of sorting by average response time and throughput. For particularly slow SQL queries, Tingyun Server provides a slow SQL recording function. At the same time, Tingyun Server's slow SQL tracking data is very comprehensive. In addition to the detailed SQL statements of slow queries (it also provides a confusion function for SQL statements, which will be mentioned in the parameter setting comparison later), it also provides a scatter plot of slow queries, the name of the application instance that executes the SQL statement, the caller PHP script, the execution plan of the statement, and detailed code call stack information down to the line of code. Among them, the scatter plot of slow SQL queries is very intuitive, and you can clearly see the time period and sample distribution of slow queries at a glance. The analysis of the execution plan also eliminates the need for operation and maintenance personnel to find DBAs to manually analyze SQL statements. The call stack allows R&D personnel to directly locate the line of code where the slow SQL query is located.

Listen to the very detailed slow SQL query tracking data provided by Cloud Server

7. Error Tracking

In order to test the function and accuracy of Tingyun's error analysis, I specially modified a page on WordPress, artificially introduced a syntax error in the PHP code, and made the application execution time exceed the default maximum execution time of PHP applications (30s) for a period of time. The figure below is a statistical chart of the application error rate and a list of error logs provided by Tingyun.

Error rate chart and error list of Tingyun Server

Tingyun Server's error analysis also classifies and sorts the error types, listing the percentage of various error types. And in the list of error logs, it also summarizes and sorts the error logs by error type. For the definition of errors, Tingyun also avoids the situation where the errors in the error rate chart seem to only count E_ERROR level errors, but the list contains warning level error messages such as E_WARNING and E_NOTICE, resulting in inconsistencies in the number of errors in the chart and list. These warning messages are actually not that important and users don't need to care too much about them.

Tingyun's ErrorTrace data provides intuitive scatter plots of error occurrences, error URIs, error time periods, application instances, times, error messages, error call stacks detailed to the line of code, HTTP request information and request parameters, etc. It should be the best in China.

Listen to the error tracking log of Cloud Server

8. Parameter settings

Regarding the settings of monitoring probes, such as the ApdexT value mentioned above, Tingyun Server provides relatively complete functions, including ApdexT value, thresholds for various tracking actions, collection parameter options, parameter and SQL statement obfuscation, and other settings. All settings can be set online and automatically take effect on the specified application probe without restarting the application server.

The obfuscation function of parameters and SQL statements is very important, and some sensitive fields in SQL statements and page forms, such as usernames and passwords, must be obfuscated before they can be transmitted, otherwise it will bring great hidden dangers to the security of the application. For the collection of trace data (such as slow SQL queries and slow transactions), you can set the query to be slow before collecting trace data to avoid collecting too much trace data.

Tingyun Server provides relatively complete configuration file settings and online parameter setting functions, especially the function of modifying parameter settings online without restarting the application server, which is very useful.

Application settings of Tingyun Server

2. Performance Test

Regarding the maturity of APM products, especially APM products in the probe mode, in addition to the functions, what I care most about is the impact on performance. Because the introduction of application probes is to solve the performance problems of applications. If the application performance is greatly reduced due to the probes, it will not be worth the loss. In the early days, some traditional APM products could only be used in test environments and could not be used in online production environments due to the severe performance degradation. So in response to this problem, I designed a test to test the response speed of the application when the probe is installed and when it is not installed, so as to evaluate the impact of the probe on the application performance.

Testing process:

The test also simulates user requests on the client test machine to see the response speed of the application. This test simulates access to Article 2 (i.e., the article that is only accessed by MySQL). In addition, in order to reduce the impact of the network on the test results, this test used a direct network port IP for testing, and conducted a large sample size and large access test to obtain more accurate statistical results. The test tool was completed using the simple but powerful ab (Apache Benchmark). The two tests were completed in the application environment without installing the probe and installing the Tingyun probe. Each test used ab to access the test target URL 10,000 times, with a concurrent number of 10. Restart the application server being tested before each test. Use the following command to test:

Test results:

The final test results are as follows. Below are screenshots of the test results without a probe and with the Tingyun probe installed:

Test results without any probes installed

Test results of installing the Tingyun Server probe

Performance test results summary

According to the test results, the probe of Tingyun Server will cause a performance drop of 105 milliseconds. The impact on application performance is within an acceptable range.

3. Stability Test

Stability should be the most basic requirement for APM products. First of all, the stability of the application cannot be affected by installing application probes. Secondly, the product itself, including probes and reports, must be stable and can provide continuous services without interruption. Stability testing requires a long time and a complex application environment to test. Since I don’t have so much time to do it, I only used the functional testing method of test one to do a day of continuous testing. The test results are as follows:

Tingyun Server's products always collected and displayed data normally during the test period

Judging from the test results, the probes and reports of Tingyun Server are relatively stable. During a day of continuous testing, they always work normally and the data display in the reports is also continuous.

IV. Conclusion

Judging from the test results in the past few days, Tingyun's current products are relatively complete. Except for some minor bugs, they can basically reach the level of use in production environments. Later, we will try to find a few application nodes to deploy in the production environment. I heard that Tingyun Server has been publicly tested, so you can try it.