The most complete and powerful analysis: Internal analysis of the Alipay wallet system architecture (architecture diagram)

Overview of Alipay System Architecture

Typical processing default

Funds processing platform

Financial Accounting

Payment and Clearing

Accounting Center

trade

Flexible Affairs

Alipay's open source distributed message middleware - Metamorphosis (MetaQ)

Metamorphosis (MetaQ) is a high-performance, highly available, and scalable distributed messaging middleware, similar to LinkedIn's Kafka. It has features such as sequential writes for message storage, high throughput, and support for local and XA transactions. It is suitable for scenarios such as high throughput, sequential messages, broadcasts, and log data transmission. It is widely used in Taobao and Alipay and is now open source.

Metamorphosis is a Java messaging middleware open sourced by Taobao. Regarding messaging middleware, you should have heard of the JMS specification and some open source implementations, such as ActiveMQ and HornetQ. Metamorphosis is one of them.

The origin of Metamorphosis is that I started from studying LinkedIn's open source MQ, which is now transferred to Apache's Kafka. This is a very uniquely designed MQ system. It uses a pull mechanism instead of the general MQ push model. It makes extensive use of Zookeeper for service discovery and offset storage. I really appreciate and agree with its design concept. I strongly recommend you to read its design document. In general, the design of Metamorphosis is completely consistent with it. But why do we still need Meta?

Here is a brief summary of why I rewrote the meta:

1. Kafka is written in Scala, I am not familiar with Scala, and the development of the entire Kafka community is too slow.

2. There are some functions that Kakfa does not implement, but we need: transactions, multiple offset storage, high availability solutions (HA), etc.

3. Meta has some unique features compared to Kafka:

The text protocol design is very transparent and supports protocols similar to memcached stats to monitor brokers

Pure Java implementation, from communication to storage, from client to server, everything is re-implemented.

Provides transaction support, including local transactions and XA distributed transactions

Supports HA replication, including asynchronous replication and synchronous replication, to ensure message reliability

Support asynchronous message sending

Consumption of messages fails, local recovery is supported

Supports multiple offset storages, including database, disk, and Zookeeper. Customizable implementation supports group commit to improve data reliability and throughput.

Support message broadcast mode

A series of supporting projects: Python client, Twitter Storm's spout, tail4j, etc.

Therefore, Meta is a huge improvement over Kafka. Meta has been widely used in Taobao and Alipay. Now Alipay routes 12 billion messages through Meta every day, and Taobao also has hundreds of millions of messages every day.

Meta is suitable for applications

Log transmission: high-throughput log transmission is Kafka's strength.

Message broadcast function, such as broadcast cache configuration failure;

Sequential data synchronization function, such as MySQL binlog replication;

Message routing in a distributed environment (broker, producer, and consumer are all clustered), which has extremely high requirements for order and reliability;

Other functions used as general MQ.