Development and deployment under microservice architecture

Microservice architecture is a concept that has emerged in the past two years. Prior to this, Internet companies had already used microservice architecture to solve practical problems in distributed systems in production environments. For example, the original Taobao system was also a monolithic application. In order to cope with the problem of insufficient system processing capacity as the number of users increased, Taobao carried out a series of service-oriented splitting and transformation of its application system. Taobao's open source Dubbo framework and its internal HSF framework are both implementation results of microservice architecture.

This article will briefly explain the practical experience of microservice architecture projects from the following aspects: architecture selection, relevant tool support in the development and testing environment, personnel division of labor and development and deployment process, relevant design and precautions. ***, based on practical experience, it will discuss the actual needs of improving the development and operation efficiency under the microservice architecture, and further clarify the perfection requirements of the container service management platform implemented by this project.

Project background and microservice architecture selection

This project is an enterprise-level container service management platform. The function of the platform is to manage the application operating environment based on containers, as well as continuous integration and continuous release during the application development phase. Simply put, one of the core functions of the platform is to manage the development and operation environment of complex applications and improve the development and operation efficiency under the microservice architecture. The development background of the project is as follows:

First, the system has typical distributed application system characteristics:

The platform runs on low-configuration servers, such as Huawei RH1288, which allow for hardware failure.

The system platform requires that it can be scalable and deployed according to the actual number of users to ensure the rational use of hardware resources;

Since the system platform needs to run the development and operation environment of several enterprise applications, reliability is very important and no single point of failure is allowed.

Secondly, the system has complex functions, and from the perspective of architecture, the system needs to be divided into multiple levels and several subsystems. Different levels and subsystems need to use different development languages ​​according to specific circumstances and be completed by different development teams.

Third, the project team members are collaboratively developed by remote teams in several cities, so a unified development environment and collaborative tools are essential.

Considering the above project background, this project chooses to develop based on microservice architecture.

Toolset for development, testing, and deployment

"If you want to do your work well, you must first sharpen your tools." Only with the help of appropriate processes and related tool sets can the efficiency of application development under the microservice architecture be improved. This project uses the DevOps process and selects a set of related tool sets to implement application development management and improve the efficiency of development, testing, and deployment.

Code repository: This project uses the distributed code repository Gitlab. Its functions are not limited to code repositories, but also include reviews, issue tracking, wiki and other functions. It is the best tool for code management and remote team communication and collaboration.

Docker image repository, Docker: This project uses containers throughout the entire software development process, and uses containers as the carrier for application release. The application development environment and test release environment are all run in Docker containers. Docker has inherent advantages in managing complex development and operation and maintenance environments. Currently, most Internet companies at home and abroad have already applied Docker to their development or production environments.

K8s: This project uses Kubernates as the basic environment for container scheduling and management. K8s is responsible for scheduling and managing Docker containers in the development environment and test environment.

Jenkins: Rapid deployment and release is inseparable from the old continuous integration star Jenkins. This project uses Jenkins tasks to build code, package applications into Docker images, and finally publish them to the K8s environment to run the container.

Shell script: Write Shell scripts to automate configuration management tasks in the development phase, such as project branching and application release, to lower the threshold for operation and maintenance and improve the efficiency of configuration management and operation and maintenance.

WIKI: The WIKI function on Gitlib is relatively simple, so the project team chose dokuwiki as a tool for remote team collaboration and communication. Team members can update design documents, knowledge sharing documents, announcement information and other information on the wiki to facilitate collaborative development.

ZenTao: In order to facilitate the association of development plans, development tasks and bugs, this project uses ZenTao for development task and bug management.

Personnel division of labor and development process

The complexity of development and deployment of microservice architecture applications is much greater than that of monolithic applications. It is obviously difficult to cope with manual configuration management by operation and maintenance personnel. DevOps advocates the use of automated task processing to achieve software delivery and infrastructure updates, which can be said to be a necessary condition for the development and operation of microservice architecture applications. This project is developed using the development process of the DevOps concept. Tools are needed to achieve automation of deployment and operation and maintenance. At the same time, DevOps emphasizes the collaboration and communication between software developers and other IT staff and management. Therefore, clear division of labor and development process are as important as tools. In layman's terms, with tools, everyone must know how to use the tools and be willing to use the tools to truly achieve the goal of improving R&D efficiency.

The main working members of the project team are nothing more than doing three types of work: development, testing and system management. Here we only explain the work content that is slightly different from the work done by the three types of personnel in the traditional enterprise application development process.

Developer:

a) When developers are doing development and design, they need to update the interface design to the wiki for review and use by callers.

b) In addition to writing program logic, developers also need to pay attention to writing unit test cases. Because distributed application joint debugging is relatively complex, doing the test first when writing a single service and then joint debugging can improve development efficiency.

c) Since this project uses Docker containers as the release carrier, developers may need to modify some parameters in the DockerFile template to package the compiled code into the image during the deployment phase. Compared with traditional development methods, this is an additional requirement for developers. It seems a bit high to let all developers understand Dockerfile. In fact, the DockerFile and scripts in each sub-project are generally good templates written by the main system configuration administrator when building the project framework. If the developer does not understand the relevant technology, he can also communicate the requirements with the configuration administrator, and the configuration administrator will modify the relevant files.

Testers: There is nothing special about the work of testers. They just need to pay attention to the fact that in addition to the testing in each Sprint phase, they also need to cooperate with developers in continuous integration testing.

System configuration manager: Generally, DevOps development relies on cloud-based platforms and automated release tools, so the technical requirements for system configuration managers are lower than those for traditional development. However, the purpose of our project development is to build a platform that can support the DevOps process, and its development itself does not have the corresponding platform foundation. Therefore, the initial system configuration management work of our project was done by architects, who mainly needed to do the following:

a) Deploy and run common service components required for project development, such as the Zookeeper registry, Docker Registry image repository, database, etc.;

b) Write a script for creating branches on git for the sub-projects to facilitate creating branches when testing and releasing versions;

c) Write Dockerfiles for publishing and deploying various types of applications into images;

d) Make or find a ready-made Docker image of the development environment on the Internet, and push it to the private image library used for project development;

e) Write a Shell script to package the sub-project into a Docker image and push it to the image repository.

f) Configure automatic compilation or deployment tasks on Jenkins to achieve continuous integration and deployment.

This article will briefly describe the project development, deployment, joint debugging, and test release processes and specifications, and provide examples of some automated script tools used in various stages of the project.

Code branch management:

As shown in the figure, every project created on git needs to establish at least two branches: develop and master. Developers only have the authority to submit code to the develop branch, and the usual continuous integration and joint debugging obtain code from the develop branch. When testing and releasing the version in each Sprint phase, the configuration manager creates a release branch for testing from the develop branch. When testing and modifying bugs, developers only submit the modified code to the corresponding test Release branch. When the test version is stable, the configuration manager merges the code into the Master branch.

Automatic deployment and release:

The project uses Shell scripts, Dockerfile, K8s configuration files, and Jenkins tasks to achieve automated continuous integration and deployment. The structure of the script file written by the configuration administrator in the project directory is shown in Figure 2.

a) Create a branch shell script, see Appendix 1 for an example;

#!/bin/bash
if [ -z "$1" ]; then
 cat <
  EOF exit 1 fi DEPLOY_VERSION=$1 RP_FILES=(subproject1/kube-rc.yaml subproject1/pom.xml subproject1/Makefile) if [ -z $(git branch -a | grep -e /${DEPLOY_VERSION}$) ]; then git branch ${DEPLOY_VERSION} git checkout ${DEPLOY_VERSION} else git checkout ${DEPLOY_VERSION} git pull fi #Replace the environment pointing in the k8s configuration file, switching from development to testing#Replace the SNAPSHOT in the pom.xml file with the release version number#Replace the latest of the image Tag released in the makefile with the release version numberfor f in ${RP_FILES[@]}; do sed -i -e "s#api.devproject.com#api.testproject.com#g" \ -e "s# 0.0.1-SNAPSHOT # ${DEPLOY_VERSION}-SNAPSHOT #g" \ -e "s#latest#${DEPLOY_VERSION}#g" $f done git commit -a -m "Create Branch ${DEPLOY_VERSION}" git push origin ${DEPLOY_VERSION}

b) Dockerfile example file, taking the Java dubbo service as an image as an example, see Appendix 2:

FROM registry.xcompany.com/java:openjdk-7-jre
MAINTAINER zhangsan
ENV spring.profiles.active="production"
 ENV JAVA_OPTS="-Xmx1024m"
RUN mkdir -p /app 
 COPY target/subproject1.war /app/
 COPY ./startup.sh /app/
 RUN chmod +x /app/startup.sh
 WORKDIR /app
 CMD ["./startup.sh"]
EXPOSE 8080

c) Makefile file: Contains command scripts for compiling the project, packaging the project into a Docker image, pushing the image to the image repository, creating a ReplicationController on K8s, and creating a service on K8s:

IMAGE_PREFIX = registry.xcompany.com/project1/
COMPONENT = subproject1
ifndef BUILD_TAG
 BUILD_TAG = latest
endif
IMAGE = $(IMAGE_PREFIX)$(COMPONENT):$(BUILD_TAG)
ifndef KUBE_OPS
 KUBE_OPS = --server=https://api.devproject.com --namespace=project1
endif
clean:
 mvn clean
compile: clean
 mvn -U -DskipTests=true -Dmaven.javadoc.skip=true package
#Package the current program into a Docker image build: 
 docker build -t $(IMAGE) .
 #Push the current image to the image repository push: 
 docker push $(IMAGE)
run: 
 docker run --rm -it -e spring.profiles.active=application -p 8080:8080 $(IMAGE)
 #Deploy RelicationController
 deploy:
 kubectl create -f kube-rc.yaml $(KUBE_OPS)
redeploy:
 kubectl replace -f kube-rc.yaml $(KUBE_OPS)
undeploy:
 kubectl delete -f kube-rc.yaml $(KUBE_OPS)
 #Create service
 deploy-svc:
 kubectl create -f kube-svc.yaml $(KUBE_OPS)
undeploy-svc:
 kubectl delete -f kube-svc.yaml $(KUBE_OPS)

d) K8s deployment configuration file, create ReplicationController, create service examples, see Attachment 4:

#Create ReplicationController
apiVersion: v1
kind: ReplicationController
metadata:
  name: subproject1
spec:
  replicas: 1
  selector:
    name: subproject1
  template:
    metadata:
      labels:
        name: subproject1
    spec:
      containers:
        - name: subproject1
          image: registry.xcompany.com/project1/subproject1:latest
          imagePullPolicy: Always
          env:
            - name: DUBBO_REGISTRY_ADDRESS
              value: "kube://zookeeper:2181"
            - name: DUBBO_REGISTRY_REGISTER
              value: "true"
          ports:
            - containerPort: 8888

#Create Service
apiVersion: v1
kind: Service
metadata: 
  name: subproject1
  labels:
    component: subproject1
spec: 
  ports:
    - port: 8888
      nodePort: 16888
  selector: 
    name: svc-subproject1
  type: NodePor

e) The configuration administrator configures automatic or manually triggered tasks on Jenkins and configures shell scripts in the Jenkins tasks to achieve one-click deployment of the application. For an example, see Appendix 5.

#!/bin/bash -e
IMAGE=registry.xcompay.com/project1/sub-project1:$IMAGE_VERSION
make compile
if [ $build = "true" ]; then
   echo "docker build -t $IMAGE"
   docker build -t $IMAGE .
   echo "docker push $IMAGE"
   docker push $IMAGE
fi
if [ $undeploy = "true" ]; then
	make undeploy
fi
if [ $deploy = "true" ]; then
	make deploy
fi
if [ $deploysvc = "true" ]; then
	make deploy-svc
fi

The specific process is as follows:

i. Pull code from Git, compile and publish the project;

ii. Package the compiled program package into a Docker image;

iii. Push the packaged Docker image to the image repository;

iv. Jenkins executes Shell script commands, pulls images from the image repository, creates pods and RCs in the K8s environment, and runs the container where the application and its operating environment are located on the K8s platform.

Testing and Release:

As can be seen from the figure, the project's development environment and test environment are two isolated environments.

a) The application code deployed in the development environment comes from the develop branch, and the corresponding Docker image tag is latest, which is used by developers for debugging and testers to assist in integration testing at any time;

b) The application code deployed in the test environment comes from the test release branch that the configuration administrator opens from the develop branch when each Sprint release is tested. The branch name corresponds to a different version number, and the corresponding Docker image tag will also change with the version number. The application deployed in the test environment is mainly used for test verification.

Deployment and joint debugging:

The project is divided into four layers: front-end UI, WEB layer with several web applications, Service layer including several distributed services, and basic bottom layer. Here is a brief description of the debugging methods between each layer:

a) Joint debugging of the front-end and Web layers: The front-end developer starts an Nginx locally and configures the nginx.conf file to point the localhost proxy to the address of the web server. This allows the developer to debug the interaction with the dynamic Web client locally.

b) Joint debugging between the WEB layer and the service layer, joint debugging between service layers, and joint debugging between the service layer and the basic layer can be divided into two methods:

Local debugging: Deploy a dedicated Zookeeper registration center. Developers can register the local address to the registration center for relevant personnel to temporarily call the service for debugging.

Integrated environment debugging: Submitting code triggers a Jenkins task, which packages the service into a container image and deploys it to K8s for joint debugging in a complete system operating environment. The specific integrated environment orchestration depends on k8s.

Microservice layering and service interaction design

There are many famous articles introducing the pros and cons of microservice architecture and design principles, such as Marin Fowler's blog post "Microservices: a definition of this new architectural term" and "Microservices in Practice: From Architecture to Deployment" from the DZone community, which have Chinese translations on technical websites such as InfoQ. This article will not go into details about the concepts and design principles. This section mainly explains the design of the logical hierarchical structure and service interaction of the project.

This project adheres to the following main basic principles of microservice architecture, but some reservations will be made based on the specific project situation.

i. Single Responsibility Principle (SRP)

ii. Ensure that the microservice design supports agile/independent development and deployment of services.

Layered Architecture Design

As shown in Figure 2, the project architecture is divided into four layers: static UI layer, dynamic WEB layer, business service layer, and basic business layer.

i. Static UI layer, which is the user-friendly operation display interface, including static UI design and JS interaction code, mainly using Angulars framework;

ii. The dynamic WEB layer is the "facade" of each business service. It calls and assembles the API of the business service layer according to the needs of the front-end design. Relatively speaking, this layer changes frequently. For example, if the system needs to optimize the process or transform the front-end UE, this layer must be changed accordingly. The dynamic WEB layer is implemented using the Java Spring or Python Django framework;

iii. Business service layer, which expands and packages the basic service layer according to business requirements and functions, is implemented using the Dubbo distributed service framework. The specific version is the Dubbox extended by Dangdang, which supports REST API and has upgraded its support for Spring to 3.x;

iv. The basic service layer is relatively stable, providing some basic functions, and is implemented in multiple languages ​​such as Go/Ruby/Java/Python.

Interactive communication design between layers

i. The interactions between different layers and within the same layer are mainly implemented in accordance with the design principles of the microservice architecture, using lightweight communication mechanisms: REST API, Dubbo API (Hessian protocol) and asynchronous messages.

ii. However, some interactions between services are implemented through a shared database, which violates the principle of "independent services, independent databases" emphasized by the microservice architecture. In this project, each service uses an independent data table as much as possible, but a shared database is used. According to the specific business scenario, taking into account factors such as technical costs and the risk of coupling, some interactions between services at different levels are implemented through data tables. This is also a compromise solution from the evolution of monolithic applications to distributed applications. If the system scale expands in the future, the cost of splitting the database will not be very high. However, it is undeniable that there are risks in using a shared database under the microservice architecture. In the future, some poor designs may increase the coupling between microservices, resulting in microservices no longer being "micro".