Availability is one of the important indicators to measure whether a system is good or not. High availability requires that the system can remain available and provide services continuously and stably in the face of various contingencies. This paper introduces the COWAVE NETWORK MANAGEMENT SYSTEM to achieve high availability, including physical layer, platform layer, basic component high availability, service high availability four aspects.
Availability measurement
Availability refers to the probability or expected time share that the system will be up and running within a certain time range.
The availability level is usually described in terms of "N nines", as shown in the figure below:
The time a service is unavailable is called the time of failure. Failure time decreases by an order of magnitude for every level of availability. For large network services, achieving 99.999% availability is already very difficult and requires addressing not only technical challenges but also cost pressures.
For Cowave satellite communication network management system, core services have achieved 99.999% availability, non-core services also reached 99.99% availability.
1. Physical layer
The Cowave satellite communication network management system adopts the deployment scheme of three hardware servers, and the image data is redundantly backed up in each node. In this way, the Cowave network management system is still available when any server fails. The architecture supports dynamic horizontal scaling, allowing users to expand to five physical nodes (or more) on demand for higher levels of availability.
2. Platform layer
The Cowave satellite communication network management system uses containers to deploy application services. Containers are similar to virtual machines, but with looser isolation, sharing operating systems (OS) between containers. Each container has its own file system, CPU, memory, process space, and so on. Because they are separate from the infrastructure, they can be ported across cloud and OS distributions. The container has the advantages of loose coupling, distribution, elasticity and high efficiency. The Cowave satellite communication network management system builds each service as a mirror, enabling each service to run in a separate container. On this basis, Cowave satellite communication network management system uses K8S as container arrangement system to manage containers. Kubernetes implements the following functions:
2.1 Service Discovery and load Balancing
Kubernetes can expose the service externally using DNS names or the container's own IP address. If there is a lot of traffic, Kubernetes can perform load balancing and network traffic distribution to ensure deployment stability.
2.2 Self-repair
Kubernetes is able to restart failed containers, replace containers, kill unresponsive containers, and not notify clients until the service is ready.
3. Basic component high availability
To achieve high availability of the system, you first need to achieve high availability of the underlying components used in the system. Cowave satellite communication network management system adopts different schemes for different basic components to meet the needs of high availability of the system.
3.1 Database - Galera Cluster
The principle of Galera cluster high availability is based on synchronous replication and multi-master architecture. Synchronous replication means that all writes must be replicated to all nodes and must be acknowledged before other reads are allowed. This ensures data consistency and integrity in the cluster. A multi-master architecture is one in which each node in the cluster can handle read and write operations at the same time and has exactly the same copy of data. This improves the performance and availability of the cluster.
3.2 Cache - Redis
The Cowave satellite communication network management System service uses Redis for caching to improve performance. In order to realize the high availability of Redis component, Cowave satellite communication network management system adopts the sentinel cluster scheme. Redis Sentinel cluster is a Redis high availability solution based on Sentinel mode and master-slave replication. In Sentinel mode, one or more sentinel processes are used to monitor the running status of Redis primary and secondary nodes, and realize the functions of fault discovery, failover, message notification, and configuration center. In the master/slave replication mode, one or more slave nodes are used to copy the data of the master node to implement data backup and load balancing.
3.3 Message middleware – Kafka
Kafka is an open source distributed event streaming platform for processing large amounts of real-time or historical data streams. Kafka lets applications publish, subscribe, store, and process event streams for data plumbing, streaming analysis, data integration, and mission-critical applications. Kafka has the characteristics of high throughput, scalability, persistence, high availability and ecosystem, and is widely used in various industries and organizations.
4. Service high availability
4.1 Stateless Service
Stateless service means that the server does not save any state information of the client. Each request is independent and does not depend on previous requests. The advantage of stateless services is that they are easy to scale, since service nodes can be added or removed at any time without affecting client access. All the services in Cowave satellite communication network management system are based on stateless implementation, and on the basis of stateless implementation of distributed multi-node load balancing, greatly improving the availability of services.
4.2 Transaction Management
DTM is a distributed transaction manager that provides an easy-to-use, cross-language, high-performance, highly available, and extensible distributed transaction solution. As a microservice system, Cowave satellite communication network management system uses DTM as a transaction manager to manage all distributed transactions in the system and solve various abnormal problems of distributed transactions in the microservice system, including network delay, process pause, clock drift, etc. The DTM component uses the sub-transaction barrier technology, which can solve the exception problem very easily.
4.3 Stateful Tasks
In addition to stateless services, the system also has some stateful scheduled tasks, such as signaling broadcast, log collection, and scheduled statistics. These tasks cannot achieve high availability through direct horizontal expansion, so Cowave satellite communication network management system adopts a distributed mechanism to independently manage the scheduled tasks in the service, to ensure that it can run normally in the distributed node, so as to achieve high availability of services.
English 
русский 
