本文共 6623 字,大约阅读时间需要 22 分钟。
转自官方文档:https://kubernetes.io/docs/admin/cluster-large/
At v1.5.1, Kubernetes supports clusters with up to 1000 nodes. More specifically, we support configurations that meet all of the following criteria:
A cluster is a set of nodes (physical or virtual machines) running Kubernetes agents, managed by a “master” (the cluster-level control plane).
Normally the number of nodes in a cluster is controlled by the the value NUM_NODES
in the platform-specific config-default.sh
file (for example, see ).
Simply changing that value to something very large, however, may cause the setup script to fail for many cloud providers. A GCE deployment, for example, will run in to quota issues and fail to bring the cluster up.
When setting up a large Kubernetes cluster, the following issues must be considered.
To avoid running into cloud provider quota issues, when creating a cluster with many nodes, consider:
To improve performance of large clusters, we store events in a separate dedicated etcd instance.
When creating a cluster, existing salt scripts:
On GCE/GKE and AWS, kube-up
automatically configures the proper VM size for your master depending on the number of nodes in your cluster. On other providers, you will need to configure it manually. For reference, the sizes we use on GCE are
And the sizes we use on AWS are
Note that these master node sizes are currently only set at cluster startup time, and are not adjusted if you later scale your cluster up or down (e.g. manually removing or adding nodes, or using a cluster autoscaler).
To prevent memory leaks or other resource issues in from consuming all the resources available on a node, Kubernetes sets resource limits on addon containers to limit the CPU and Memory resources they can consume (See PR and ).
For :
containers: - name: fluentd-cloud-logging image: gcr.io/google_containers/fluentd-gcp:1.16 resources: limits: cpu: 100m memory: 200Mi
Except for Heapster, these limits are static and are based on data we collected from addons running on 4-node clusters (see ). The addons consume a lot more resources when running on large deployment clusters (see ). So, if a large cluster is deployed without adjusting these values, the addons may continuously get killed because they keep hitting the limits.
To avoid running into cluster addon resource issues, when creating a cluster with many nodes, consider the following:
Heapster’s resource limits are set dynamically based on the initial size of your cluster (see and ). If you find that Heapster is running out of resources, you should adjust the formulas that compute heapster memory request (see those PRs for details).
For directions on how to detect if addon containers are hitting resource limits, see the .
In the , we anticipate to set all cluster addon resource limits based on cluster size, and to dynamically adjust them if you grow or shrink your cluster. We welcome PRs that implement those features.
For various reasons (see for more details) running kube-up.sh
with a very large NUM_NODES
may fail due to a very small number of nodes not coming up properly. Currently you have two choices: restart the cluster (kube-down.sh
and then kube-up.sh
again), or before running kube-up.sh
set the environment variable ALLOWED_NOTREADY_NODES
to whatever value you feel comfortable with. This will allow kube-up.sh
to succeed with fewer than NUM_NODES
coming up. Depending on the reason for the failure, those additional nodes may join later or the cluster may remain at a size of NUM_NODES - ALLOWED_NOTREADY_NODES
.
关于GCE机器类型的描述:
Standard machine types are suitable for tasks that have a balance of CPU and memory needs. Standard machine types have 3.75 GB of RAM per virtual CPU.
Machine name | Description | Virtual CPUs1 | Memory (GB) | Max number of persistent disks (PDs)2 | Max total PD size (TB) |
---|---|---|---|---|---|
n1-standard-1 | Standard 1 CPU machine type with 1 virtual CPU and 3.75 GB of memory. | 1 | 3.75 | 16 () | 64 |
n1-standard-2 | Standard 2 CPU machine type with 2 virtual CPUs and 7.5 GB of memory. | 2 | 7.50 | 16 () | 64 |
n1-standard-4 | Standard 4 CPU machine type with 4 virtual CPUs and 15 GB of memory. | 4 | 15 | 16 () | 64 |
n1-standard-8 | Standard 8 CPU machine type with 8 virtual CPUs and 30 GB of memory. | 8 | 30 | 16 () | 64 |
n1-standard-16 | Standard 16 CPU machine type with 16 virtual CPUs and 60 GB of memory. | 16 | 60 | 16 () | 64 |
n1-standard-32 4 | Standard 32 CPU machine type with 32 virtual CPUs and 120 GB of memory. | 32 | 120 | 16 () | 64 |
1For the n1
series of machine types, a virtual CPU is implemented as a single hardware hyper-thread on a 2.6 GHz Intel Xeon E5 (), 2.5 GHz Intel Xeon E5 v2 (), 2.3 GHz Intel Xeon E5 v3 (), or 2.2 GHz Intel Xeon E5 v4 ().
2Persistent disk usage is charged separately from
432-core machine types are not available in Sandy Bridge us-central1-a
and europe-west1-b
.
转载地址:http://gtgmb.baihongyu.com/