Multi AZ Kubernetes with kops in AWS regions with only 2 AZ
Peter Gundel

Peter Gundel @peterfication

About: Freelance software engineer. Former CTO at store2be.com in Berlin, Ruby ninja with JS experience, interested in Rust and Kubernetes.

Location:
Munich
Joined:
Mar 28, 2017

Multi AZ Kubernetes with kops in AWS regions with only 2 AZ

Publish Date: Mar 30 '17
0 5

At the moment we are setting up Kubernetes at store2be. We are a company from Germany and most of our business is taking place in Germany. Therefore, we want our AWS servers to be in Frankfurt.

We use kops for our Kubernetes cluster management and we want our cluster to be a multi availability zone (AZ) deployment. With this in mind we tried out the following command:

$ kops create cluster \
    --node-count 4 \
    --zones eu-central-1a,eu-central-1b \
    --master-zones eu-central-1a,eu-central-1b \
    --node-size t2.medium \
    --master-size m3.medium \
    --yes \
    kubernetes-cluster.example.org
Enter fullscreen mode Exit fullscreen mode

But the result was not that promising:

There should be an odd number of master-zones, for etcd's quorum.  Hint: Use --zones and --master-zones to declare node zones and master zones separately.
Enter fullscreen mode Exit fullscreen mode

So unfortunately, at the moment it seems that it is not possible to use kops out of the box in an AWS region with only 2 AZ. But after some digging we found a Github issue that described our problem and even had a work around for that! So the following steps are based on the solution provided by Kamil Hristov.

1. Create a cluster.yml for a AWS region with 3 AZ

Create a cluster in eu-west (because it has 3 AZ):

$ kops create cluster \
    --node-count 4 \
    --zones eu-west-1a,eu-west-1b,eu-west-1c \
    --master-zones eu-west-1a,eu-west-1b,eu-west-1c \
    --node-size t2.medium \
    --master-size m3.medium \
    --yes \
    kubernetes-cluster.example.org
Enter fullscreen mode Exit fullscreen mode

Dump the cluster and instancegroup configuration and append it:

$ kops get cluster --name=kubernetes-cluster.example.org --output yaml > cluster.yml && \
  echo "\n---\n" >> cluster.yml && \
  kops get instancegroups --name=kubernetes-cluster.example.org --output yaml >> cluster.yml
Enter fullscreen mode Exit fullscreen mode


`

2. Adjust the cluster.yml

  • Replace all the occurrences of eu-west with eu-central
  • Replace all the zone definitions of eu-central-1c with eu-central-1a
  • Replace all the name definitions of eu-central-1c with something different than eu-central-1a like for example eu-central-1a2 (otherwise there will be name conflicts)

3. Create the cluster by using the adjusted cluster.yml

`

We have to use replace here, because the cluster spec already has

been created in step 1.

$ kops replace -f cluster.yml
$ kops create secret --name kubernetes-cluster.example.org sshpublickey admin -i ~/.ssh/id_rsa.pub
$ kops update cluster --yes
`

And ✨, there is the multi AZ cluster in Frankfurt 🎉

I created this article because it took us a while to find out about this solution and I hope we could help someone with it. Thanks to Kamil Hristov for sharing the solution. Thanks to Tom Houlé for reviewing the article.


Hi there, we're store2be, a Berlin based startup that builds a SaaS enabled marketplace for short term retails space. If you like what we are posting you might wanna check out the store2be tech page or follow our Medium channel.

Comments 5 total

  • fii
    fiiMay 6, 2017

    How is storage shared between the AZs ? is does your storage span all AZs ?

  • Yevgeniy Ovsyannikov
    Yevgeniy OvsyannikovDec 25, 2017

    This should be fixed!

    kops create cluster --zones=us-east-1c --master-count=3 k8s.example.com will create 3 masters all in us-east-1c

    You can also:

    kops create cluster --zones=us-east-1b,us-east-1c --master-zones=us-east-1b,us-east-1c --master-count=3, it will round-robin around the master zones, so it will so pick b,c,b

  • Janusz Bialy
    Janusz BialyMar 8, 2018

    In terms of high availability and in order to maintain proper quorum for the cluster, an odd number of master nodes should be used!

    github.com/kubernetes/kops/blob/ma...

    • Peter Gundel
      Peter GundelMar 8, 2018

      This is true and this article doesn't state otherwise. It's just about getting a kind of high availability if you only have two AZ available.

Add comment