Docker Claims Swarm Exceeds Kubernetes Performance at Scale
FOD Analysis – Docker claimed its recently published study on performance comparisons was independent research. They may wish it was “independent” but they seem to have spent the money for the comparison which doesn’t pass the “giggle test” as being independent. It would have been better if Docker developed a performance test that everyone can use.
It’s frustrating to see reports like this being used to justify marketing claims that, upon further review, turn out to be misleading. It is especially disappointing when there are some really terrific developers trying hard to actually get the features right. The companies listed in the survey are all interesting and if there was to be a performance test that is relevant, the test should be as open as the software they’re all working on and that each can run the same test on their own work. Instead of
One claim that Docker makes from this study is “Docker Swarm is 5 times faster.” It sounds like an important point till you read the details – “Half the time Swarm will start a container in less than .5 seconds as long as the cluster is not more than 90% full. Kubernetes will start a container in over 2 seconds half of the time if the cluster is 50% full or more.” Do they really mean that they can save an operations team several seconds during a day and that this is important? A shorting of a coffee break could mean that Starbucks beats them all. It is hard to see this making any difference at all in the life of a management system. These groups are working on important products – it’s too bad that there isn’t a more meaningful way to support the value propositions that matter – it would have been better if Docker maybe turned down the big headline for something more appropriate to the functionally and value of the applications.
Docker also would like to claim this “is a three way orchestration race between Docker Swarm, Google Kubernetes, and Amazon EC2.” While they may wish it to be a three way race, the reality is that it is such an early stage for these products – much will change that’s right, but the finish line is not even in sight.
From the Docker Blogpost.
… A recent survey, of over 500 respondents, addressing questions about DevOps, microservices and the public cloud revealed a three way orchestration race between Docker Swarm, Google Kubernetes, and Amazon EC2 Container Service (ECS).
When you think about which orchestration tool is right for your environment, we believe the following three key things must be considered:
Performance: How fast can I get containers up and running at scale? How responsive is the system when under load?
Simplicity: What’s the learning curve to set up and ongoing burden to maintain? How many moving parts are there?
Flexibility: Does it integrate with my current environment and workflows? Will my applications seamlessly move from dev to test to production? Will I be locked into a specific platform?
Docker Swarm leads in all three areas.
Performance at ScaleWe released the first beta of Swarm just over year ago, and since then we’ve made remarkable progress. In less than a year, we introduced Swarm 1.0 (November 2015) and made clear that Swarm can scale to support 1,000 nodes running in a production environment, and our internal testing proves that.
Kubernetes previously released their own blog detailing performance testing on a 100 node cluster. The problem for customers is that there was no way to really compare the results between these two efforts as the test methodologies were fundamentally different
In order to accurately assess performance across orchestration tools there needs to be a unified framework.
To that end Docker engaged Jeff Nickoloff, an independent technology consultant, to help create this framework, to make it available to the larger container community for use in their own evaluations.
Today Jeff released the results of his independent study comparing the performance of Docker Swarm to Google Kubernetes at scale. The study and article, commissioned by Docker, tested the performance of both platforms while running 30,000 containers across 1,000 node clusters.
The tests were designed to measure two things:
Container startup time: How quickly can a new container actually be brought online versus simply scheduling it to start.
System responsiveness under load: How quickly does the system respond to operational requests under load (in this case listing all the running containers)
The test harness looks at both of these measurements as the cluster is built. A fully loaded cluster is 1,000 nodes running 30,000 containers (30 containers per node).
As nodes are added to the cluster, the harness will stop and measure container startup time, and system responsiveness. These breakpoints happened when the cluster was 10%, 50%, 90%, 99%, and 100% full. At each of these load levels 1,000 test iterations are executed.
What this means is that, for instance, when the cluster is 10% full (100 nodes, and 3,000 containers), the harness will pause adding new nodes. It will instead measure the time it takes to startup a new container (in this case the 3,001st container), and how long it takes to list all the running containers (3,001). It does this particular sequence 1,000 times. The 3,001st container is created, the startup and list times are measured, and the container is removed 1,000 times.
The results show that Swarm is on average 5X faster than in terms container startup time and 7X faster in delivering operational insights necessary to run a cluster at scale in production.
Looking more closely at the results for container startup time, there is a clear performance advantage for Swarm regardless of cluster load level.