Return of the Clustering: Kubernetes for Drupal

>> TESS FLYNN:  All right.  Let's get started in case you don't know what room you're in this is "Return of the Clustering:  Kubernetes for Drupal".  So the first question might be had is where can I find the links in this presentation yes this presentation is already online at socketwench.github.io/return-of-the-clustering.  So take a moment to write that down and then we'll get going.  Socketwench.github.io/return-of-the-clustering.  How did the Jedi do this?  This keeps falling down on my face.  They must have special Jedi hairpins or something.  Okay.  So you might be wondering who is this weird person in front of you that's talking to you from your computer and why are they dressed like a Jedi.  And well the answer to this is I'm Tess Flynn otherwise known as socket wench, that's wench not wrench I'm a blogger and DevOps engineering at TEN7.  This is a full service Drupal web agency we create and care for Drupal powered Websites.  We are based out of Minneapolis, Minnesota.  Mostly.  Actually we have several employees outside of Minnesota right now and we are fully remote distributed company.  And if you are interested, we have just released a podcast helping you to give you tips on how to convert your business to work from home.  

You can find that at TEN7.com.  We provide a number of different services, site and housing migration, Drupal 9 upgrades site audits and consultancy give us a hi for the wonderful humans@TEN7.  TEN7.com.  As you might guess from the name of this presentation this is part of a trilogy this is a series presentation I've been giving the first a ride the whale Docker for Drupalists tells you all about how to put Docker into your project workflow.  And how to actually standardize your environment throughout your team.  Also avoid deep hurting deployments beyond git which tells you how to create a new continuous integration distribution system for your site.  And reduce any additional process overhead necessary to deploying new releases as you need.  

So a while ago my boss came to me and asked, hmmm, we already use Docker locally.  But wouldn't it be great if we used Docker in production?  

And of course, you know, being a good DevOps person I said, sure, sure, it would.  

(Chuckles).

>> TESS FLYNN:  And then a tumbleweed passed between us.  But well, let's talk about Docker in production.  

So earlier before that conversation I had been researching this Docker in production can't be that hard, right?  You take a Linux server.  You put Docker on it.  We put containers on the Docker and that's it.  You're Docker in production.  Right?  

Well, I did some research.  And there was a number of valid criticisms some people called it insecure.  Some people had problems with its performance and some people had less constructive criticism about it.  And then I realized it was going to be a little bit more complicated.  In order to actually get this to work.  

So let's talk about the big one, Docker security.  

Docker maps the users between the host operating system and the container.  Which means that whatever user the containers are running is the same user it's running on the host operating system and this is fine except most containers run as root.  And this is really not good because now despite the fact that there's a number of different security sandboxing mechanisms within Docker to prevent access to the underlying host operating systems there are a number of security bugs which allow container escapes so this is actually a problem and we need to work around that so we don't want to use just any container we find.  We want to secure any containers we use and we want to make sure that they never run as root.  

Another problem is passing configuration.  Usually we're used to using Docker compose and doing that locally.  And a lot of configuration is passed via environment variables.  And this works locally because we really don't care too much about the security of our local development environment on our own laptops.  We're mostly concerned about the security in a prod or a shared environment that has multiple people accessing it remotely.  Environment variables are really not a great mechanism for that.  And the reason why is because a lot of developers don't treat environment variables as securely as files you can see this easily by going to any hosted Drupal site and going to admin reports, status report, go to the PHP information and on several major Drupal hosting platforms you'll find your database log-in right there.  

That's not good.  

So instead we want to use a mechanism within Kubernetes called secrets and config maps actually Docker also supports secrets but we'll talk about that.  

Okay.  So running the containers as non-root.  We're not using environment variables to do any of our configuration.  So now we can just put them all in a Docker host and -- wait a minute.  What about multiple Docker hosts?  Hmmm . . . so a single node Docker is really easy.  All we have to do is install Docker on a single serve and run all of the containers on the same system and everything works the problem is that the scaling is vertical only.  Your only choice when you need more resource is to make the server bigger.  And that runs into a number of limitations very, very quickly.  Some research has indicated that X 86 architecture tops out around 16 cores of CPU.  Any more than that and performance goes down.  Not up.  

The other problem is orchestration.  We need some component to actually coordinate multiple Docker hosts to act as one so they can deploy instances, distribute load and deliver resources.  

Every -- it's not every individual server within the cluster for itself.  They all have to act as one.  

Now, Docker does provide an orchestrator out of the box called swarm mode it's Docker's own orchestrator and it's in every installation of Docker even on your own laptop.  It's built in.  And it's free to use.  

So we'll use Docker swarm.  That doesn't seem so wrong, right?  We'll do that.  Fine.  

So how Docker swarm works is that when you enable swarm mode on a Docker host, it becomes what's called a node.  And here is where all of us Drupalists have to sigh a little bit.  Yes, another node.  Ugh . . . 

If we add another Docker host to the swarm, another node, that actually becomes our swarm and we can continue to add as many nodes as we need to in order to fill out all of our capacity.  

The problem is this isn't really a complete solution.  In order to make it work, we need a front side load balancer that has the ability to know where the containers are within the swarm that need to receive the traffic.  

Once they receive and once the load balancer receives traffic it needs to route it to the particular ports within the swarm.  

Also we need to figure out persistent storage.  Swarm mode doesn't care about storage at all.  

So we have to set up a SAN, an NFS, some kind of external storage mechanism that multiple containers can access simultaneously.  

So really swarm mode is only one piece of a cluster.  It's not the whole cluster.  

Also swarm mode had a number of different problems.  It had a lack of hosting choice.  There was Docker cloud a few years ago.  And it was way too expensive and that's probably why it doesn't exist anymore.  

But the other choice is that we could self-host Docker swarm and then sort out the load balancer and storage that's actually a perfectly valid option.  

The problem, though, for me personally is that Docker swarm just felt too uncontrolled.  It made too many decisions for me.  And made me modify my application versus the infrastructure.  

The problem is that containers were the only manageable unit within swarm mode.  Everything had to be a container.  And every container also got its own internal load balancer.  And this created a number of different Drupal problems because you can't have multiple instances of MySQL go through a load balancer because it gets really confused.  

The same with Solar and in some cases memcache, as well.  

So after researching all of this, I just kind of felt like, that's it, I'm done.  There's nothing more for me to do with containers.  I might as well just go back to normal servers.  That's all.  

But then, you know, I found something.  Like a relic from a more fantastic age.  And I knew that things were going to be different.  

So let's talk about kee-be-what is.  It's Kubernetes it's down by the cloud compute foundation and the thing I really like about Kubernetes is it has expressive power.  Pods are only one of many manageable objects Kubernetes.  This allows you to tailor Kubernetes to your application rather than the other way around.  And this is vital for Drupal sites because Drupal and a lot of times that we use it, functionally it's a black box.  We don't modify core deeply in order to run it on every individual piece of infrastructure.  

We want to install modules and treat those modules more or less as a managed unit outside of ourselves.  So it's much more important for us to tailor the infrastructure to Drupal rather than the other way around.  

So how does Kubernetes work?  In Kubernetes we're going to have one or more servers that are worker nodes.  These workers are going to run pods, which are our application.  

In order to make sure that all of these worker nodes work together, we also have a dedicated manager node.  The manager node is going to monitor the health of the cluster and make sure that every node has an equal amount of work provision to it.  

We can add as many workers as we need to to the cluster and we can do this dynamically as we need to and even take nodes away when demand lessens and this is our Kubernetes cluster.  

We can connect the Kubernetes cluster directly to the Internet.  We don't need a front side load balancer.  

Now, persistent storage is a bit complicated still for Kubernetes.  But less so for a variety of reasons that we'll get into.  

The thing is this is just how Kubernetes works physically how does it work logically.  So when we actually interact with Kubernetes, we're going to create a deployment object.  That deployment object describes our application and the containers that it needs to run.  In order for that deployment to be accessible to -- from the Internet or to access other services within the cluster, we defined another option called a service.  

Then if we need to access persistent volumes we can create something called a volume claim that represents the persistent storage.  So our deployment can actually specify how you can access me and what storage I need from inside Kubernetes itself.  It doesn't need any external information for that.  

If we want to actually connect multiple applications together, we can actually have different applications reference the other deployments service object and that way they can communicate to each other so this is a lot of ways to describe your workload and really this is only part of the Kubernetes model.  

There's a lot more that I haven't really covered in here.  And it's taken me about a year to figure out just the pieces that we need in the Drupal world in order to make an effective architecture for Kubernetes.  

So let's go over these again.  Deployments describe an application.  Not just what containers to use.  But the amount of copies of that application that are running called replicas.  Any health checks to make sure that we need to restart the application when necessary.  And where in the cluster to place those applications.  

Services are a consistent application endpoint.  They can be internal, external, or load balanced application also persistent volume claims called PVCs, not a pipe these are a request for persistent storage how much from where and how sharable it is.  

So that's a lot of objects that we're already talking about.  How do we actually get these into the cluster?  How do we deploy this?  

Actually if you have worked with Drupal 8 configuration before you already know how to do this everything in Kubernetes is defined as declarative YAML and pushed to the manager node this YAML can be patched, diffed or even replaced.  It's very much like git.  It's really, really nice.  When I saw this, I thought I never imagined there was this much DevOps in the entire galaxy although Hope probably gave me some commentary for that to be referenced.  

So we have this wonderful infrastructure.  Now, this is all shiny.  But where do we actually use this?  

(Chuckles).

>> TESS FLYNN:  So let's get into our -- let's go to Kube station we can always self-host which is a perfectly good option for Kubernetes and this might be the only option if you actually have an on-premises requirement or certainly legal obligations like HIPAA that requires your infrastructure to be in premises in a secure facility so you can do this an external and on existing hardware or if you can use an third party VPS service, you can take raw servers and create a Kubernetes cluster on a VPS service like DigitalOcean or Linode.  

There's a lot less lock-in with Kubernetes because there's no really reliance on special APIs.  Kubernetes is Open Source.  And one Kubernetes is pretty much like every other Kubernetes.  It's not really that different.  

This -- the thing is, though, if you're going to run Kubernetes yourself, you still have to worry about having staff to manage the cluster, to set up storage.  To make sure that the cluster is working correctly.  And that everything is secured.  This is very important because Kubernetes does not install as a secure by default configuration that's something that's very essential to know when you are running self-hosted Kubernetes.  

Storage is still tricky.  Because you could set up a traditionally managed NFS.  This becomes a single point of failure within your cluster but it still can be effective other options might be Ceph, rook or heketi that have native Kubernetes and are more fault tolerant so those are different options that you can look into.  

So when I did this once, I actually decided to set up Kubernetes over a weekend myself on Linode and it was a lot of resource just to run one site I was running three nodes at six gigabytes and it was a lot of resource and I'm a cheapskate, I just didn't want to do this and I didn't want to pay for the extra manager node either and I just thought, you know, there's got to be an easier way.  

And fortunately for us there is.  

Over the last two years managed Kubernetes has exploded as a service field.  It's like a virtual private server but for Kubernetes workloads.  Since you're not buying a single server you'll now by a cluster and config that to your needs typically you only pay for worker nodes instead of the manager node as well furthermore the storage configuration is typically baked in so you don't need to do it yourself.  

There's a number of different providers that you can find to do managed Kubernetes on your site, Amazon, Azure, DigitalOcean, Google GKE and a whole bunch of others there used to be a wonderful list of these around.  Last night I found out that it no longer exists.  So generally I can just tell you experiment.  Go out and find each -- go and research each one.  See which one of the pros and cons suits you best.  

Now, for us at TEN7 and for me personally we settled on DigitalOcean and the reason why is this is a worldwide Kubernetes provider Ty has a really straightforward rest based API but the most important part was the pricing was really, really obvious they literally will tell you how much a month your cluster will cost.  Upfront.  Before you allocate.  You really can't get that from Amazon, Azure, or GKE.  You basically have to wait for a bill for a lot of those.  That might have changed.  But that was the last time I looked at it.  Yeah, you just have to wait for a bill.  

All right.  So now we know where we're hosting this.  Now we have to actually start interacting with our cluster.  

So let's figure out what our architecture is going to be to host our Drupal sites.  Now, when I started out, I decided to use the smallest nodes available on DigitalOcean and create a three worker cluster of those notes.  And the reason why is because I wanted to start small and see if I needed to scale up later.  Also we're going to use TEN7's flight deck for containers mostly because we were already using them internally we were very familiar with them and if we needed to modify them at all to make it easier to work in Kubernetes we totally could do that it also helped that I wrote them and I was making the clusters.  So I'm cheating.  So first of all we start with a MySQL container.  And we're going to back it up by a persistent volume claim disk.  And that's going to be our database server in our cluster.  Then we're going to have three memcache instances that will cache any queries from MySQL.  We have one memcache instance per worker node so each one of those will be on their own independent hardware.  So far, so good, already.  Then we're going to have three web engines.  Web engine?  What the heck is this?  This is something that's a big different from Docker.  In Docker you only manage containers.  But in Kubernetes you manage pods, a pod can be one or more containers.  So our web engines are actually two different containers in a single pod one container for varnish and one container for Apache mod PHP we could also use varnish and use three different containers in the same pod.  Varnish, nginx and PHP, there was no advantage with Kubernetes architecture with FPM versus mod PHP and some arguments to Apache being slightly better in that case. 

Then we can connect all of this to the Internet all right so let's look at some actual code so here we have a deployment that's going to describe our application.  The first line that's going to be the API version.  Now, this API version is not the version of your application, it's not the version of the file it's not the version of Kubernetes you're working with.  It is the conversion of the API that this file is communicating to within Kubernetes.  

There's actually a page in -- buried in the documentation which could tell you which one you're supposed to use.  But I have to look it up every time myself.  

For deployments, the standard one is apps/v1 also there's the kind which is the kind of object that this is.  This is a deployment.  So we just have deployment.  

Next we have the name of the deployment.  In this case it's under the metadata section name is web so this deployment is called web.  We have three different replicas.  And we're going to also have a key value pair here that's going to be used by the service.  Hmmm, let's get to that later.  

Then we're going to have the image to use which is TEN7/flight-deck-web then down at the bottom we say open port 80 so that we can communicate to Apache and port 6081 to communicate to varnish.  All right so far so good now let's talk about that service.  API Version 1 kind service metadata equals web.  Wait a minute we have a deployment named web and a service named web, don't that conflict?  And the answer is no, it doesn't.  And the reason why it doesn't is because these are different object types.  So as long as they are different object types the names don't conflict.  

I actually tend to name my services in my -- and my deployments the same name every time just so I know which one is associated with each other without having to open the definition.  

And how do we know they are not associated it turns out it's the selector down here the selector app equals web corresponds to the selector up here app equals web.  In our deployment.  

And that's how the two are actually associated with each other.  

This service also is going to open port 80 and port 6081 and advertise that to the entire cluster we'll load balance that as well so that any time that traffic goes to our web engines, Kubernetes decides which one of those containers it's going to go to.  

So what else do we need to do?  We need to figure out where to put the database log-in in all of this.  Where do we put our configuration?  So in Kubernetes we can define something called a secret.  It's defined as its own object, not as part of container.  And it is mounted into the deployment as a disk.  

So here is a secret.  API Version 1 kind equals secret metadata name equals MySQL and afterwards we have a data section.  Now this data section is going to be defined as one or more items which are key value pairs.  Now, I tend to like naming the item name after a file name.  Because this is how it's actualized inside of the cluster.  

So we have Drupal-db-name.txt and user and pass.  And all of these must be base 64 encoding.  

Then we have our deployment so let's go ahead and get the secret in use.  

So we have our deployment, our web deployment, and way at the bottom we have a new volumes statement which says we're going to use the secret MySQL and we're going to mount that under the volume mounts section.  Same volume name at secrets/MySQL so now this container -- the container will be able to find the files within secrets/MySQL slash whatever these file names are in order to get that particular information.  Cool.  All right.  

What else do we need to do?  

So it seems like we got pretty much all of this licked right now, now all we have to do is make a shared disk for all of our web engine replicas so they all have a place to put sites default files, right?  It should be pretty easy.  Right?  Right?  

It's not easy.  

This is where I really had a hard time because one of the problems is that within a persistent volume claims in Kubernetes on managed hosting such as DigitalOcean, GKE and others, you can only have a single writer.  So you can't actually share it between multiple web pods.  This becomes a huge problem.  Now, you could orchestrate a way of directing traffic only to the first web pod and that's the writer and everyone else is just a copy that does only reading.  But that gets complex really, really quickly.  

So what we decided to do at TEN7 instead is just not use any attached storage for the web engines.  Instead we decided to use fly system or S3FS and then back that by an object store such as AWS.  DigitalOcean spaces or any other option that can work with S3 this way we no longer need a file system and the reason why this is actually important is because Drupal primarily manages static files.  Now, this is static files from a cloud architecture perspective.  We don't need block storage.  And the reason why we don't need block storage for Drupal sites is because from a cloud architecture perspective, block storage is meant for high bandwidth frequently changing performant data after a bunch of research to uncover what that gobbledygook is supposed to mean I found out the good example was the files which make up a MySQL database.  

So we're not storing that insights default files we don't even need to worry about that so we don't need block storage, we don't need attached storage for this we can use S3 and it will work.  

And the nice thing is that object stores are multi-writer by default.  We don't need to worry about multiple writers accessing the same endpoint because it is multi-writer out of the box.  

So what else do we need?  Memcache and MySQL don't need any Kubernetes load balancing in fact we want to directly access these without any load balancing because load balance can confuse these services.  

The only problem is that we do need predictable hostnames within Kubernetes.  Now, Kubernetes does not like giving you predictable hostnames by default.  That's intentional.  The reason why is to prevent you from building hard requirements within your containers.  

So one way around this is to use called a stateful set.  It's a special kind of deployment.  The host name is assigned deterministically rather than non-deterministically with a box of hexadecimal gobbledygook this allows you to access it -- directly to predict which container you want to talk to a stateful set kind of looks exactly like a deployment the only difference really is we'll have a service name field that's going to match whatever the service is after that everything else works exactly the same.  So this is a stateful set for our memcache so API Version apps/V1 so kind stateful set, name memcache.  Three replicas and then we have the memcached official container the Alpine version and then we expose the default memcache port of 11211.  

Okay.  We need to expose this stateful set with a service, as well, so we can make it known to Kubernetes so again API Version 1 kind service metadata name equals memcache and then the ports and of course our selector.  The important part is that with our stateful set we want to turn off Kubernetes load balancing and the way to do that is this weird line called cluster IP equals none.  And what this does is it tells Kubernetes, yes, this service exists.  Yes, please build DNS entries so people can actually access it but no don't use load balancing when people are accessing it instead.  Let people directly access the underlying containers instead.  

So we have one more spot left in our infrastructure.  We need HTTPS.  So the thing is that we don't want to run certificates all the way down our infrastructure into our web engine.  It would be far better if we could terminate SSL at the very front of our infrastructure.  

And what we can do for that is called an ingress.  

Now, an ingress is a Kubernetes edge load balancer.  The maps domains names to Kubernetes services and it can terminate SSL.  

So here is what the ingress definition looks like API apps/V1 kind ingress metadata my-cluster-ingress we have a bunch of annotations here which allow us to configure the ingress to fix rewrites and cores, set the file upload size.  And then afterwards, we have our host name.  So any time that someone requests examples.com from our cluster, we're going to actually -- we're going to look for a service called web.  At this particular path.  And access it through 6081 which is our varnish port for TLS we can have TLS hosts, example.com and here is the secret that contains our certificate chain.  

What's that look like?  So that ingress TLS secret provides the combined certificate chain and the private key so it looks like this API Version 1 it's actually a kind equals secret.  And it has two different items TLS.crt and key both are base 64 encoded.  That actually allows us to put a certificate within Kubernetes in order to actually SSL encrypt our traffic externally.  And terminate it internally from our cluster.  

But there's a problem.  And the problem is, well, what about let's encrypt?  

We don't want to pay for certificates because that just seems like a racket.  So how do we get let's encrypt in this?  Typically we'll use Certbot in order to actually go and request a new certificate and it does some configuration with Apache.  But we're not doing that.  We're using an ingress.  How does this work?  

Well, actually a lot of people have the same problem and there's a product out there called cert manager it's an Open Source certificate store and certificate provisioner for let's encrypt you can find it at cert-manager.io.  

So we can see that in the ingress it's going to allow us to do name-based resolution.  Like example.com or example2.com can go to different places within our cluster and that's fine.  

But how do we make sure that those different sites are separated from each other?  How we do that in TEN7 is we actually assign each individual site its own namespace within the cluster.  Named after the site URL.  

This allows consistent deployment and service names across every individual site, which makes it easier for us to manage.  

All right.  To me all of this put together was really just DevOps happiness it allowed me to do everything I needed to do to run a site.  And it was an effective infrastructure that was Kubernetes compliant without massive modifications to Drupal itself.  Basically I only needed to configure memcache I only needed to set up fly system and I was good.  

So now that we know our cluster is going to look like, how do we actually get it into DigitalOcean?  

So let's get into our action we're going to talk about the build chain.  So we need to deploy all of this YAML we were just writing how we can do this manually is we use the Kubernetes commandline application key control or Kube cuddle whatever makes you giggle there's five or six this kubectl apply and path to my object you can repeat this for every YAML file you want to deploy.  Now, if you need to do this with namespaces because you're using our model for multi-tenancy you can use the dash dash namespace parameter on cube control and apply it only to a tib namespace the namespace has to exist first and this link will tell you how to do that.  

So the problem is, though, we can combine all of these files to make it easier but we really need to remove humans from the process.  Humans can be tired, frazzled, fried, they can make mistakes and typos and that can cause us to have a very bad day we need to remove humans from this deployment process to make sure we can do this much more consistently with a lot less effort on our part so the thing is that almost every site that we work with is going to kind of the same.  We need a repeatable cluster clients tend to only differ in sizes long-ins and some features like memcache versus Redis so in order to make this less repeatable and reduce work on ourselves we'll bring in another tool called Ansible and you can see avoid deep hurting which tells you all about ansible we'll use the K8s module within this in order to deploy YAML how it works our DevOps person will define a bunch of group variances which has a cluster this goes into git repository GitLab CI picks that up and reads the playbook that's inside that read file and gives it over to ansible which will interact with the DigitalOcean API to provide the hardware and kubectl in order to actually send out all of our resources all of our YAML.  How do we authenticate this entire process?  For DigitalOcean we can create a access token and pass it to DigitalOcean API as an HTTP header what about creating the cluster itself?  Well, a lot of this is going to actually be done with post requests with JSON but it's a lot of work to figure that out and fortunately I've done all the work for you use the TEN7.  digitalocean role on ansible galaxy this handles scaling node pools and a bunch of more options here is an example playbook which actually provisions a cluster so we have our ansible playbook DigitalOcean API token DigitalOcean clusters and here is what our cluster looks like it's going to be called my K8s cluster and the SF 02 region it has two node pools one will have 1 VCPU 2 gigabyte 3 servers and another server for 1 VCPU 2 gigabyte just one copy for the database and that's it that sets up the entire cluster for us and we don't have to do anything else and this can go right into our repos. 

So the problem, though, is we need to authorize kubectl and the thing is that when you create a cluster within DigitalOcean you'll get a kube config file that file only has a limited amount of life span those files are regularly rerolled by DigitalOcean as a security procedure to make sure we get a fresh kubectl every time we use another role called TEN7.  digitalocean underscore kube config that interacts with the API via the access token to get our authentication, our kubectl to talk to the cluster.  

Here is what a playbook that uses that looks like.  And it's just DigitalOcean kube config the name of the cluster and where to put the file and then we call the role.  That's it that's the whole playbook.  All right what about the Kubernetes YAML itself?  We actually have another role that we use to deploy all of our sites called TEN7.flightdeck cluster it's going to create secrets deployments and services everything you need to run a flight deck Drupal flavored cluster in DigitalOcean in fact it should support virtually any kind of Kubernetes provider.  Even self-hosted.  

You can find plenty of examples on GitHub.com at github.com/TEN7 -- 

All right, finally after all of this time we can stand over -- oh we can stand over our enemy you've failed Lord hostedness for I am a deaf -- I'm a DevOps like my mentor before me.  

I should probably get the site on the cluster right now.  Yeah, let's do that.  

We're all clear, K8s let's see deploy this thing and go home.  We're already home.  Well, let's keep going.  

(Chuckles).

>> TESS FLYNN:  Usually we're used to updating a site in place.  The server remains permanent and fixed.  And we just swap out the site code when we do an update.  

This is usually used for shared hosting and some VPSs.  Typically we can deploy a site using this using SFTP, git deploy and some CI systems do this.  

The problem is on Kubernetes we have to deploy to each replica.  And that's not simple.  Sometimes we'll have three replicas which mean we have to do three different deploys.  And those might be incongruent while we're deploying each one and that can be a problem.  

We could use NFS to host code so each one is updated simultaneously but that just ruins our performance in other ways.  

So the thing is that this is just not how the K8s works.  What do we do instead?  Well, the more I thought about it, the more I realized that the site is the application.  It should be the container.  

So what we should do is not use a container that just has Apache PHP on it.  But it should have Apache PHP, and all -- and all of our site code built into it.  How does this work.  So we have our developers.  Developer is going to push the site code to a git repo.  GitLab CI is going to pick up those changes to our site and inside of that repo is a Dockerfile that Dockerfile is going to be used to build our Drupal 8 site so it's going to run composer and MPM run, and anything else that it needs to in order to actually build the Drupal site code.  

Afterwards we're going to take that and we're going to get an image from it, a Docker image.  In order to make sure that that code is going to be preserved accessible to Kubernetes we upload it to a private container registry.  

That registry just hosts the container mechanism.  It's nothing particularly fancy.  After that, kubectl can actually read from the registry to pull that custom image down and then it gets deployed to the cluster.  

Where can I find an example Dockerfile which builds a Drupal site you can find this at github.com/TEN7/flight-deck-Drupal.  

This creates the necessary directories and also builds Drupal itself within the container.  You can -- it also customizes the entry point so every time the container starts up it writes the settings.PHP file so they all have access to the correct database.  

What about that private registry?  Well, private registry just stores built, site-specific images.  And Kubernetes needs an auth token that's passed in secrets in order to access it that's a lot to cover and we don't really have enough time to do it there's a wonderful tutorial on docs.Docker.com in order how to set up a container registry.  Also a number of different providers such as DigitalOcean itself has a container registry which is currently in beta and a whole bunch of other providers which actually allow you to have private registries as a service rather than self-host it.  

So when we actually have our container custom built we need to modify our deployment.  So now our deployment needs to actually have reference our custom registry.  So my registry's port and then the container's name that has our site and whatever version it is.  We actually don't use numeric versions for our containers we use the GitHub short commit ID instead because it's a lot more consistent in order for Kubernetes to understand which item to deploy.  And the reason why is the tag names have to be different each time otherwise Kubernetes will not redeploy it, even if the container image has been overwritten.  So it's important to know that the tag names are very, very important.  

Also we have the my registry key which is going to be a secret which is going to have our authentication token for our private registry.  

So the last thing that we need to worry about is how do we control the update?  When we deploy this out, we don't want -- we want to minimize downtime.  So we have three replicas that have three different web containers.  Which are all going to get updated.  If we do them all at once we have downtime.  That might not be so bad.  But is there a way that Kubernetes can fix this for us, too?  And the answer is yes there's something called a rolling update.  It replaces running containers with the new image.  And it does so sequentially.  In this -- and this reduces outages.  Flight deck cluster does this out of the box and by default for any web container.  So here is how it works.  We're going to modify our deployment for our web deployment.  And we're going to say there's a new strategy, type, rolling update and in that rolling update we're going to say allow one additional container in addition to the three ones that we already have.  Make sure to only update one at a time.  And then wait three minimum -- five minimum seconds after updating that container to make sure that before moving on to the next one to update.  This way it reduces downtime.  Every update is done sequentially.  The best part about this is that the load balancing within Kubernetes is aware of the deployment process so if anything goes wrong with the first container deploying, it gets stuck.  But your site never goes down because traffic is now going to the other two containers.  Now, that was a lot that we have to further and if you want to learn more there's a wonderful book called Kubernetesbyexample.com it's free and online and tells you everything you need to know about Kubernetes than I covered here there's an interactive tutorial on Kubernetes.io and also we have our own class called hosting Drupal on Kubernetes on YouTube that also has all of the lab material and slides online for you to use and learn how to interact with Kubernetes directly. 

I want to take a moment here to tell a special thanks for having this event -- for having -- for having me here today at this event even though I'm in my own house for TEN7 for sponsoring -- also the wonderful patrons and my maker efforts and my fiction right you can find this at -- you can find this presentation at GitHub.io.

>> DWAYNE McDANIEL:  Thank you I'll unmute everyone at once and give a big round of applause.  

(Applause).

>> DWAYNE McDANIEL:  Back on mute you all go if you would like to ask a question, please raise your hand.  And I will unmute you.  And we will -- I'll unmute you, Tess.  So please raise your hand and I'll mute you and we can have some conversation about this and again, please feel free to use that chat function to type out your question.  

I thought I saw someone raise their hand maybe they are just raising the roof.  

(Chuckles) 

>> DWAYNE McDANIEL:  Any questions out there?  If you're not familiar with Zoom -- there we go.  Benji has a question so we'll unmute you and lower your hand you're on.

>> Hi this is sort of a silly question but I notice your slides are on GitHub GitHub.io and of course you're the DevOps expert so I wonder if you have a DevOps process for your slides.

>> TESS FLYNN:  The slides are written in reveal.js I tried to use git flow with them but I got lazy last night and pushed directly to master but oh well but it is managed in git so I can do that.  Also all of the slides are done with Open Source technologies only.  So all of the graphics are actually made in either Gimp or Inkscape which is a lot of what the vector graphics are and the source files -- the source SVG files for those graphics are actually within the repo, as well so if you want to reuse them, go right ahead.  They are -- I think I put them under Creative Commons so go ahead and reuse them if you need to.

>> I do my slides using Markdown and then I convert them to reveal.js with pan doc and all in GitLab CI but I'm much lazier about my graphics than you are.

>> TESS FLYNN:  Yeah, I mean, I like doing the graphics.  That's kind of half of the fun of doing these for me I guess.  

(Chuckles).

>> TESS FLYNN:  Yeah, I don't really build any particular automation around the deployment.  I just code it directly in HTML.  Because there's really not that much additional work in setting up the auto -- and settings up the automation just didn't seem like it was necessary plus it allows me to run this locally if I need to without having to run a build process.  

>> DWAYNE McDANIEL:  All right.  Thanks, Benji, we have a few more questions coming in over on the chat.  So just curious why the choice of cache per node or per pod and not a separate cache architecture whether that's also hosted in Kubernetes or a global/mass cache like global flare cloud front Akamai.

>> TESS FLYNN:  Mostly because we wanted to simplify the architecture and make sure if you want to add additional components outside of that, you certainly could we do have a couple of clients at TEN7 which use Cloudflare and use that front side cache and then it gets moved over to Kubernetes.  The idea behind this presentation was to present the simplest amount of architecture possible that allows you to build on top of it rather than a very complicated architecture that has dictated components.

>> DWAYNE McDANIEL:  Thank you very much.  That was from Adam Arrowwood.  Howard I skipped your question you asked first I apologize.  Could you talk a little bit about performance compared with a single larger server?  

>> TESS FLYNN:  So the biggest problem with this infrastructure that we're running into is the performance of the object store.  Surprisingly, we found out that we could run four to six sites that have our traffic on it per cluster and we have them running I think 2 VCPUs per node 4 gigabytes memory and we're not anywhere near capacity on those systems so we have plenty of CPU performance, memory performance.  The biggest bottleneck is going to be the object store.  You have to make sure that the object store has a -- it has enough bandwidth and enough network location in order to access it quickly.  Because imagine regeneration will really kill your performance really quickly.  So that is the biggest challenge.  

We are looking into maybe using min  IO for a lot of that and an external CDN instead of using some other solution like just a direct space that has access and the reason why is because that will reduce network hops and probably make image derivative generation a lot faster.  

>> DWAYNE McDANIEL:  Thank you.  And reminder, feel free to use that raise your hand button if you want to get on the air and talk here.  And I've got another question from Howard.  How do rolling updates work with db updates?  Won't code be out of sync with the database?  

>> TESS FLYNN:  It can be for a few seconds.  But in Drupal 8 a lot of the traffic is going to be anonymous and it will hit the caches first so that's not going to typically be a big problem.  So it doesn't completely solve the problem of outages.  But it does reduce several major cases of outages that are due to deployments.  And typically the deployments take about four to five minutes even for a large site so it's not that long an amount of time to be out if it has to be out.  

>> DWAYNE McDANIEL:  Awesome.  Here is another one from Pushpinder if I mispronounced your name do you see any issues with caching -- I'm sorry; do you see any issues with caching/configuration-import when you use this architecture in case if one pod goes down during deployment, another pod comes up?  

>> TESS FLYNN:  So typically what happens is that when we do a deployment, if the first pod that's doing the deployment fails, it just gets stuck in a retry state and it doesn't actually serve any traffic when it's in the deploying phase.  So Kubernetes takes it out of the deck entirely and only relies on the running existing pods.  This happens a lot of times with our deployments if we have some problem or if some configuration doesn't work or we just make a typo somewhere and something gets screwed up.  And that stuck deployment is actually timed out after five minutes and that's when it pings my phone to let me know that something went wrong.  

So it's actually worked out pretty well for us whenever we've had a deployment that didn't work out, the site never really goes down it's only in rare cases where the site actually continues to keep working and then something later fails on it.  

We've also had problems sometimes where an individual pod will have a problem.  And the best solution isn't to try to troubleshoot the pod but rather just delete it and then have Kubernetes make a new one.  And that typically also solves the problems, too.  

>> DWAYNE McDANIEL:  Right on.  Thank you very much.  We got a thank you as well in there.  From the asker.  

Any other questions out there in the world?  While we have Tess in the hot seat.  

Here we go.  This is from Adam Arrowwood if the content of a node is changed, how do you handle cache flushing of that node?  Or do you just wait for the TTL to expire?  

>> TESS FLYNN:  Typically we actually set our TTLs fairly low.  That is one of the limitations of this simpler architecture.  We have been looking at ways of actually making the varnish into its own tier within Kubernetes itself.  That we can actually use much more complicated tagging architecture to individually flush particular pages.  

We started looking into that.  We haven't really deployed that yet.  We're still in early stages.  

But for the last few months -- last year, two years almost at this point, we have been running this with low TTLs and it's been fine.  

>> DWAYNE McDANIEL:  Awesome.  All right.  So I'll put one more time out there, are there any other questions or comments?  Speak now or you'll have to contact Tess offline or I guess online somewhere.  

It looks like that's wrapping it up.  

So thank you very much again, Tess.  I'm going to do that one more time I know I caught everyone by surprise last time so everybody get ready give a big round of applause or tap your mic if you're in the office or whatnot so unmute all and thank you very much.  Tess.  

(Applause).

>> Whoo.

>> DWAYNE McDANIEL:  I think that might have been louder if the Zoom didn't have the auto canceling function.  Trying not to have everyone touch over each other.  So I'm going to unmute and I'm going to turn the ability to unmute yourself back on for the time being.  And stop the recording.  

All right, great job, Tess, that was awesome.

>> TESS FLYNN:  Thank you.