{"id":284,"date":"2021-08-31T14:40:23","date_gmt":"2021-08-31T14:40:23","guid":{"rendered":"https:\/\/fde.cat\/?p=284"},"modified":"2021-08-31T14:40:23","modified_gmt":"2021-08-31T14:40:23","slug":"hadoop-hbase-on-kubernetes-and-public-cloud-part-i","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/hadoop-hbase-on-kubernetes-and-public-cloud-part-i\/","title":{"rendered":"Hadoop\/HBase on Kubernetes and Public Cloud (Part I)"},"content":{"rendered":"

Authors: Dhiraj Hegde, Ashutosh Parekh, and Prashant\u00a0Murthy<\/em><\/p>\n

At Salesforce, we run a large number of HBase and HDFS clusters in our own data centers. More recently, we have started deploying our clusters on Public Cloud infrastructure<\/a> to take advantage of the on-demand scalability available there. As part of this foray onto the public cloud, we wanted to fundamentally rethink how we deployed and managed our HBase clusters. This post outlines how we ended up using Kubernetes and the challenges we had to overcome in this transition.<\/p>\n

\"\"<\/figure>\n

For many years Salesforce has been running HBase on a static set of bare metal hosts in our data centers. Operating system updates on these hosts were managed by Puppet, and HBase\/Hadoop deployments were managed using Ambari<\/a>. These tools were geared towards mutable infrastructure where you modified binaries and configuration \u201cin place\u201d on the hosts. Such upgrade processes can result in partial changes to hosts when failures occur, resulting in a more complicated recovery process. While robust idempotent deployment mechanisms can overcome such issues, the more common issue seen in such environments is a temptation for engineers to apply manual fixes for urgent issues. These fixes are often forgotten, resulting in lingering config\u00a0drifts.<\/p>\n

In public cloud, Virtual Machines (VMs) and Containers provided us with the opportunity to embrace a more immutable form of deployment where the set of binaries and configuration are part of the VM or container image itself. If one tried to modify the image with a local change, it would be reset to the original image the next time the Container or VM is restarted. This kind of immutable environment enforces good engineering discipline.<\/p>\n

In the virtualized environment of public cloud, we also found resource usage advantages like<\/p>\n

    \n
  1. Software-driven infrastructure deployment that could be elastically adjusted based on\u00a0usage.<\/li>\n
  2. Right-sizing of virtualized hosts for specific needs in terms of CPU, memory, and network bandwidth.<\/li>\n<\/ol>\n

    VMs vs Containers<\/h3>\n

    We had to address the question of whether to deploy HBase\/HDFS directly on VMs or within containers. At first glance, there were several factors that seemed to favor\u00a0VMs:<\/p>\n

      \n
    1. The dominant container management system Kubernetes started with stateless application management with subsequent enhancements to work with stateful applications like DBs being added as an afterthought. This did not inspire confidence.<\/li>\n
    2. Containers brought with them their own approach to networking, which seemed to add unnecessary complexity.<\/li>\n
    3. For an application that has been running primarily on bare metal hosts, containers seemed to suggest additional OS level indirection, whereas using a VM provided an environment more similar to the existing\u00a0one.<\/li>\n
    4. On VMs, one could perhaps reuse existing (mutable) deployment tools for bare metal and gradually evolve to a truly immutable approach.<\/li>\n<\/ol>\n

      However, as we dug deeper we found some of the pros and cons\u00a0changed.<\/p>\n

        \n
      1. While Kubernetes added support for stateful applications much later, the additions were pretty well thought out. In addition, we found Kubernetes to be very extensible. Any limitations in the features could be overcome by making our own enhancements on top of Kubernetes APIs.<\/li>\n
      2. Salesforce embraced an immutable approach for OS updates across all teams. Layering a mutating application deployment approach on top of this OS layer (with a plan of gradually making it immutable) made little sense. The binaries would have to be reinstalled every time the OS was\u00a0updated.<\/li>\n
      3. Containers are very lightweight constructs (essentially a jailed process), and the OS level performance implications in using them turned out to be negligible.<\/li>\n
      4. Container management systems might be prescriptive about how networking among containers works (as mentioned above in cons), but with Kubernetes, all major cloud providers had built plugins that made communication between containers and the outside world as seamless as that between VMs and the outside\u00a0world.<\/li>\n
      5. Kubernetes provides a powerful standard mechanism for application deployment and management across cloud providers<\/em><\/strong>.<\/li>\n<\/ol>\n

        The last point is one of the more significant insights. Cloud infra deployment tools like Terraform supported management of disks and VMs across different cloud providers. However, each cloud provider resulted in very different manifests, as there was very little abstraction and reusability in the manifests. Kubernetes, with its opinionated approach to managing compute, storage, and networking of containers, provided a much more consistent deployment and management interface across cloud providers.<\/p>\n

        Kubernetes and Stateful applications<\/h3>\n

        We will cover some Kubernetes concepts here that will help in understanding the rest of the\u00a0blog.<\/p>\n

        Pods as Containers<\/h3>\n

        Kubernetes manages deployment of containers, monitors their health, and restarts them in case of application failure or relocates them to new hosts in case of host failures. Hosts in Kubernetes are called Nodes<\/strong>. In Kubernetes, a container is wrapped inside a construct called a Pod<\/strong>. A Pod allows multiple containers that need to be co-located on the same host to be deployed together. Typically, most applications have multiple supporting processes (log forwarders, cert\/key refreshers, etc.) so a Pod makes it convenient to wrap these containerized processes into a single deployable unit. Each Pod has a unique IP address associated with it, which basically allows it to behave like a pseudo host. All containers within a Pod share that IP address and can also use the loopback address (127.0.0.1) to communicate among themselves. This gives them the same environment that processes in a single host have. The main difference is that each container within the Pod has a distinct view of its file system and of the processes running inside it. The containers can still share storage within a Pod, but they would have to explicitly mount the same volume in each container of the Pod to do so.<\/p>\n

        Since creating each Pod individually would be laborious for users, higher level constructs called Workload Resources<\/strong> are provided<\/a>, which define a template for a Pod and the number of instances of the Pod needed. The rest is left to the controller of that workload resource, which automatically creates and manages the Pods in Kubernetes cluster.<\/p>\n

        Persistent Volumes (PVs) for\u00a0Storage<\/h3>\n

        Kubernetes provides a simple abstraction called Persistent Volume (PV)<\/em><\/strong> to represent storage volumes. This storage can be local volumes on the nodes or network attached volumes like EBS volumes in AWS. For the purposes of this blog, we are only considering network attached volumes. When a Pod needs a volume, it specifies a PV Claim (PVC)<\/em><\/strong> in its Pod manifest that describes the type and size of storage desired. Kubernetes responds by creating the volume and represents it with a PV instance. The PV and PVC are mapped to each other in a 1-to-1 relationship called a binding. When the Pod is created on a host, the PVs bound to its PVCs are mounted on that node as a volume and hence made available to the containers. Containers within that Pod can then mount that volume into their file systems.<\/p>\n

        A PV continues to be retained by Kubernetes as long as the PVC is present. The PV mounts wherever the PVC goes. In the diagram below, you can see that a Pod can be removed in one node and then later recreated in another node, and the PV will follow it as long as the Pod manifest refers to the same PVC. To recycle the PV, both the Pod and the PVC have to be\u00a0deleted.<\/p>\n

        \"\"<\/figure>\n

        Pod and\u00a0State<\/h3>\n

        Pods in Kubernetes are inherently stateless; when they are updated or moved from one node to another, they are destroyed and recreated. Any persistent state should be kept in attached PVs. However, early workload resources in Kubernetes only created Pods with temporary and randomized names (like http-nmx8). So when a Pod is deleted and recreated, it gets a new name. This worked great for applications whose instances were totally stateless and could be placed behind a load balancer (called a Service in Kubernetes) as shown\u00a0below.<\/p>\n

        \"\"<\/figure>\n

        The clients only need to know the virtual hostname (or VIP) of the load balancer; there is never a need to know the actual Pod\u2019s hostname. But when you look at HDFS and HBase, the above model does not apply. In these applications, each application instance (Pod) needs to be individually addressable by the clients directly. The client typically contacts the specific Pod expecting to find some specific data in its PV to read or modify. If the Pod changes its identity, then the client is forced to error out or refresh any cached metadata, which is disruptive to the system. In the HBase architecture diagram below you can see there are a number of components that are intertwined by this pattern of communication among them. Each arrow indicates a specific instance of an application talking to another instance of another application with a well-defined hostname.<\/p>\n

        \"\"<\/figure>\n

        For applications like HBase (and others like Cassandra, Redis etc.) Kubernetes eventually introduced a workload resource called StatefulSets<\/a>. This creates Pods with a well-defined name (and hostname). For example if you define a StatefulSet named zookeeper with three instances, then the Pods get created with the names zookeeper-0, zookeeper-1 and zookeeper-2. The number that keeps increasing in the names is called the ordinal number. If PVs are required, then PVCs with a similar ordinal based naming convention are specified within each of these Pod definitions (for example Pod zookeeper-0 can have a PVC named zk-data-0). Each PVC is bound to a different PV and that PV will be mounted wherever the uniquely named Pod lands. So now we not only have state (a PV) but also state that is permanently associated to a Pod with a fixed hostname. The fixed hostname allows all the Pod\u2019s clients to cache information about it and locate the state associated with\u00a0it.<\/p>\n

        As you might have guessed from the HBase diagram above, all key components were deployed as StatefulSets in our clusters. The components are tabulated below. They are classified as master<\/em><\/strong> if they are meant for coordination, management and metadata components. The worker<\/em><\/strong> components are the ones carrying out the actual data processing.<\/p>\n

        \"\"<\/figure>\n

        Availability Zones\u00a0(AZs)<\/h3>\n

        In public cloud one can choose to run replicas of the components across fault domains to limit the impact of catastrophic failure in one domain. This usually translates to choosing to run your replicas across regions (geographically widely dispersed locations) or across availability zones (AZs), which are located within a region, but perhaps in separate buildings. Typically, latency between regions is too great for spreading replicas across them, so AZs with their low latency are the best choice for a database like HBase. We designed the component to run across three\u00a0AZs.<\/p>\n

        Spreading Across\u00a0AZs<\/h3>\n

        Public cloud Kubernetes nodes have labels that identify the AZ and the region in which the node exists. This is leveraged while scheduling Pods on them. Two mechanisms were used in Kubernetes to guide the scheduling of\u00a0Pods:<\/p>\n