Minikube Workflows and Tips

Overview

If you ever used docker, You must have heard people raving about Kubernetes, but getting Kubernetes up and running locally was hard. Surely it was missing the great local development platform.

Minikube is the easiest way to run Kubernetes locally. It runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.

Since Minikube runs locally instead of on a cloud provider, certain provider-specific features like LoadBalancers and PersistentVolumes will not work out-of-the-box. However, you can use NodePort, HostPath PersisentVolumes and several add-ons like DNS, dashboard, CNI, etc. to test your apps locally before pushing to production-grade cluster.

Installation

Minikube is VM provider agnostic and supports several drivers like VMware, Virtualbox, xhype, Kvm etc. You will also need kubectl to interact with cluster.

Installing Minikube is pretty easy since it’s available as a single binary. At the time of writing this article 0.17.1 is the latest version, to install -

$ export MINIKUBE_VERSION=0.17.1

$ curl -Lo minikube https://storage.googleapis.com/minikube/releases/$MINIKUBE_VERSION/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

Please visit this page for the latest release of Minikube

In order to validate our deployment and check that everything went well, let’s start the local Kubernetes cluster:

$ minikube start
  Starting local Kubernetes cluster...

Let’s start by first checking how many nodes are up:

=> kubectl get nodes
NAME       STATUS    AGE
minikube   Ready     3d

You can checkout other awesome blogs regarding how to deploy your applications on newly created cluster.

Architecture

Minikube is built on top of Docker’s libmachine, and leverages the driver model to create, manage and interact with locally-run virtual machines.

Inspired from RedSpread’s localkube, Minikube spins up a single-process Kubernetes cluster inside a VM. Localkube bundles etcd, DNS, the Kubelet and all the Kubernetes master components into a single Go binary, and runs them all via separate go-routines.

Limitations

Though community is working hard to make Minikube feature rich, it has certain missing features like -

• LoadBalancer
• PersistentVolume
• Ingress

Helping Tips & Gotchas

Sharing docker daemon

Minikube contains a built-in Docker daemon for running containers. It is recommended to reuse the Docker daemon which Minikube uses for pure Docker use-cases as well. By using the same docker daemon as Minikube, you can speed up your local experiments. To share daemon in your shell:

$ eval $(minikube docker-env)
$ docker ps

Now when you do Docker builds, the images you build will be available to Minikube.

Xhype Driver

if you are on Mac, it’s better to use [xhype] hypervisor for speed and less momory usage.

$ minikube start --vm-driver=xhyve

Or set it permanently

$ minikube config set vm-driver=xhyve

Addons

Minikube aims to advance features via several add-ons, like ingress, registry credentials, DNS, dashboard , heapster etc.

To enable an addon, run - minikube addons enable

You can also edit the addon’s config (YAML) file and apply against cluster to fulfill your usecase.

Switching cluster version

Minikube have relativelly unknown and very useful feature to switch cluster version to smoke test your app. To start minikube with a cluster version -

$ minikube start --kubernetes-version=v1.5.2

Using private images

Minikube is that it won’t always have credentials to pull from private container registries. You should use ImagePullSecrets in PodSpec as defined on offical Image page.

Port forwarding

When you are working locally, you may want to access cluster services using cli tools to quickly debug things. For example you can expose redis using

=> kubectl port-forward redis-master :6379:6379

Now you can simply use cli clients like ( redis-cli or psql, mongo) with the services running inside cluster. Forwarding a port with kubectl is fairly easy, however, it only works with single Pods and not with Services. Pretty neat.

Hot reloads via host mounts

It would be pretty neat if your code is reloaded when you change in editor, rather than building the image, and deploying again. On OSX, All you have to do is create a host mount to mount your local directory into your container like :

volumeMounts:
- name: reload
  mountPath: /app
volumes:
  - name: reload
  hostPath:
path: /path/to/app/

Also enable your webserver to listen to file changes ( like --reload flag for gunicorn ). Unfortunately, host folder sharing is not implemented on Linux yet.

Conclusion

I hope this post helps people who are struggling setting up their own Kubernetes cluster and getting them quickly started. Here we got a single node Kubernetes cluster running locally, for purposes of learning, development and testing, and will be using Minikube in next article to set up applications. Thanks for reading through this tutorial! Let me know if you found it useful in the comments ;-)

Advisory Locks in Postgresql

PostgreSQL provides various lock modes to control concurrent access to data in tables. Advisory locks provide a convenient way to obtain a lock from PostgreSQL that is completely application enforced, and will not block writes to the table.

Imagine you have a scheduled task that runs in the background, mutates the database and sends information off to another 3rd party service. We can use PostgreSQL Advisory Locks to guarantee that the program cannot cause any unexpected behavior if ran multiple times concurrently. Concurrency is fun, but surprise concurrency can be brutal! In our case it’s beneficial to have a guarantee that the program in question can never run concurrently.

Application Enforced Locks

Throughout this post we will explore Postgres advisory locks, which are application enforced database locks. Advisory locks can be acquired at the session level and at the transaction level and release as expected when a session ends or a transaction completes. So what does it mean to have an application enforced database lock? Essentially, when your process starts up, you can acquire a lock through Postgres and then release it when the program exits. In this way, we have a guarantee that the program cannot be running concurrently, as it will not be able to acquire the lock at startup, in which case you can just exit.

The benefit of this is that the tables are never actually locked for writing, so the main application can behave as normal and users will never notice anything is happening in the background.

Here is the syntax for obtaining the lock:

SELECT pg_try_advisory_lock(1);

Now, there’s a few things happening in that statement so lets examine it.

• pg_try_advisory_lock(key); is the Postgres function that will try to obtain the lock. If the lock is successfully obtained, Postgres will return 't', otherwise 'f' if you failed you obtain the lock.

• pg_try_advisory_lock(key) takes an argument which will become the identifier for the lock. So, for example, if you were to pass in 1, and another session tried to call SELECT pg_try_advisory_lock(1) it would return 'f', however the other session could obtain SELECT pg_try_advisory_lock(2).

• pg_try_advisory_lock(key) will not wait until it can obtain the lock, it will return immediately with 't' or 'f'.

Implementation

So how would we go about using this in Ruby?

def obtained_lock? connection.select_value(‘select pg_try_advisory_lock(1);’) == ‘t’ end

We can grab our ActiveRecord connection and call #select_value in order get back a 't' or 'f' value. A simple equality check let’s us know whether or not we have obtained the lock, and if we haven’t we can choose to bail and exit the program.

class LockObtainer
  def lock_it_up
    exclusive do
      # do important stuff here
    end
  end

  private

  def exclusive
    if obtained_lock?
      begin
        yield
      ensure
        release_lock
      end
    end
  end

  def obtained_lock?
    connection.select_value('select pg_try_advisory_lock(1);') == 't'
  end

  def release_lock
    connection.execute 'select pg_advisory_unlock(1);'
  end

  def connection
    ActiveRecord::Base.connection
  end
end

Additional Information

There are a few interesting things about Advisory Locks:

• They are reference counted, so you can obtain a lock N times, but must release it N times for another process to acquire it.

• Advisory Locks can be acquired at the session level or the transaction level, meaning if acquired within a transaction, an Advisory Lock will be automatically released for you. Outside of a transaction, you must manually release the lock, or end your session with PostgreSQL.

• Calling SELECT pg_advisory_unlock_all() will unlock all advisory locks currently held by you in your session.

• Calling SELECT pg_advisory_unlock(n) will release one lock reference for the id n.

Effective Rails Deployment

Capistrano has been an excellent tool for years to deploy simple or multi-stage complex applications to remote servers. However the rise of PASS like Heroku, Dokku, I got in mind to use some automatic orchestration framework.

I looked over Chef, Puppet, SaltStack and Ansible on a weekend and decided to go with ansible since it looked easiest to get hands dirty and have great documentation. After some initial hicups I was able to deploy some of our inhouse applications on EC2 and introduced it to my coworkers at codecrux.

Since then, we use ansible to configure and manage remote server, and also configure our local development environment running on top of Virtualbox with vagrant.

Ansible uses YAML for configuration and for actual commands it has less flexibility compared to capistrano which uses Ruby DSL, but at the same time helps you to keep your playbooks simple. ansible also extremely good with orchestration and rolling deployments. We were able to keep a homogenous command set and duplicate most of Capistrano’s features in very small amount of code.

Let’s take an example to deploy lobster.rs. Here is how we did it -

Configuration

First of all you need to define required variables in a yaml file for ansible to pick them up.

  app_name: lobsters
  rails_env: production

  git_url: git@github.com:jcs/lobsters.git
  git_version: master

  app_path: '/'
  shared_path: '/shared'
  releases_path: '/releases'
  current_release_path: '/current'
  app_public_path: "/public"
  app_config_path: "/config"
  app_temp_path: "/tmp"
  app_logs_path: "/log"

  keep_releases: 5

You should also define your remote host in ansible inventory/host file like

[production]
lobsters.dev   ## replace with your domain name

Playbook

Now when we have configuration we can create the actual playbook for doing capistrano-style deployments. Create deploy.yml file:

---
- hosts: all
  tasks:
    - set_fact: this_release_ts=
    - set_fact: this_release_path=/

    - debug: msg='New release path '

    - name: Create new release dir
      file: path= state=directory

    - name: Update code
      git: repo= dest= version= accept_hostkey=yes
      register: git

    - debug: msg='Updated repo from  to '

    - name: Symlink shared files
      file: src=/ dest=/ state=link force=yes
      with_items:
        - config/database.yml
        - config/secrets.yml
        - config/unicorn.rb
        - log
        - tmp
        - vendor/bundle

    - name: Install bundle
      command: 'bundle install --deployment --without="development test"'
      args:
        chdir: ''

    - name: Precompile assets
      command: rake assets:precompile chdir=
      environment:
        RAILS_ENV: ''

    - name: Migrate database
      command: rake db:migrate chdir=
      environment:
        RAILS_ENV: ''

    - name: Symlink new release
      file: src= dest= state=link force=yes

    - name: Restart unicorn
      command: sudo restart 

    - name: Cleanup
      shell: "ls -1t |tail -n +|xargs rm -rf"
      args:
        chdir: ''

Deploy

Once everything is set up, you can run the following command to deploy:

ansible-playbook -u railsbox -i production deploy.yml