Thoughts on the journey towards containers

I spent a couple of days this week at ContainerSched 2016 where I took part in a panel discussion, sharing insights from our journey into containerisation. All the sessions are available to watch online.

The event wrapped up with a cracking keynote by Adrian Colyer entitled “Making sense of it all” in which he presented his perspective on where the world of containerisation is today, and where it’s headed, supported by evidence gleaned through direct discussions with many movers and shakers in the industry. You should definitely go check out the recording!

Having spent more time talking about containers in two days than the last two months, I thought I’d share some thoughts based on takeaways from the conference and my own experience, in case it helps others undertaking a similar journey. There’s a TL;DR at the end if you need it 🙂

Containers are in production

A common concern about using containers is that they’re simply not ready for mainstream production use yet – they’re too new, too immature, too insecure etc. That’s simply not the case. An O’Reilly survey carried out in May last year found 40% of respondents were using containers in production, with more than half planning to enter production within 6-12 months.

A similar survey by ClusterHQ around the same time found similar results, with 38% of respondents currently in production. Their 2016 survey (due for release shortly) shows this figure to have climbed to 76%, supporting last year’s predictions (see Adrian’s talk for more details).

So containers are no longer confined to dev/QA environments – they’re very much out there in the wild.

Beware the concept count

For many (most?) people, the first encounter they have with containers will be via Docker. When you read/watch the various introductory Docker tutorials out there, it can seem pretty simple to spin one up. There is a huge difference between running a container on your laptop and deploying a containerised application stack to production however.

Once you start down the containerisation path, the concepts come thick and fast, with an ever-increasing number of concerns you have to investigate, learn, make judgments and decisions about, and build confidence in (both within your team and the wider organisation). You need solutions to networking, security, service discovery, storage, logging, monitoring and so on.

Don’t underestimate the learning curve. If you, your team or your organisation shies away from learning new things, or has a tight deadline, chances are it’s not the right time for you to explore containers. If you don’t have the problems that containers solve, then it’s definitely not the right time.

Be pragmatic

As alluded to above, the Docker/container ecosystem is huge, with new tools and approaches emerging every day. It can be tempting to feel like you have to learn everything.

The reality is you don’t need everything. As exciting and shiny as a solution may seem, do you actually have the problem it’s trying to solve? A funky auto-scaling setup may sound perfect, but if your fluctuations are mild and predictable enough, then maybe a manual approach would suffice for now, leaving you with one less concept to care about?

Just because there are lots of people choosing Kubernetes, or extolling the virtues of Weave, or advocating Project Calico doesn’t mean that each of those is suitable or essential for your circumstances, especially if your environment is relatively simple rather than involving 1000s of containers across dozens of hosts.

Speaking for myself, after surveying the various options, I settled on Rancher for our particular orchestration needs – it’s pretty simple to setup, with a lot of stuff available out of the box (overlay network, service discovery, load balancing, clean UI). Sure there are lots of capabilities it doesn’t have that you can find elsewhere, some of which a very strong case could be made for. However we just don’t need anything more right now, so the added complexity has no justification.

Obviously Rancher might not be right for you – ultimately there is no single “best solution”. Context is king – be realistic about what your needs really are and take care not to over-engineer.

Miniliths vs microservices

No that isn’t a typo, it’s a word I made up… One of the main drivers for the adoption of containers is the adoption of a microservices architecture in place of a monolithic one. However, just like containers, microservices bring with them a whole world of painful overheads. For every microservice you introduce, there’s a cost to pay, and you need to be careful any benefits aren’t swallowed by the downsides.

Where appropriate, consider a “minilith” approach i.e. keep the number of microservices to a minimum, grouping related functionality as pragmatically as possible whilst leaving doors open to allow them to be split out further later if required (which may possibly be never).

Yes, this won’t be pure microservices, and it may hurt a bit inside, but it’ll mean reduced complexity with fewer processes to monitor, log, deploy, recover, scale and so on.

Stand on the shoulders of giants

With the ecosystem moving so fast, rolling your own solutions should be a last resort. It might be tempting to build a custom orchestration framework, but unless you keep it actively developed it won’t take long until it falls behind third party solutions and you’ll miss out on capabilities you would otherwise get for free. Not to mention having a harder job of getting others up to speed on how it works.

Once the pace of change slows down and the ecosystem matures, then sure, give it a go (if you’re sure it’s absolutely essential). But until then, don’t be tempted into striking out alone unless you can clearly quantify the benefits.

Don’t forget the humans

All the talk of automation makes it easy to overlook the human element of developing and maintaining systems. However slick a solution you may have in mind, without the right culture and mindset amongst your teams, you’re going to have problems.

Remember that DevOps is about communication and collaboration as much as automation, and can require a cultural shift to facilitate this. Having people on your team who don’t work well with others is a much more important problem to solve than a bit of automation. And if you can’t change behaviours, perhaps it’s time to manage people out.

Networking (the human kind)

Watching session recordings, reading blog posts etc is all well and good, but you really can’t beat networking with others and sharing thoughts and experiences. This might sound obvious – you could say the same about any topic. However, given how early-on we are in the process of adopting containers, it feels especially applicable at the moment.

There’s a lot of collective wisdom that is yet to be distilled down into blog posts and best practices that you can only really access by talking to people who have been there and done it (and quite possibly got burnt by it!). Put aside your British reserve (if you are British, obviously) and get out there and interact!

Storage is a persistent problem

When asked in the panel discussion which area we felt Docker is most lacking in, each of the three of us picked storage. This was a common theme amongst the many discussions I had too – it’s pain that is clearly widely felt. Even something as simple as inspecting the contents of a Docker volume from the command line is harder than it needs to be.

Yes, there are various storage solutions out there (e.g. Gluster, ClusterHQ, Convoy), and new ones entering the market all the time. But this is a tough problem to solve – if you want distributed storage you have to bear in mind the laws of physics in terms of latency and architect your application to handle your use cases and available IO accordingly.

Basically, for anything stateful, your life is going to be much harder than if it was stateless. Where possible, farm data off somewhere to a non-containerised external service.

Finding good people is hard

I suspect I’m preaching to the choir here (literally everybody I speak to has agreed with this), but one trend that just won’t go away is that finding good people with relevant skills is hard. Even harder than building distributed storage systems. Bearing in mind how new the ecosystem is, most knowledge and experience lives within developers and engineers that are not yet ready to re-enter the job market.

Consequently this means if you need extra pairs of hands, you’re most likely going to need to turn to the contractor or consultant markets. Which may not be the end of the world (though possibly the end of your budget). But don’t rely on filling a permanent vacancy in a hurry, especially if you’re outside London.

This is another reason to keep your concept count as low as possible and lean on third party offerings – if somebody on your team leaves, how quickly will it take somebody new to get fully up to speed?

TL;DR

I appreciate much of this is fairly conventional wisdom that could be applied to many more areas than containers. But given the penchant developers have for shiny things, it’s easy for it to fall by the wayside sometimes.

The world of containers is huge and can feel daunting, possibly a mountain you daren’t tackle. It’s definitely not a silver bullet, but once you cut through the hype and take the knife to the concept count to eliminate anything you really don’t need, it becomes far more approachable. Don’t be afraid to give it a go (if appropriate), but keep your eyes open and talk to others. And make sure you get that culture right.

I’d love to hear about your experiences – feel free to comment or contact me via whichever channel you prefer.