ContainerCon 2015

Colin Walters, Platform Engineering, Red Hat, Inc.

keybase.io/walters | walters@{redhat.com,verbum.org}

My Background

  • Before Red Hat: Emacs, Rhythmbox, Debian, SELinux
  • Red Hat: Fedora, Red Hat Enterprise Linux, DBus, GNOME (glib/introspection/gjs), systemd, polkit, NetworkManager, OSTree, rpm-ostree...
  • Project Atomic

Before Docker: RPMs/debs: A local maximum

Effort to be useful and different is high

rpm/dpkg etc. all evolve to similar point

Before Docker: Build -> Deliver -> Manage

Problem domains tied together

Continuous Delivery is very different from CI

Management tools (Puppet/Ansible/OpenSCAP/CloudForms/System Center) are important

Before Docker: Distributions and cadence

Common pattern: Take base distro as frozen set with security updates, move slowly on it, iterate on application

Distribution model not great at 3rd party app support (ABI guarantees etc.)

Before Docker: OpenShift v2

PaaS (on premise, online) primarily focused on "web app" languages (Java/dynlangs)

pid namespace, but not mount namespace

nginx cartrige not ergonomic

Why Docker

  • Containers are not new, and yes you still want to use virtualization
  • Dockerfile makes diving into containers really easy: Mash together rpm/dpkg + pip/cargo/rubygems/godeps + config files
  • Layering model easy to understand
  • Major investment: RBAC, Storage, Kerberos, Networking

Post Docker: Project Atomic

  • Atomic Host: Docker + Kubernetes + rpm-ostree + kernel(LVM,SELinux,...) + systemd
  • Atomic Host: Not a distribution, an upstream for technology that goes into Fedora/CentOS and enterprise
  • Maintaining Docker base image
  • /usr/bin/atomic: A new command line tool for super-privileged containers
  • atomic-reactor: Layered Docker build system
  • nulecule: Multi-container app abstraction

Post Docker: OpenShift v3

  • Kubernetes + PaaS
  • S2I lighttpd: RUN yum install -y lighttpd ...

Project Atomic Timeline: 2013-2014

Project Atomic Timeline: 2015

  • 2015-01-14: /usr/bin/atomic (super-privileged containers)
  • 2015-03-05: Red Hat Enterprise Linux Atomic Host
  • 2015-05-26: Fedora 22 (with Atomic Host)
  • 2015-06-24: OpenShift Enterprise v3
  • 2015-07-03: atomic-reactor build system: More robust/controlled Docker image builds

Docker/distro: Packages will continue to exist (but should evolve)

  • Inventory: What version of OpenSSL is in this image?
  • Share content between different images (libc, ca-certificates)
  • Source -> artifact tracking/binding (GPL compliance, rebuild)
  • Traditional deployment will continue for a long time

Docker/distro: Future: One layer base, vs app-only containers

  • Deep layering not really valuable assuming inventory-aware updates
  • Streamlined containers: app+dependencies
  • One layer base: OpenShift v2, FROM centos/debian:jessie, xdg-app, sandstorm.io, ...

Docker/distro: One SELinux policy, many apps

  • Generic broad svirt_lxc_net_t type applies to all containers by default; MCS and kernel namespaces add separation
  • Running apache in Docker no longer has httpd_t
  • In my opinion, having coarse but always on policy is good
  • Should better support controlling information flow on mounted volumes in Kubernetes
  • Labeled networking? /OVS

Docker/distro: systemd

  • Unit files are nice
  • Start on boot: service state better solved in distributed fashion by kubernetes
  • runc/libcontainer from unit file
  • "virt-lite" (systemd-in-docker) should be supported
  • systemd-nspawn oriented towards stateful virt-lite
  • journal is nice, and Docker logging has been slow to evolve

Docker/distro: systemd vs microservices

  • An important end state is "thin" immutable state-stored-elsewhere containers
  • Container with just nginx: Package includes unit file (and systemd dep)
  • Add support to kernel pid namespaces to handle “pid 1 duty” like auto-reap zombies etc.
  • Parse unit files and generate Dockerfile fragments? Seccomp rules, etc.

Docker/distro: Distro value

  • https://fedoraproject.org/wiki/Changes/CryptoPolicy example of distro value
  • Know that no software is willing to speak SSLv3
  • Official Docker Hub nginx container is FROM debian:jessie
  • Want to control ca-certificates, but /etc/ssl (Debian) vs /etc/pki (Fedora)
  • shared-mime-info

Docker/distro: Image maintenance

  • Base images from distro to Hub: Painful, no provenance/signing support
  • "Maintainer": bugzilla, cadence
  • Content policy (Fedora: FOSS)

Docker/distro: Fedora layered images

  • Work ongoing to deploy atomic-reactor in Fedora for layered images
  • "tools" image and containerized Cockpit UI for Fedora 22 blocked on this
  • Ongoing discussion on registry

Docker/distro: Release engineering

  • Hard and generally thankless problem domain
  • Not many people, lots of different technology
  • Do not underestimate the cost of new delivery types
  • Example: Fedora mirrormanager/metalinks and local mirrors

Docker/distro: Cost of two ways to do it

  • Maintaining containerized and non-containerized deployment simultaneously is hard
  • Fedora.next/Rings: Fedora.next
  • Fedora Atomic under Cloud, with mainline image
  • Keep httpd_t?

Atomic Host

  • Tuned devicemapper partitioning for Docker (or overlayfs)
  • rpm-ostree with docker+kube+flannel
  • A/B type update scheme with minimal package set
  • /usr is mounted read-only always
  • Writable /etc - Ansible works

Atomic Host: Software on the edges

  • Added both screen and tmux
  • Parts of Cockpit included in base tree
  • Networking: IPSec
  • iSCSI in host, but not LUKS (currently)
  • oVirt
  • 3rd party kernel modules, PAM modules...

Atomic Host: SPCs

Future w/OpenShift

  • Sharing more code (focus on Kubernetes)
  • Project Atomic has upstream kube, Origin has a lot of patches
  • Shared installer (Ansible/cloud-init), Vagrant boxes, etc.

Future tooling wishlist

  • Better package system support for microservice containers (no yum in container, no rpm, just rpmdb)
  • Non-root package systems (prior art exists)
  • More sophisticated pkg <-> image tooling (rpm-ostree, vs Dockerfile/deb2aci)
  • Ansible like layer for sophisticated application config

Future security wishlist

  • Strong seccomp rules always on (sandstorm.io), perf, network syscalls
  • Enforced code signing (notary vs `rkt trust` vs package systems)
  • Kubernetes MustRunAsNonRoot vs signing and vs setuid binaries
  • OSTree-like storage backend (build containers with rofiles)
  • User namespaces exposed to uid !=0 is scary

projectatomic.io