Commit 27670dab authored by Brad Beyenhof's avatar Brad Beyenhof
Browse files

add Linux at Netflix Scale notes

parent 0a79fa7f
# Deploying Linux at Netflix Scale
## Netflix Scale
- 100s of millions of content streamed per day
- 100s of thousands of EC2 instances
- 1000s of daily production changes
## Environment
- AWS EC2
- Ubuntu LTS
- Jenkins/Spinnaker
- Aminator (build AMI images)
- Packer is the mor modern tool for this
## After PXE boot/install
- config/tuning
- storage
- networking
- library/package dependencies
- environment (users, directories, permissions)
- install the application
## Tailoring upstream distro
- focus on value to the company
- vanilla Linux base
- mainstream kernels
- mainstream packages
- As much as possible
- a few exceptions
- chrony (patched to do DNS lookups rather than binding to a specific IP forever)
- network driver
- python, java, node, etc.
- tunings & sysctl parameters
- Stuff that nobody should have to worry about
- Caching proxy for apt repositories
- Insulate developers to keep old versions around just in case
- apt config
- no auto upgrades
- don't install recommends (X11 through dependencies of ssh/ssh-askpass/etc)
- /var/log bind mount to ephemeral/EBS
## Conventions
- directory ownership, premissions
- create data & log directorie
- log file location, rotation, aggregation
- extensible environment variables
- user data injection
- Build & mojng RAID0 drive array
- automount EBS volumes
- automatic ENI attachment
## Value added features
- ezconfig app bootstrapper
- common conventions
- no init scripts
- sysvinit? deamontools? upstart? systemd?
- factor those away, but allouw devs to write scripts
- generate config files
- default apache, tomcat, gunicorn tuning parameters, etc.
- java GC settings
- security integrations (nearly) free
- SAML for webapps
- manual TLS for APIs
## Base image package development
- ~40 modular packages
- everything in git
- package source
- config files
- build scripts
- unstable/release git branches for every package
### examples
- SSH config (user key management?)
- log rotation & config
- network drivers
- environment-driven MOTD
- perf/tracing tools
- system tunables
- Atlas system metrics
## Branching model
- unstable/release
- nightly rebases of release onto unstable
## Base image promotion model
- Ubuntu image
- unstable foundation ------------> release foundation
- unstable base <-- debian pkgs --> candidate base
- release base
- Ubuntu & Netflix package hotfix promotion
- Changelogs are published internally, including commit history for local packages
## Metadata
- abstract technical details away
- build on a base image
- developers use a label (release/candidate)
- Tools map to a name (xenialbase-20180224)
- For access, too!
- ssh/scp wrappers to log in or copy to AMIs
- $ our-ssh perfeng,0
## Automate everything!
# This is what, now HOW
## Basic tenets
- devs focus on product
- immutable infra
- Deplyo on latest base
- Avoid runtime config as much as possible
- stateless instances
- every instance in a cluster is identical
## Design for availability
- Plan for failure
- Microservice API fallback: error, wait, substitute generic data?
- Chaos engineering
- Chaos monkey
- Manually failover to AWS region
- Failover testing so problems are under control
- deploy in multiple zones/regions
- canaries in production
- small clusters with new version
## Build and deploy apps
- CI builds emit a .deb
- red/black (blue/green) testing
# What about the future?
## Possible directions
- isolate applications from the system
- chroot/container/runc model
- separate dependency chains
- without extra runtime complexity
- while being immutable
## Challenges
- meta-packaging is immature (tools to build containers)
- clean up for lightweight, minimal containers
- played around with runc to start Tomcat without a base OS in the container
- binaries, dependencies
- Technology stacks are more diverse
- not just Tomcat / Java anymore
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment