- Play in a simulated environment
- Simulation environment
- Simulation environment for crisis situations
- Good weather doesn’t make good sailors
- Prepare for extreme situations
- Things are unreliable until they are tested
- How to play? game preparation
- Supervisors & players
- Make a plan role play & scope definition
- Set a date (deadline) make it fun
- Create the simulation environment
- Gameday environment aws account a production environment aws account b
- Cloudformation template
- Use aws iam control who can do what during the game
- Activate aws cloudtrail gameplay recording & auditing
- Simulate activity
- Pre-recorded network activity
- (throttled) load test use spot instances + tools
- Using amazon ec2 to simulate 2.4 million players
- Communication channels irc, shared docs, viber, …
- Play! the fun part
- Simulate failures
- Terminate resources change security groups change iam roles disable iam user change /etc/hosts file amazon rds fail-over test
- Record & annotate players actions
- If you don’t measure it, it didn’t happen
- After party closing the game
- Debrief where the magic happens
- How long does it take to detect an event?
- New hire? go to the simulator
- Key team players? simulate “failure” situation
- Validate assumptions prove your architecture knowledge of procedures
- Communication chanel & chain of command