Testing for Graceful Failure with

Vaurien & Marteau

Testing

Starting Point

• You have a web application
• You are confident that it works
• You have a solid deployment and monitoring setup

Starting Point

• You have a web application
• You are confident that it works
• You have a solid deployment setup
• Now...what happens when things start to break?

What do I know anyway?

• Mozilla Services
• Firefox Sync Server
• FirefoxOS Marketplace
• More coming...
• Organic process and tooling
• This will be a demo, not a tutorial

A Real-Life Bug

A Real-Life Bug

A Real-Life Bug

A Real-Life Bug

A Real-Life Bug

A Real-Life Bug

A Real-Life Bug

A Real-Life Bug

A Real-Life Bug

How do you
find these bugs?

Do it Live!

Doing it live:

1. Deploy it
2. Stress it
3. Break it

Doing it live:

1. Deploy it
2. Stress it
3. Break it
4. ???

Doing it live:

1. Deploy it
2. Stress it
3. Break it
4. ???
5. Profit

Deploy it

Stress it

FunkLoad

FunkLoad is:

• A functional- and load-testing tool
• A report generator
• A bit clunky to be quite honest...

FunkLoad Report

(opens in new tab)

Marteau

Marteau is:

• A front-end for FunkLoad:
• Create worker nodes in AWS
• Manage job queue
• Generate and save reports
• French for "hammer"

Marteau Web Interface

Break it

Vaurien

Vaurien is:

• A misbehaving TCP proxy:
• Behaviours: delay, hang, error, ...
• Protocols: raw tcp, http, memcached, mysql, ...
• Easily extensible for your needs
• French for "rapscallion"

Ideas for Use

• Combine with loadtests to check for cascading failures
• Integrate into functional tests, for specific error sequences

Summing Up

Take Home Thoughts:

• How does your app interact with the outside world?
• What happens when the outside world misbehaves?
• How can you simulate that under controlled conditions?
• Come hack with us!
  • "Continuous Stress Testing"

one more thing...

FunkLoad

FunkLoad

loads

loads is:

• A better FunkLoad:
• Easier to write tests
• Much more concurrency per node
• More real-time feedback, stats, etc
• Not ready yet, but stay tuned!
• Not French for anything AFAICT...

Thanks

Thanks

• Tarek Ziade
• Alexis Metaireau
• James Bonacci

More Info