Monday, April 18, 2016

DevOpsDays: Empathy, Scaling, Docker, Dependencies and Secrets

Last week I attended DevOpsDays 2016 in Vancouver. I was impressed to see how strong the DevOps community has grown from the time that I attended my first DevOpsDays event in Mountain View in 2012. There were more than 350 attendees, all of them doing interesting and important work.

Here are the main themes that I followed at this conference:

Empathy – Humanizing Engineering and Ops

There was a strong thread running through the conference on the importance of the human side of engineering and operations, understanding and empathizing with people across the organization. There were two presentations specifically on empathy: one from an engineering perspective by Joyent’s Matthew Smillie, and another excellent presentation on the neuroscience of empathy by Dave Mangot at Librato, which explained how we are all built for empathy and that it is core to our survival. There was also a presentation on gender issues, and several breakout sessions on dealing with people issues and bringing new people into DevOps.

Another side to this was how we use tools to collaborate and build connections between people. More people are depending more on – and doing more with – chat systems like HipChat and Slack to do ChatOps. Using chat as a general interface to other tools, leveraging bots like Hubot to automatically trigger and guide actions, such as tracking releases and handling incidents.

In some organizations, standups are being replaced with Chatups, as people continue to find new ways to engage and connect with other people working remotely and inside and outside of teams.

Scaling DevOps

All kinds of organizations are dealing with scaling problems in DevOps.

Scaling their organizations. Dealing with DevOps at the extremes, at really large organizations and figuring out how to effectively do DevOps in small teams.

Scaling Continuous Delivery. Everyone is trying to push out more changes, faster and more often in order to reduce risk (by reducing the batch size of changes), increase engagement (for users and developers), and improve the quality of feedback. Some organizations are already reaching the point where they need to manage hundreds or thousands of pipelines, or optimize single pipelines shared by hundreds of engineers, building and shipping out changes (or newly baked containers) several times a day to many different environments.

A common story for CD as organizations scale up goes something like this:

  1. Start out building a CD capability in an ad hoc way, using Jenkins and adding some plugins and writing custom scripts. Keep going until it can’t keep up.
  2. Then buy and install a commercial enterprise CD toolset, transition over and run until it can’t keep up.
  3. Finally, build your own custom CD server and move your build and test fleet to the cloud and keep going until your finance department shouts at you.
Scaling testing. Coming up with effective strategies for test automation where it adds most value – in unit testing (at the bottom of the test pyramid), and end-to-end system testing (at the top of the pyramid). Deciding where to invest your time. Understanding the tools and how to use them. What kind of tests are worth writing, and worth maintaining.

Scaling architecture. Which means more and more experiments with microservices.

Docker, Docker, Docker

Docker is everywhere. In pilots. In development environments. In test environments especially. And more often now, in production. Working with Docker, problems with Docker, and questions about Docker came up in many presentations, break outs and hallway discussions.

Docker is creating new problems at the start and end of the CD pipeline.

First, it moves configuration management upfront into the build step. Every change to the application or change to the stack that it is built and runs on requires you to “bake a new cake” (Diogenes Rettori at Openshift) and build up and ship out a new container. This places heavy demands on your build environment. You need to find effective and efficient ways to manage all of the layers in your containers, caching dependencies and images to make builds run fast.

Docker is also presenting new challenges at the production end. How do you track and manage and monitor clusters of containers as the application scales out? Kubernetes seems to be the tool of choice here.

Depending on Dependencies

More attention is turning to builds and dependency management, managing third party and open source dependencies. Identifying, streamlining and securing these dependencies.

Not just your applications and their direct dependencies – but all of the nested dependencies in all of the layers below (the software that your software depends on, and the software that this software depends on, and so on and so on). Especially for teams working with heavy stacks like Java.

There was a lot of discussion on the importance of tracking dependencies and managing your own dependency repositories, using tools like Archiva, Artifactory or Nexus, and private Docker registries. And stripping back unnecessary dependencies to reduce the attack surface and run-time footprint of VMs and containers. One organization does this by continuously cutting down build dependencies and spinning up test environments in Vagrant until things break.

Docker introduces some new challenges, by making dependency management seem simpler and more convenient, and giving developers more control over application dependencies – which is good for them, but not always good for security:

  • Containers are too fat by default - they include generic platform dependencies that you don’t need and - if you leave this up to developers - developer tools that you don’t want to have in production.
  • Containers are shipped with all of the dependencies baked in. Which means that as containers are put together and shipped around, you need to keep track of what versions of what images were built with what versions of what dependencies and when, where they have been shipped to, and what vulnerabilities need to be fixed.
  • Docker makes it easy to pull down pre-built images from public registries. Which means it is also easy to pull images that are out of date or that could contain malware.
You need to find a way to manage these risks without getting in the way and slowing down delivery. Container security tools like Twistlock can scan for vulnerabilities, provide visibility into run-time security risks, and enforce policies.

Keeping Secrets Secret

Docker, CD tooling, automated configuration management tools like Chef and Puppet and Ansible and other automated tooling create another set of challenges for ops and security: how to keep the credentials, keys and other secrets that these tools need safe. Keeping them out of code and scripts, out of configuration files, and out of environment variables.

This needs to be handled through code reviews, access control, encryption, auditing, frequent key rotation, and by using a secrets manager like Hashicorp’s Vault.

Passion, Patterns and Problems

I met a lot of interesting, smart people at this conference. I experienced a lot of sincere commitment and passion, excitement and energy. I learned about some cool ideas, new tools to use and patterns to follow (or to avoid).

And new problems that need to be solved.

53 comments:

Cengiz Kayay said...

In our enterprise, there are generic data services used by several departments. Each department asks for enhancements on these services.
There are times that the enhancements are finalised within the same week and to be released in that weeks release train.
But each department(user) wants to test it all to make sure the enhancement from other department did not break the service.
Thus, until both department's own QA agree on the results, the service can't be released. (Thus creating inter department decencies).
How to resolve this issue?
They ask me to split the service so that each channel has its own service to maintain.
But that is wrong approach as this will not be a generic micro service but project level service per user/customer. One option is to make one department to wait for the next release but that also delays the time to market.

Is there any options other than using feature toggles? I think, toggles are difficult to maintain...
Thanks...

@philn5d said...

It might not be so wrong to split it if each channel has different needs. This sounds like a classic case of trying to maintain a global domain model. It does depend, but you are in an enterprise environment rather than producing for public consumption. I've found that different departments in an enterprise have different definitions and completely different use cases/logic for the same entity (Employee for example). Perhaps you can make channel services that extend existing services when it becomes so much of a problem - better than fighting to maintain a globally unified model with irrelevant data and properties on it. Hope that helps ease the pain :)

nortonsetuprmt said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such blog.

www.norton.com/setup

www.spectrum.net/login

kaspersky activation

aolmail.com

aolmail.com

mcafee.com/activate

Andrew303john said...

Finance Assignment Help

We at Fullassignment.com bring to you the most significant Finance assignment writing service at the best cost. With long stretches of understanding we are prepared to give assignment help over the globe.You will be guided here with a portion of the information of Finance assignment which could assist you in deciding writing a Finance assignment. Nonetheless we uneuqivocally prescribe you to benefit Managerial accounting assignment help from our specialist to find out about marketing and its scope.

https://fullassignment.com/

kasperskyvbh said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such blog.

www.norton.com/setup

www.spectrum.net/login

kaspersky activation

aolmail.com

aolmail.com

mcafee.com/activate

kasperskyvbh said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such blog.

norton.com/setup

www.spectrum.net/login

kaspersky activation

aolmail.com

mcafee.com/activate

Anjali Vinu said...

Excellent post, it will be definitely helpful for many people. Keep posting more like this.
Blue Prism Training in Chennai
Blue Prism Training Chennai
Blue Prism course in Chennai
RPA Training in Chennai
RPA Classes in Chennai
Robotic Process Automation Training
UiPath Training in Chennai
Automation Anywhere Training in Chennai
Blue Prism Training in Velachery
Blue Prism Training in Anna Nagar

kasperskyvbh said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such blog.

norton.com/setup

www.spectrum.net/login

kaspersky activation

aolmail.com

mcafee.com/activate

chilan said...

Thanks for sharing, nice post! Post really provice useful information!

Hương Lâm chuyên cung cấp máy photocopy, chúng tôi cung cấp máy photocopy ricoh, toshiba, canon, sharp, đặc biệt chúng tôi có cung cấp máy photocopy màu uy tín, giá rẻ nhất.

chilan said...

Thanks for sharing, nice post! Post really provice useful information!

Hương Lâm chuyên cung cấp máy photocopy, chúng tôi cung cấp máy photocopy ricoh, toshiba, canon, sharp, đặc biệt chúng tôi có cung cấp máy photocopy màu uy tín, giá rẻ nhất.

Dwayne Johnson said...

Yahoo is one of the fantastically dependable email gadgets. It has received the attention of millions of users international, varying from one-of-a-kind age institutions and profiles.
||Yahoo Mail Customer Support Phone Number
||yahoo live chat customer service
||yahoo 24 hour customer support
||yahoo mail pro customer support
||yahoo toll free customer service
||how to contact yahoo customer service
||yahoo customer care live chat
||yahoo customer care live chat
||yahoo customer service contact number
||yahoo support
||yahoo tech support
||yahoo tech support phone number
||yahoo customer service number
||yahoo tech support number usa
||yahoo tech support number canada
||yahoo help
||yahoo support phone number
||yahoo telephone support number
||go to yahoo support
||yahoo 24x7 technical support number
||yahoo live chat technical support
||yahoo phone number tech support
||yahoo technical support usa
||yahoo help support canada

Shawn Michaels said...

In this day and age, email correspondence has become the most noteworthy piece of the present business. The truth of the matter is that we can’t live without email administrations and the vast majority of our budgetary exchanges rely upon it.
||https://roadrunneremailservice.com/
||https://roadrunneremailservice.com/roadrunner-support-number/
||https://roadrunneremailservice.com/roadrunner-customer-service/
||https://roadrunneremailservice.com/roadrunner-technical-support/
||https://roadrunneremailservice.com/roadrunner-email-settings/
||https://roadrunneremailservice.com/roadrunner-email-password-reset/
||https://roadrunneremailservice.com/roadrunner-email-problems/

kasperskyvbh said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such blog.

norton.com/setup

www.spectrum.net/login

aolmail.com

mcafee.com/activate

kasperskyvbh said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such blog.

norton.com/setup

www.spectrum.net/login

aolmail.com

mcafee.com/activate

www.norton.com/Setup said...

||Norton.com/nu16
||norton.com/nu16
||norton nu16 download
||norton utilities 16
||how to get norton utilities?
||norton utilities
||install norton utilities using product key
||norton nu16 download windows 10
||norton nu16 install for windows 10
||download norton utilities 16
||norton utilities 16 download setup
||norton utilities 16 activation code

Unknown said...

As claimed by Stanford Medical, It is really the SINGLE reason this country's women live 10 years longer and weigh 42 pounds less than we do.

(And by the way, it has totally NOTHING to do with genetics or some secret-exercise and EVERYTHING about "HOW" they are eating.)

BTW, I said "HOW", not "WHAT"...

CLICK this link to discover if this little quiz can help you decipher your true weight loss potential

kasperskyvbh said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such blog.

norton.com/setup

www.spectrum.net/login

aolmail.com

mcafee.com/activate

nortonsetuprmt said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such blog.

norton.com/setup

www.spectrum.net/login

aolmail.com

mcafee.com/activate

roadrunneremail said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such blog.

norton.com/setup

www.norton.com/setup

www.spectrum.net/login

roadrunner email

charter email

aolmail.com

mcafee.com/activate

ester and christmas ornaments

roadrunneremail said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such blog.

www.norton.com/setup

www.spectrum.net/login

aolmail.com

aolmail.com

mcafee.com/activate

ester and christmas ornaments

roadrunneremaillogin said...

I really happy found this website eventually. Really informative and inoperative, Thanks for the post and effort! Please keep sharing more such blog.

www.norton.com/setup

roadrunner email login

aolmail.com

rajanikota said...

Hi Very nice blog thank you for sharing this blog.
DevOps Training
DevOps Online Training

Unknown said...

Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end.
Salesforce Training | Online Course | Certification in chennai | Salesforce Training | Online Course | Certification in bangalore | Salesforce Training | Online Course | Certification in hyderabad | Salesforce Training | Online Course | Certification in pune

rajanikota said...


Hi, Nice and Valuable information you explained in this article. Thanks for Sharing!!
DevOps Training
DevOps Online Training
DevOps Training in Ameerpet

augustwalker said...

See the slides to check the means to fix a deadly blunder happened check the hp association administrator by dialing Windows specialized help telephone number to HP specialized help group. We Fix HP Connection Manager Fatal Error In Windows by HP client care administration group.

Trakdinam said...

Get high quality backlinks through Adult Blogging for your Adult Business.
Adult Guest Blogging
https://adultguestposting.com/why-the-men-should-go-for-the-best-london-escorts-services/
https://adultguestposting.com/some-essential-features-of-the-female-strippers-services/
https://www.warriorforum.com/members/adultguestposting.html
https://www.openlearning.com/u/adultguestposting-qc4131/
http://seqanswers.com/forums/member.php?u=122899
http://www.mappery.com/user.php?name=adultguestposting
https://www.heatbud.com/barnaby-estrada
https://www.apsense.com/article/why-the-men-should-go-for-the-best-london-escorts-services.html
https://mamby.com/post/some-essential-features-of-the-female-strippers-services

sara mohamed said...

شركة تنظيف منازل بجازان
شركة تنظيف منازل
شركة تنظيف بأبوعريش جازان
شركة تنظيف بأبوعريش
شركة تنظيف كنب بجازان
شركة تنظيف كنب
شركة عزل مسابح بجازان
شركة عزل مسابح
شركة عزل خزانات بجازان
شركة عزل خزانات
شركة عزل مسابح بجازان
شركة عزل مسابح
شركة عزل اسطح بجازان
شركة عزل اسطح
شركة مكافحة حشرات بجازان
شركة مكافحة حشرات
شركة تنظيف المجالس في جازان
شركة تنظيف المجالس
شركة تنظيف موكيت بجازان
شركة تنظيف موكيت
شركة تنظيف فلل بجازان
شركة تنظيف فلل
شركة تنظيف سجاد بجازان
شركة تنظيف سجاد

MS Dynamics said...

Hi there, I enjoy reading through your article post. Thank you for sharing.
DevOps Training
DevOps Online Training
DevOps Online Training in Hyderabad
DevOps Online Training institute
DevOps Training Online
DevOps Online Course

catsbill gst billing said...

nice article , worth reading
Visit catsbill.com!

Mercy said...

Now book your flight tickets at lowest price, Aeromexico Reservations provide you very easiest way to boook your flight reservations.

Tech And Travel said...

Hello! Thanks for sharing this amazing content your information is really very awesome to read. Keep it up and best of luck for your future updates.
For booking cheap flights to usa visit at
Aeromexico Reservations Telefono.

Muhammad Azwar said...

Nice Post Also Check This Out
About Islam
Dynamic Movies

Hritik said...

Thanks for sharing awesome blog post, I read the whole stuff.
FDA fast track
Emergency Use Authorization
drugs FDA
regulatory affairs
Expert witness anesthesiology
FDA label
pharmaceutical scientist
fda 510k
IND FDA

Giselle Lily said...

I appericiate your comments i want to share that Get all data about the flight booking at the brisk and proficient helpline helping a large number of travelers.

spirit airlines reservations
southwest airlines reservations

allegiant airlines resrvations
eva airlines resrvations

avianca airlines reservations
southwest airlines phone number

Sally Berg said...

How To Cancel Prime Subscriptions
How To Cancel Paypal Subscriptions
How To Cancel Amazon Prime Subscriptions

clarkdevis said...

United Airlines Phone Number

clarkdevis said...

American Airlines phone Number

clarkdevis said...

Southwest Airlines Phone Number

Rocky Johnson said...

Due to the excellent features of Eva Airlines, I always want to book my Eva Airlines Reservations because this is the best airline to travel in the air with 100% comfort in an affordable budget.

Rocky Johnson said...

They are offering cheapest airfare flights booking choices and alternatives on the Avianca Airlines Reservations. Therefore, I just want to say thanks to this flights booking agency which is providing extra airfare discounts on air tickets booking.

Mercy said...

Are you looking for best way where you can easily book your flight tickets at United Airlines Booking

jasmin said...

Nice, book your flight tickets at United Airlines Flight Reservations

Anonymous said...

Nice blog..
Are you want to book your flight tickets at United Airlines Booking, but worry about how to book then here you will get complete information, our customer service 24*7 available, if you face any issues then contact with our technical experts.

Sophia said...

Thanks for sharing this information you can easily book your flight tickets at united airlines reservations official site, we provide 24*7 customer service.

Radhya said...

Great blog thanks for share with us.
Best website development company in Lucknow.

camila smith said...

If you are looking for southwest airlines reservations then here I will give you best way to book your flight tickets.


Southwest Airlines Reservations || Southwest Airlines Flight Reservations

Carry Roberson said...

Delta Airlines Reservations are quite cheaper option for the travel in the United States. You can filter the best airfare on the booking of Eva Flights from the right website for the booking. Delta Airlines Official Site is the right place to find the Delta Airline Tickets Booking at the Cheap Cost. You can also compare the airfare for the airfare booking deals to get the best price reservations.

Carry Roberson said...

For managing online Eva Airlines Reservations you have to browse at the Eva Airlines Official Site. This place guides you to make sure the booking online at the lowest cost. You can find the deals and compare the airfare for the booking of Eva Flights at the Eva Airlines Site.

camila smith said...

Now book your flight tickets at Southwest Airlines Reservations, they provide best way to book your flight tickets at lowest price, our customer service 24*7 available for customer help.

Rajesh Verma said...

Thanks for your article. It was interesting and informative.
Here I also want to suggest your reader who usually travels one place to another they must visit Airlines Gethuman that offer best deals to book your seat on Southwest Airlines Flights. So do hurry and avail the best deals and get rid of to check different websites for offers.
Get more help on
Southwest Airlines Flights

camila smith said...

If you are looking for best international flight booking system, then here Delta Airlines Reservations customer service available to help you. For more details, visit link: Delta Airlines Reservations

camila smith said...

Now book your flight tickets at Southwest Airlines Reservations, they provide best way to book your flight tickets at lowest price, our customer service 24*7 available for customer help.

DEVID WARNER said...

123.hp.com/setup | Set up and Install 123.hp.com dj2635 Printer USB Setup, Print from Windows and Mac using 123 HP Deskjet 2635 Printer USB Setup.

Site Meter