Reflections on Velocity and DevOpsDays

Sitting on a plane on my way home from Velocity and DevOpsDays, I am thinking about the people I met, the things I learned, and the ways we need to improve communicating the power of this culture and community. Bethany and I had a great time (she was able to get a pass to attend the social events with me) talking with interesting folks including old friends like @mtnygard, @sascha_d, and the @seadevops crew.


The Velocity crew I met were all great (with a special shout-out to Patrick for his assistance and general awesomeness) but the conference did have some issues. I was a little late arriving for the keynotes on Tuesday and the main ballroom was standing room only. More specifically there was no good way to get to the empty seats spread throughout the hall and the chairs were so tightly packed that people felt the need to take an extra half seat each. I expect that the Velocity crew will need to evaluate the space and if they can make it work next year, but cramming more chairs into the room is not going to help.

There were a lot more women at Velocity this year compared to previous years according to others who have attended in the past. Attendees were well behaved and at least I did not see any sexism on the part of my fellow conference goers. Some of the vendors, however, did not get the memo. The new CTO of GoDaddy put his foot in his mouth with a “Sorry I couldn’t get any GoDaddy girls here today… Danica Patrick is stuck in traffic” joke that pissed off at least a few people. I had a good conversation with him and others from the GoDaddy booth the day before and they indicated that they had gotten the sexism/SOPA messages and were changing their style, but that was undermined by his performance. We will see what happens with their upcoming Olympics advertising campaign, but I am not hopeful.

And then there was Edgecast… Adrian Cockcroft from Netflix said it best:

Why does Edgecast torture boothbabes by making them balance on high heels all day? Embarrassing. No BBs at #velocityconf 2013 plz


On the flip side I also realize I need to watch my assumptions. Apparently one vendor happens to actually have three blonde, Swedish gals in their twenties working as interns. They are all persuing CS degrees and one is working on her thesis on gossip protocols. My assumptions were wrong and that is on my head.

As for the sessions, the talk by Dr. Richard Cook on how complex systems fail was very interesting. He told a story about a hospital that standardized on one manufacturer of infusion pump, used to deliver fluids and medications to patients. The new pumps were all introduced in a relatively narrow window of time and a year after the first pumps were put in service they started to fail. Specifically they would not take a new configuration. It started with one pump, early in the morning and within a few hours as much as 20% of the pumps were non-responsive.

The culprit? The pumps had a mandatory upgrade period and it seemed that it was appropriate at the time of configuration to set that to one year. When the upgrade period passed the pump would not take a new configuration. Once they figured it out they had to reconfigure all the pumps in the hospital. The configuration was made because it seemed to be the right thing to do, but the consequences were not fully understood.

Check out the full video from Velocity if this has piqued your interest.

Mike Christian from Yahoo! also had a great talk entitiled ”Frying Squirrels and Unspun Gyros”. When most engineers build a datacenter they add generators, uninteruptable power supplies, and other technologies to allow the systems to ride out a disaster. What we don’t take into account is the additional complexity and therefore additional risk this introduces. Everything from squirrels chewing electrical systems to cascading HVAC failures, sometimes the complexity we accept isn’t the complexity we need.

As I was getting on the plane I saw that Amazon Web Services had issues in one of their datacenters in Virginia. They tend to get flak but I think it is more that many high profile companies are in their datacenters and while they build the infrastructure they don’t control their customers’ architecture. Infrastructure fails no matter who builds it, but systems can be architected to be resilient to that failure. Practicing failover between datacenters regularly is one way. Using techniques that allow you to move traffic between locations in another. It is not simple, but architecting for the public cloud regardless of where your data is stored will give you more resiliency, provided you practice the diversion of traffic all the time.


Thursday and Friday were DevOpsDays in Mountain View. DevOpsDays is a combination of prepared talks and an unconference format that allows people to suggest topics and get some space to talk about them.

A lot of the conversations I had were about culture. We talked about how we can prevent a loss of ‘control’ over the term DevOps when the kinds of vendors who are always looking for the new bandwagon start circling. We talked about building a body of knowledge that anyone can contribute to. Mostly we bonded, shared stories, conspired against the future, built friendships, and drank beer.

I can’t wait until next year.

TWIL: YUM, Proxies, RabbitMQ, and Chef

This week I learned about some esoteric looking (but simple to solve) errors.

RabbitMQ and Chef

I was assisting a client with setting up and testing a new Chef 0.10 instance. He completed the steps we had outlined an used in previous environments, but we were getting a strange error when attempting to run “knife configure -i”.

Creating initial API user...
ERROR: Server returned error for http://localhost:4000/clients/root, retrying 1/5 in 3s
ERROR: Server returned error for http://localhost:4000/clients/root, retrying 2/5 in 6s
ERROR: Server returned error for http://localhost:4000/clients/root, retrying 3/5 in 11s

We tried several things, checked some logs when we found this line with a substantial stack trace:

merb : chef-server (api) : worker (port 4000) ~ Connection failed - user: chef - (Bunny::ProtocolError)

After some head scratching and iterating on search terms I found a blog post by Jonathan Polansky indicating this was a problem with RabbitMQ not having the Chef specific configuration. He helpfully provided the commands to run that only needed tweaking to add the default password for the RabbitMQ user ‘chef’ (which is, BTW, ‘testing’).

rabbitmqctl add_vhost /chef
rabbitmqctl add_user chef testing
rabbitmqctl set_permissions -p /chef chef ".*" ".*" ".*"

Thanks to Jonathan for finding that!

YUM and Proxies

Working with the same customer on spinning up a test machine with their new Chef infrastructure we came across this error:

Setting up repositories
http://[REDACTED]/RedHat/repodata/repomd.xml: [Errno 4] IOError: <urlopen error (111, 'Connection refused')>

We could retrieve the file via CURL from the same machine without a problem but yum refused. Finally we figured out that our standard configuration in Chef for yum required a proxy. We had set this up in other environments, but we have not retrofit the new Chef instance to be configured by Chef.

Daily Routine, One Year Later

This short article was originally posted in April 2010 on Jeffrey’s old blog.

So it has been over a year since I first posted my daily routine, what worked, what didn’t, and what I felt I needed to do different. That post has been a favorite, so I figured a year-later followup would be useful.

To review last years list:


Review my todo list first thing when I get into work in the morning.

I have gotten good about this, largely because I am keeping ALL of my work in my todo list. I work in an IT Operations team, so when a new ticket comes in I create a placeholder for it and start building out tasks needed to complete the work.

Check my work email only a couple of times per day.

This depends on the type of work I am doing at the time. I have had to compromise on this a little and, rather than set specific times I check my email, I set time blocks I do nothing other than the work task at hand.

Prioritize my work.

I used to rank my work A-E and the end result is everything that stays on my list is priority A. This is not useful. I am using a tool (a separate post for that coming soon) that allows me to stack rank everything I am doing. I do not apply a false ranking to my todo items, but a true rank (should I do this before or after the other things on my list). I usually have a couple of things in a ‘working state’ at a time since my job requires a certain level of multitasking.

Update my todo list with everything that I get done or need to do.

Once you have everything in front of you it can be simultaneously stressful and freeing. Get everything down and then be realistic about what you can do. I know people who try to do it all and end up making themselves sick. They seem to fear putting it all out there because it will be overwhelming. Remember the story of Vice Admiral James Stockdale, a prisoner in the “Hanoi Hilton” during the Vietnam War. He reported that the people who did not survive were the optimists, those who always believed they would be home by some particular date. He said, “They died of a broken heart.”

From this comes the Stockdale Paradox. In the man’s own words: “This is a very important lesson. You must never confuse faith that you will prevail in the end — which you can never afford to lose — with the discipline to confront the most brutal facts of your current reality, whatever they might be.”

Build a list of things you think you need to do. Filter them through your goals. Be brutal, but know you are going to prevail.

Spend time on professional development.

This blog is part of it. I am working on developing my consulting business, so I am spending time on that when I have it. I am currently reading ‘Switch’ on how to affect change when change is hard.

Get a few minutes of face time with my boss.

My boss is a nice guy with a wicked sense of humor, so this is pretty easy.


Getting on my computer before work in the morning.

I have mostly stopped this bad habit. My wife and I commute together (and work for the same company) so timing our schedules to be ready at the same time has helped.

Check my personal email at work more than once a day.

I probably look a couple of times per day, but I have also worked to reduce and filter the amount of mail I get in my personal account. Fewer mailing lists, vendor emails, and the like have made processing my email box a quicker proposition.

Things I need to be doing

Purge items from my todo list/replyto mailbox that I have not gotten to and is not important to my goals.

The first part of this is to have goals. On the site, Brazen Careerist, I have listed my goals as:

  • Establish my consulting business
  • Present at technical conferences
  • Be indispensable

The last item is from Seth Godin’s book, Linchpin. It is by far the most compelling and difficult to define goal I have at this point in my life.

Define a list of goals for this point in your life. Use that list as a filter for everything on your list. Remember that there are three types of work in life: The Critical Few, The Functionally Mandatory, and The Trivial Many. What do you want to spend your time on?

Breaking tasks into manageable next actions consistently and immediately.

I have gotten better about this with keeping all of my work stack ranked. I have to break larger tasks into smaller pieces so I can work them in with all the other demands on my time. I have found a project undivided is a project never started.

I will be thinking over the next bit and a while about what, if anything needs to change on this list. Until then, stay productive.

Passion and the Job

This short article was originally posted in February 2009 on Jeffrey’s old blog.

I have had an interesting weekend. My wife, Bethany, and I drove over the mountains to visit an old friend and mentor, John and his wife Sandy. We got in about 4pm on Saturday, sat and talked for a while, and went a lovely dinner. Today we went to breakfast about 9am, then went to a wine tasting at Kestrel Vintners. Bethany and I then helped John lay out the foundation for his new barn that arrives next week and dug 20 holes in the ground. Hard work and math (to get the building square) but oddly fun. We came back in the house, warmed up, watched some NASCAR (Sandy is the jock in their family), ate dinner, and watched a couple of okay movies (while tearing them apart for bad physics).

All this time we talked about all kinds of things. John and I worked together in some great jobs and some really crappy ones, but one of the most important things I learned from him (and remembered this weekend) is having passion for your work.

My current employer is great. It is a fabulous atmosphere, a strong business model, and an interesting challenge. The dynamic of my team sometimes leaves things to be desired, but I think I can work on that. Being around John has reminded me of my core values.

No. 1 — You build success for yourself by building success for others.

John has been a great mentor to me, both as a boss from time to time and as a friend all the time. I have a responsibility to mentor others, share my skills, and build philosophy to temper technology. I did a fair bit of mentoring with Taylor, but not as much with my current teammates. That is a mistake I plan to fix. My deliverables and deadlines and paycheck are important, but more important is the positive impact I can have on the people around me.

No. 2 — My strength as an engineer is understanding the problem from both the technical and the business side.

I am an odd duck. I think like a scientist and engineer, but can talk like a business person. I have let this skill slip over the last couple of years and need to seize it again. I have a goal for the first half of the year to spend 15 hours learning from members of the business side of the house. I really need to step that up and make it happen.

No. 3 — Ask guiding questions. If you get in a battle of wills, you have already lost.

A question is a powerful thing. I believe it was Pablo Picasso that said, “Computers are useless. They can only give you answers.” Having the humility to ask a question allows you to shape to conversation or debate without creating combat. “Never argue with an idiot. They’ll drag you down to their level then beat you with experience.”

No. 4 — Keep your tools fresh. They will rust without use.

There are many things I learned from John (like the art of guiding questions) that I have let atrophy. You have to keep your tools active or they will rust. I have a couple of books I am going to reread over the next couple of weeks as a start in that direction.

No. 5 — Leadership is more powerful than management.

Management is a set of responsibilities granted to you by a higher power. It usually implies paperwork, meetings, politics and such. Leadership is an act; a choice of how you are going to proceed through the day. It is about guiding and growing those around you. It means doing your job with excellence while encouraging and building the people that you work with.

Leadership, in thought and deed has been my passion. It is one I will be working to reach again.

An Implementation is Not a Pattern

This short article was originally posted in October 2006 on Jeffrey’s old blog.

I am learning about patterns and pattern languages here at OOPSLA and PLoP in Portland and I see a misconception in how patterns are used in some parts of industry, in PHP development in particular…

Namely, an implementation of a pattern is not the same as a pattern.

If you implement the patterns from the Gang of Four’s Design Patterns: Elements of Reusable Object-Oriented Software in your favorite language that does not make your work a pattern. A pattern is a thing you build, a process for building it, and a rule of when to build it. It does not define if you build it in Java, C# or PHP.

This does not reduce the value of the work, but it is confusing to call them patterns. Patterns transcend implementation. If you look to the proverbial godfather of patterns, Christopher Alexander, and his book A Pattern Language you see that in some ways his patterns transcend even the domain (architecture and building design) that they were written for. These pattern are in some ways generative, creating or inspiring thought in areas other than in originally intended audience.

Need Help?

Are you ready to discuss how we can help you?

Contact Us »