The Agile team’s guide to harnessing change and responding to disruption

>> MEGHAN DAVIS:  Thanks for joining today.  And also thank you to the organizers of MidCamp for putting together a virtual conference.  I understand it was a tough call to make.  But excited to hear how they did so  so successfully and happy to be here for Day 2 of this conference so thank you.  

I'm here today to present on the topic of incident management, specifically how Agile teams can harness change and respond to disruption in ways that are overall effective and efficient.  

My name is Meghan Davis.  And I am a Senior Project Manager and team lead with Palantir.net.  I've been with Palantir now for about just over two years.  And I'm based in Cape Cod in Massachusetts.  

And at Palantir, I lead a number of different projects, including some larger portfolios as well as digital transformation projects across several industries including healthcare and State Government.  

And I've been on several teams at Palantir.  And prior.  Where we have experienced incidents and based on those experiences, I've developed a series of best practices in place that we use at Palantir during times of or moments of disruption.  

These plays allow us to collaborate more efficiently, respond faster when there's an incident.  

And overall allow us to iterate and refine and really hone in on our process over time.  And that's what I'm here to share with you today.  

So a few of -- an overview of how this presentation is structured, the first portion of the presentation focuses on the foundational concepts.  So who is this session for?  What is incident management?  And why do we even talk about it?  

We'll then dive into the practice of incident management.  And how high-performing teams manage incidents in moments of disruption.  

You'll leave with a series of tools and best practices that you can implement today.  And then we'll also cover how you and your team can continue learning after the incident is resolved.  

To the foundational concepts.  Who is the session intended for?  

This session and the practice of incident management is for any member of a team who provides client or customer support or services.  Anyone who manages a team who provides the same support or services.  

It's intended for Agile practitioners, project or program managers.  And then members of the development or an Operations Team.  So in summary, incident management best practices are for everyone.  

What is an incident?  When we talk about an incident, as opposed to a risk, we're really referring to an event that causes any sort of disruption to or a reduction in the quality of a service that we provide and it typically -- not always -- requires either a time-sensitive or a timely response.  

An incident is different from a risk in that a risk is generally something that you're forward thinking and maybe realize but you can put a plan in place to mitigate it.  Whereas an incident is really something like a moment that you're living in, something that you're experiencing right now that has to be resolved quickly.  

In incidents I think a most tangible one is if a site crashes.  This is something that I experienced recently.  And we were able to put into play the incident management practice or best practice and play in order to get that site back up and running.  This is perhaps the most clear kind of clear-cut example of an incident because it's generally client facing.  And it's user facing.  

It has a high impact.  It disrupts business operations potentially.  And then given the sector that your client is in, there may be even more of a need to get that site back up.  

So the example that I -- the one that I experienced was with a healthcare site.  And given the current state of the public health crisis, it was all the more imperative that we move quickly and efficiently as a team to get that site back up and running so that critical public health information could be shared widely.  

An example of an incident may be that a piece of functionality stops working.  So revisiting the example of working with a healthcare client.  Their site may be up and running.  But the find a doctor piece of functionality may not be working.  This would be an example of an incident.  And one that you would want to apply the incident management practice to get that piece of functionality back up and running.  

Because as a key part of the daily business operations and daily business goals of the client.  

Another example, it may be one that you may not immediately think of as an incident.  This one I've experienced where you could apply this practice to to reach a resolution.  So a client may come to you and request a number of iterations in either the design or strategy or DEV phase of a project that goes well beyond the scope -- the scope, timeline and budget of the engagement.  You would be able to apply the best practice of incident management here because it's something that requires a time-sensitive response because it has a potential to really either jeopardize a project or really explode that timeline and budget that you had agreed upon.  

So why do we even talk about incident management?  Incidents are costly to both your team and to the client.  

In the first example that I provided of a site coming down, every moment that site is down comes at a cost of the business to the client.  So getting the site back up and running is imperative so that you're minimizing kind of the money loss for a client.  

On the process side, if incidents are not handled well, what ends up happening is you have a team that's kind of confused about their roles and people are working not collaboratively.  And that becomes expensive and taxing on a project budget.  So it could have the potential to jeopardize reputation, that's your reputation, your project team and your company.  And then it also affects customer attrition.  

Customer attrition is really what keeps a company thriving.  It's that support client who signs on year after year.  Or that client who liked the first project you did so they bring in more work.  It's what keeps the client coming back to your company.  And that customer attrition really hinges on open, honest communication and trust.  Which in moments of disruption is really what is at risk.  That level of trust between you and your client.  

Incidents are disruptive and can be very confusing to a project team.  They have the potential to steer attention away from work and towards the incident in a way that can be very unproductive and costly or taxing on a project budget.  

So what are the benefits of having an incident management play at your company.  

First it's -- it allows for an increased sense of collaboration among your team members.  The process that we'll go through today is one that's inclusive of key stakeholders' opinions.  It's one that encourages participation in a meaningful way to solve a problem.  So you'll certainly see an increase in the amount of team collaboration that's happening.  

Benefits are direct communication and the building of trust really go hand in hand.  

Clear, concise, empathetic and relevant messages are of communication have a way of building client trust and loyalty because the client is informed more quickly and honestly about what is happening.  

Internally direct communication can provide a shared understanding of who is doing what, and when, and what is the current state of the incident.  

This has a way of building trust internally.  Team members have an understanding of who is doing what.  So they can trust that their teammates are going to execute during moments of disruption or in times of an incident.  

When we get to the reflection portion of the incident management process or timeline, you'll see that having an incident management play encourages honesty and openness in some of the best ways.  It really creates an environment where high-performing teams can thrive.  And it allows teammates to raise their hand or wave a flag when they see that either something has gone awry or it needs to be fixed as soon as possible.  

Everyone okay so far?  

>> DOUG VANN:  We still lose a couple of syllables at the beginning.

>> MEGHAN DAVIS:  Oh, no, okay.  Thanks.

>> DOUG VANN:  But it's flowing very nicely.

>> MEGHAN DAVIS:  Thank you.  

Why don't we now dive into the practice of incident management.  

An example of this presentation -- in this presentation we'll cover kind of the timeline of events and the process that you're going to go through from the moment you discover or realize an incident through when you're taking action to resolve the incident and close it out.  All the way to the final step of reflecting on what happens so that you can assure that you don't make the same mistake twice as a team.  

Part of incident management is discovery.  

There are ways that you can detect an incident.  The first and often the one that may not feel that great is when a client or customer raises an alarm.  So you get that email saying, our site is down.  What are you doing to get it back up.  That's generally one way that you'll learn of an incident.  

I've noticed that sometimes a customer or client will take action by filing a JIRA ticket or casually mentioning that something isn't working on a call.  They don't raise the flag entirely.  But they sense that something is not right.  Say a piece of functionality is not working and they are like, oh, we're working to try to find it and get it back up and running.  That generally should perk your ears that perhaps there's an incident here and we should ask more questions.  

There's also monitoring of systems.  So again, an email saying your site is down or something is not performing the way that it's intended.  The fourth way is that a teammate makes an observation or flags in Slack or email or you get a phone call that something has gone wrong or that they notice something is awry.  

This is generally what I would take is if a teammate is empowered to do this you're generally seeing the signs that you're on a high-performing team because they feel empowered to raise their hand and say that something is not functioning correctly or they need help.

And then the fifth way is a teammate similar to the client experience, they mention that they are experiencing some challenges in their work.  And the more questions you ask, you realize that there's an incident at hand.  That you need to apply this practice to.  

You're now entering into the act phase.  So you recognize that there's an issue.  Your client has reached out or your teammate has posted something in Slack saying that there's an incident.  What you do next is what's going to make or break a high-performing team.  

So the tools that you're going to need for open, honest, frequent communication.  There are several different avenues that you can take.  The goal is to create a shared space or a safe space for this type of communication to happen.  It first begins with a shared understanding on the team that everyone at some point in your company, on a project team, will experience an incident.  But you need to have the tools in place for people to be raising their hands that something has gone awry.  

So here are a few examples of what we use at Palantir.  

So the messaging tool that we use is Slack.  And that's generally what I've noticed where teammates will post if there's a potential incident.  

We have specific project channels that are fully transparent to anyone in the company.  You can join if you have an interest in that project.  And teammates will generally use the @here to notify everyone on the team that something is off or there's a potential incident.  

We then have a client communication tool, we use email and phone.  When you're checking in with your clients as a lot of -- at least for Palantir, a lot of our clients are now moving to a work-from-home environment.  What this has brought is an update in all of their contact information.  Email stays the same.  But no one is at their office anymore.  So we've had to go through and really update all of our contact information for clients, especially their phone now that most of them are in a work-from-home situation.  

You will want to use some sort of shared collaboration space that provides live updates.  So at Palantir we use a combination of Google Docs and then Atlassian's Confluence.  And with Confluence every project has its own space with a section for incident management.  

Video chat, this is perhaps one of the most critical ones.  We use Zoom or Google Meet.  But for an incident we generally will just spin up a Google Meet.  

A disruption or request from a client has come in, your teammate receiving this should notify the full project team.  The next thing that you're going to want to do is start a video call with your team.  It's important that you get everyone on the call so that you can go through the next few steps.  

So determine the incident team.  When you're doing this, though, is you're going to want to create a team based on what you are trying to solve for.  It is not always the same team.  So on the examples I provided earlier of a site going down versus a client requesting to iterate on design a few more times than anticipated, you're going to likely have a very different incident team.  It's not always going to be the same people.  

But you want to get ahead of that early on and make sure that everyone has a well-defined role.  

So first begin by examining what area of the project has been impacted.  

Then who is the decision maker?  At Palantir a sensible default for the decision maker is generally the product owner role of a project.  You then determine who is the incident manager.  And a sensible default would be the Project Manager, though I've seen it be different roles depending on the project.  

Another part is perhaps the most important, it's what really differentiates a high-performing team from not quite as performing, you're going to determine who is consulted and who is informed.  Anyone who is consulted would be part of the incident team.  So it generally means that they have special knowledge of the tool that you're using or really important information related to this incident.  They would be on the incident team.  Those who are informed progress with meaningful working outside of this incident management process that would advance the project in other ways.  

You can frame that work as in a just-in-case or just-in-time fashion where you're signing a ticket out every hour.  But you do want to have a team that's progressing with meaningful work outside of this Incident Management Team.  

Who manages client communications, who will keep the client informed?  And then you communicate these roles out to the full team.  This provides transparency and it helps to build that trust of your team knowing who is doing what when.  

So a status page for the incident manager to set up what we call an incident status page.  Generally we have this as a template so you're able to quickly set it up in a templated form.  In a Google Doc or in Confluence you want to set this up quickly before the disruption is fully understood so you get this up and running with any information that you have.  You're going to want to include the project team name, the core deciders, and who is the incident manager?  Who is responsible for the notes and the client communication?  

You want to get the message and the updates out there while you're still primarily focusing on solving the issue.  

You're going to want to share this document first and then improve later.  

I can say when I've been the incident manager, I'm generally taking notes very quickly, it's a rapidly evolving incident and people are troubleshooting.  The doc -- you're going to have spelling errors.  You're going to have typos.  The goal is to get everything you can in there and then refine later.  

Use clear, plain language.  And I generally include screenshots.  I find that helpful.  Because as a team is moving quickly, if I'm not able to capture something in writing, I at least have a screenshot showing, for instance, like the error message that appeared.  

And then you want to update with brief, honest, frequent updates.  

I generally try not to let a lot of time, you know, fall between the number of updates I'm providing.  So even if it's like we're continuing to troubleshoot or we're waiting for this thing to run, I continuously update that doc so that anyone who is in it, which may include your leadership, can see the most up-to-date status of the incident.  

This is what Page 2 or 3 may look like.  

So when you're doing the status page, you want to have the most recent updates op top.  So that means all of the older updates are towards the bottom.  The moment you recognize the incident is probably the last thing in this document.  And it works up in terms of timeline.  

Say your company leadership or your client leadership are looking at this doc, the goal is to communicate out the most recent update.  So you don't want to have them scrolling through two, three, four pages to know where this incident stands.  At a quick glance they can see the most recent update on that first page.  

You -- I do tend to capture the date and the time in whatever time zone I'm in.  And then I retroactively go back and put in the Central Time Zone, which is where what Palantir operates under or I'll put it in the time zone of the client.  But I'll generally keep it up to date based on my own time zone, because I'm the one keeping the clock going.  

Engage in the advice process.  This advice process, it borrows from lean methodologies, Agile and kanban and it's a process that you can go through to get all the most relevant and key information presented so that a decision can be made.  So during this time when everyone is on this Google Hangout, the Incident Management Team is on the Google hangout, the incident manager is updating the status page, the decision maker is seeking out the various perspectives.  They are gathering advice from anyone and everyone who would be meaningfully impacted or those who have subject matter expertise.  

The decision maker may form a written proposal.  That could be the outcome of the incident.  So in the case of a client wanting to iterate for months longer than originally scoped, a written proposal is likely the outcome.  Or the decision maker makes the call on how to move forward.  So in the case of a site being down, they would be the ones determining how to get that site back up and running.  

When an incident is resolved, you're going to want to enter that into the status page and fill out as much information as you can.  So you're going to have the resolution.  And then if you have a summary of the incident, you can enter it in now.  That's also something that you can fill out later.  

So next is the resolve.  So you have reached a resolution.  What do you do now?  

First review the status page.  So speaking from experience, mine are usually -- you know, they have a lot of spelling errors because you're moving quickly and taking a lot of notes.  That's the time where you go and clean that up, make sure it all makes sense.  

I generally will pull in -- if I'm the incident manager, I generally will pull in the core decider from the incident and hop on a Google Meet and review the whole doc together.  

The value of this is that you're going back and you can enter in more information.  You can clarify anything that may not make sense that you captured in a note but you weren't certain while it was happening.  This is an opportunity to really refine your incident status page and make sure it's as comprehensive as possible.  

>> I do have a question on the review that you just mentioned.

>> MEGHAN DAVIS:  Sure.

>> How do you wrangle that?  Do they want to wander off in other things and new discovery and other tickets and other issues?  How do you control that?  

>> MEGHAN DAVIS:  That's a great question.  Generally we'll start at the very bottom.  So when the incident was realized.  And this is a one-on-one conversation between the incident manager and the core decider.  And the framing I provide is we are going to flesh this whole document out.  So anything that's in there, you're defining further, you're adding context.  But you want to keep it focus on the goal of we're going to flesh this doc out as much as we can.  

Perhaps they are going -- the team is going off and picking up different tickets or they are minus somewhere else on the incident I will generally step away from this and be like, okay, we will revisit -- actually, I'm sorry, I think I would actually ask the core decider if their mind -- like you really have to put that down so you can focus on refining that.  The more time that passes, the greater chance that you may miss something in this doc.  So I think my approach would actually be like we need to really focus and flesh this out within the hour after something is resolved.  So that you don't forget what happens.  

>> DOUG VANN:  That makes sense, thanks.

>> MEGHAN DAVIS:  Yeah.  

Closing communications.  I will generally write up an incident report.  And it's written in plain language.  As someone who is not a DEV, I'll probably bring in the core decider, have them explain everything to me.  Perhaps to contribute to this doc itself.  But I think it's important as -- if you're creating a client-facing document that it's in plain language so there are no technical barriers to your client understanding what happened during the incident.  

The information that should be captured in this incident report is a summary of what happened.  A timeline.  I generally will put the whole status update or the status page into the incident report.  You're going to want to do a Root Cause Analysis of why did this happen.  

And then recommendations, how can we prevent this from happening again?  How do we move forward with the work that we were doing?  

You want to share with your clients, your project team, key stakeholders and if it's part of your process, to share with the whole company.  

I do find that sharing incident reports out with the whole company, including leadership, provides a layer of transparency into what happened.  It shows how quickly you were able to resolve an incident.  And it also fosters an environment where anyone on any project are empowered to raise their hand knowing that there's a process in place to help fix something.  

To discuss the incident, we'll generally hop on a brief call and walk them through the report.  And I also generally will walk them through the full process that we go through so that they understand kind of the timeline we were working in or the process we were working in to resolve the incident.  

The incident report we put on Palantir -- we add our logo, we add our colors, we add the team name.  And then I normally will mark it internal until it becomes client facing.  

We have the summary, the timeline, the root cause and then our recommendations.  

Having a space in Confluence purely for incident management per project has been very helpful.  So this is an example of a template that we created on one of our projects that I replicate any time there's an incident and then just add in the information or add in the link.  In this example, it contains the incident status page link, a link to the postmortem, which we'll discuss, any client communication, the incident report and then any notes to meetings that you may have had with the client.  

This serves as kind of a single source of truth for all of the documentation related to a particular incident so that you don't have to go dry diving to find all of this information.  

Now we're getting into the final phase.  When the teams I've been on, what I've noticed is this is generally the one that gets dropped, the reflection period.  So you've reached the resolution.  Your site is back up and running or you have a plan to move forward with your client within the timeline and scope, taking a moment to reflect on the process that you went through and the incident that was found allows space to talk about how we may not encounter this incident again and this is definitely the area that defines a high-performing team from those who are not as high performing.  

So why reflect on the incident?  Generally it's like a day or two after.  You give your team space to free up their mind or go and get lunch or go for a walk, they come back and reflect and it encourages support and collaboration among team members.  It fosters communication, growth and openness.  It increases the chances that a teammate will come forward if they recognize an incident in the future.  It's, again, creating that environment and that space where a teammate may raise their hand if they see an incident or if they notice that something has gone awry.  

>> DOUG VANN:  Sorry o to interrupt.  I just checked with the other rooms and no one is having the problem we are having here of losing the first couple of syllables we're thinking whichever mic you're using currently which might be the mic attached to your headphones I don't know but if you can possibly switch the mic I don't know if you can do that quickly on the fly.  But otherwise if you have pauses if you can start the next sentence with a little bit of verbal garbage.

>> MEGHAN DAVIS:  Let me see if I can switch my mic.

>> DOUG VANN:  We appreciate that greatly.  Loving the content.  I'm saving some questions for later.

>> MEGHAN DAVIS:  Can you hear me.

>> DOUG VANN:  That's perfect.

>> MEGHAN DAVIS:  Okay.  Awesome.  It must have been the AirPods.  Okay.  Thanks for the heads-up.  

Let's see.  So now we're in the moment of reflection.  The way that you reflect what I found to be quite effective is by engaging in a blameless postmortem.  As a project incident, as the incident manager, I'll generally spin this up about a day or two after the incident.  Not enough time for people to forget what happened but it is enough time for people to go get lunch and to step away from their computer, to step away from the incident.  The framing for a postmortem is that every single teammate is beginning with the assumption that we all acted from a place of good intent.  The communication is fact based.  It's honest.  But it's subjective.  It's not a session for finger pointing or blame.  It's really coming together as a team.  

During this time, which I've found is generally you want to block off at least an hour.  You may need an hour and a half.  You're going to reflect on what happened.  Why it happened.  Look at how the team responded.  And then identify what can be done to prevent these repeat incidents.  So if a site went down because we were doing a particular deployment, how do we move forward so that sites don't crash if we move forward with that deployment for a client.  And then you're going to identify ways that you can improve your future responses.  

So if you're looking at both the incident itself and the process that you went through to resolve the incident.  

This is documented in a shared space, such as Confluence and then I like to share it out with the full project team and again with the full company I think is really important to create that atmosphere and that environment for continuous learning so that we can all become better as teammates and as a collective.  From learning from one another.  

So I think that actually wraps it up.  Once you're at the -- once you've completed your blameless postmortem, you've shared out your documentation, you can close out that incident as complete or resolved.  Thank you.  

I guess we can open it for questions, if anyone has any.

>> DOUG VANN:  Well, I'll take advantage of my host status and just jump right in.  

You mentioned plain language, using plain language with the client.

>> MEGHAN DAVIS:  Uh-huh.

>> DOUG VANN:  So I'm curious, what is your comfort with the Drupalese that someone would speak?  And to what extent do your clients prefer or not or not prefer to use any of that Drupalese?  

>> MEGHAN DAVIS:  Sure, that's a really good question.  I've found that depending on the client, either you -- some of our clients can speak Drupal, they understand it, they are webmasters.  Some, depending on the level of leadership, also, may not choose to speak in Drupal.  They want to know why their site went down.  And my general practice as a non-developer is if -- is if I have a developer explaining something to me and I don't understand it, I try to go back and really ask all of the questions so I can better explain it to the client.  

Some of them can go deep and talk Drupal.  I would say the majority just want to know what happened, why, and how you're going to move forward from it.  Getting into the Drupal I think is secondary sometimes.  So yeah, I think that's -- I think keeping it plain language is really important.  Unless you have a deeply technical client.

But even in that case, I can imagine the client -- the information that they want is like the why and the how.  And how are you going to prevent this from happening again.  

>> DOUG VANN:  Excellent and with regard to yourself and the other PMs at Palantir how technical do you feel your PMs are if the client wants to get all technical.

>> MEGHAN DAVIS:  That's a good question.  So I generally will partner with a product owner who is a subject matter -- who has subject matter expertise or we have a role that's called the architecture owner on a project.  So we'll partner together and approach the client together.  

If it's something that's deeply technical.  I can speak to the high level, this is what it means for your business.  And then the architecture owner or the product owner can go deeper if the client wants to speak, you know, deep Drupal topics.  

>> DOUG VANN:  Excellent, thank you for that very much.  

>> MEGHAN DAVIS:  Are there any other questions?  

>> My name is Scott, thank you for this fantastic presentation.  I do have a question.  

When you're in the heat of the incident, whatever it may be, how do you -- I guess a few questions around that.  How do you capture I guess the time spent would be the most practical first part of that question.  And then the second is, do you capture almost like a stream of consciousness of yes we have this incident, we think this is our cause, this is what we're looking but when we looked there we didn't find it and then we went to this other area and looked and resolved it there.  Do you kind of keep a narrative of almost I guess kind of how you went from incident identification to resolution?  Do you keep that like stream of consciousness so that you're kind of showing your homework?  Or is it really just focused around we did this to resolve?  

>> MEGHAN DAVIS:  Good question.  

So my personal approach is to capture all of that in the status page.  And it's not so much to prove all of the steps that you went through.  But it's to capture that information for future learning like we tried this step, it didn't work.  We tried this step, it did not work.  

So I do try to capture all of that comprehensively in the status page.  But I'll generally mark it as when we're talking to a client, we talk about -- if they want to talk about all the steps we took we can but generally we're pulling out the key highlights like we tried this first troubleshooting step and it didn't work, what did work was this and that's what we advise moving forward.  

The stream of consciousness sometimes is hard when you're in the movement and stuff is moving fast to capture all of that written word in the status page.  And I think that's where I've found that screenshots are really helpful.  Because even if you can't keep up with writing what's happening at a specific time, you can at least screengrab what someone is sharing on their laptop or in the Google Meet.  Put that in the status page and revisit it to fill that information in, like flesh out kind of what was happening.  

So I can say as an incident manager, there are times when you're capturing stuff where it's like there may be a moment where it moved so quickly you weren't able to write down exactly what happened but I think a screengrab would be helpful there.  

And then when I go to the client, I bring out the really important pieces in the troubleshooting steps that we took.  Probably not talking about everything we did.  Unless they want that.  

>> Okay.  Great.  Thank you.  

>> DOUG VANN:  Forgive me was that Geoff's question that I also see in chat or have we not addressed that question?  When you have the incident report that you are updating live, at that point is the doc just shared internally or with the client, too?  

>> MEGHAN DAVIS:  That's a good question.  That's one I'm actually -- I've been going back and forth on.  The incidents that I've managed with Palantir, the doc has been kept internal.  But they have also been generally like outside of the client's working hours so we're emailing them, keeping them informed.  But there hasn't been a time where I've shared the status page with the client.  But like those moment-to-moment updates, though I'm totally open to that being a possibility.  

The incident report for me is where we provide that client-facing document of here is what happened, here are the steps we took to troubleshoot.  The key steps we took to troubleshoot.  And here is our recommendation moving forward.  So that -- I've always thought of the incident, the debrief doc is what you share with the client, we keep the status page internal but I would be open to talking through that internally if that gets shared out with the client.  

I'm curious if anyone has any thoughts on that.  

>> Depending on the client.  

>> Yeah, I think it would depend on the client.  I think that it may need to be filtered a little bit for messaging.  Because you don't want to be in the -- you want to have that blameless moment.  But you don't want -- you also want to be transparent and say, well, their DevOps person blew away the production database.  We need to call it out.  But we don't want to like say, so-and-so really you know caused this problem.  

>> MEGHAN DAVIS:  Yeah.

>> You want to -- I guess want to turn it to be more focused on prevention next time and keep it the production database was deleted rather than, you know, Scott deleted the production database.  

>> MEGHAN DAVIS:  Yeah.  And I think I'm with you on that.  

The way that I've done the status page is like comprehensively everything that's happening in the moment.  And then if I'm relaying messages to the client, it's like those key moments that they need to know.  Like we are taking the steps to troubleshoot.  And like every about ten minutes or so, you'll decide on the frequency as a team.  But you are updating your client.  But I generally keep that client communication separate from that status page.  Because I do -- like I can imagine that it could get overwhelming to keep up with.  And Scott, I think that was you that said keeping it blameless.  And not pointing fingers.  I think that's a really great -- that's a great point.  

Has anyone had experiences with an incident at their company?  Does this process resonate?  Or have you seen deviations?  

>> I think anybody who says no is lying.  

(Chuckles).

>> Yeah, I think sometimes we -- I've seen projects get tripped up.  And it's not necessarily in like the site is live incident detection.  It's like during like the building phase that there's a conflation of an incident and a risk.  

Like the client just hypothetically, the client missed the sign-off of the final design.  You know, and we say, oh, that's a risk.  Well, in the true sense, that's an incident.  That deadline was missed.  That is an event.  

And it does, you know, create risk.  But just the act itself of not signing off on the design is the incident and not the risk.  

>> MEGHAN DAVIS:  Uh-huh.  So in that case, how would you -- would you apply the incident management process to that scenario when a client misses like authorizing or signing off on something?  

>> I think not in the, you know -- our team has to resolve it.  But I think it is one of those moments where you document it as an incident.  And say, the resolution is, you know, so-and-so or this team needs to sign off on this.  And until that's done, development can't start.  And when you have to have those conversations around budget and timeline, you have that not as a point of contention.  But just to like show this is something we raised.  Here is -- it was supposed to be signed on Monday.  And it was the following Tuesday, a week later, that it was signed.  So we're a week behind.  

>> MEGHAN DAVIS:  Yeah.

>> And I feel like clients are really receptive to again, the blameless thing of, you know, Joe didn't sign this so we couldn't get started.  Not doing it that way.  But more of we're waiting for the sign-off so that we can begin our part.  And that's why we're -- the trains have stacked up.

>> MEGHAN DAVIS:  Uh-huh.  Yeah, I think that's a really good point.  And I've experienced something similar where a client request came in that it was so big that it could have disrupted the timeline and the budget and the scope drastically.  So we applied this process.  And we were able to come to a resolution.  With the client in a way that was blameless and everyone felt included and involved in that decision.  But you're certainly right, it's not a risk.  It's an incident at that point.  

Okay.  Are there any other questions or anything you want to talk through?  

>> DOUG VANN:  Actually since we have some time here this is stump the PM now, I like this, if you like it.  

How involved are you in the requirements gathering process at Palantir?  

>> MEGHAN DAVIS:  Sure.  So that's a good question.  The way that our teams are shaped currently, we have the team lead, which is the Project Manager.  The product owner, which is generally a Subject Matter Expert.  They come in and they play the role of the product owner.  An architecture owner.  And a design owner.  

And then we have the rest of like the teammates.  And I would say it's a combination of the product owner in combination with the team lead or the architecture owner who does that requirement gathering at the beginning of a project.  But we do work within an Agile environment.  

So as more information is learned, we're able to adjust to accommodate that and to shift to different priorities as best that we can.  But I would say that core team of the product owner, the Project Manager and the architecture owner meet at the beginning.  Like that's the core team that is helping to define those requirements and helping us shape the priorities for the project.

>> DOUG VANN:  And in that combination of people, maybe I missed it, but who is the most Drupal savvy person in there?  

>> MEGHAN DAVIS:  It depends.  So sometimes an engineer will serve as the architecture owner or as the product owner.  

Sometimes a designer is serving as the product owner.  So there's a shift based on roles and who can fill them.  My experience has been that the architecture owner owns a lot of the Drupal knowledge.  The product owner generally has a high sense of what is happening, as well, that can go in and test functionality and participate in that way, as well.  

So like an engineer I guess could fill different roles on a project.  It could be the product owner or he or she could be the architecture owner.  So we don't have kind of -- your job title wouldn't necessarily dictate kind of the role that you're going to play on a project.  So anyone with Drupal experience could be in either of those roles.  

>> DOUG VANN:  Thank you.  

>> MEGHAN DAVIS:  Any other questions?  

Well, I'll share the last -- what was that?  

>> DOUG VANN:  Exactly, there we go.

>> MEGHAN DAVIS:  Yeah.  So I encourage everyone to please provide your feedback.  The top rated sessions will be captioned.  Thank you to Clarity Partners.  And I put up my session name there is 6331.  

And then also a plug for Contribution Day this Saturday from 10 a.m. to 4 p.m. central.  You don't have to know code to give back.  If you want to learn more information about Contribution Day, you can visit the MidCamp Website, MidCamp.org.  

And I think that wraps up the session.  So thank you, everyone, for joining.  And thank you for all of your thoughtful impacting questions.  

>> DOUG VANN:  Thank you, Meghan.  Thank you very much.  Thank you.  

(Applause).