Newsletter Archives

Validating Competencies For 360-degree Feedback

We periodically receive inquiries similar to this recent request:

I'm interested in learning more about what you have to offer in terms of surveys/survey items related to leadership competencies. I'm currently working with a client whose senior management team has identified several leadership competencies that they would like a 360 systems designed around. I'm interested in questions that specifically relate to these competencies, that have been well researched, tested, and validated. I'm also interested in reliability estimates for questions related to a single competency.

Can you provide this type of service/information?

I asked Jerry Talley of Organizational Diagnostics to respond to Sylvia's inquiry. Jerry is the creator of Problem Solving 2.0, a powerful new approach to diagnosing and responding to organizational problems. I asked his permission to include his informative and practical reply in our newsletter this month.

Dear Sylvia:

You asked about valid and reliable questions around particular leadership competencies. I imagine you are looking at building an instrument for your client and want to make sure the questions will provide a solid basis for decision making.

While your concern for validity and reliability are the traditional questions to pursue, my experience suggests another area is more critical for securing the best outcomes: the decision process at the end of the 360 feedback. Most executives are quite comfortable with questions that have face validity alone. Where they are most concerned is with how the data will be used...and they're right to worry about it.

Feedback data can be easily misused. And even when used appropriately, it is easy to distort the data collection process. Agreements up front on appropriate use and how
to ensure the integrity of the data will have a much stronger impact on the final result than time spent on crafting the most sophisticated set of questions.

I find that using scales of several questions is the best way to ensure validity. So a dimension such as " Exemplifying teamwork" is made up 3-4 questions that come at the same topic from slightly different perspectives. For example, I might ask...

1. This person demonstrates by his or her own behavior that they are able and willing to collaborate with their peers.

2. This person requires that their direct reports work together collaboratively.

3. This person makes sure that people are rewarded as much for their teamwork as for their individual achievements.

4. This person actively solicits a range of opinion, even when there is an emerging consensus of thought.

Statistical analysis of the data (Cronbach's alpha) will tell us if these questions are behaving similarly and can be justifiably taken as an indicator of a single concept. Prior to administration, those being evaluated can be queried if these questions capture their notion of supporting teamwork.

But the real question is how the data will be used. Will it be public? Or only given to the target? Will part of their bonus be contingent on their scores? Or on their improvement in scores? Or will their improvement plan become part of their performance contract (independent of the data)? What resources are available to the target to improve their scores? The real goal here is not just measurement, but improvement. What's the mechanism to support change?

If people will be compared to their peers, how will those comparisons be interpreted? For example, someone overseeing a sales group has a much lower need for teamwork than someone overseeing a design department. Demanding equally high scores would seem foolish.

In summary, I'd focus on negotiating the understandings for use of the data. Once that structure is in place, you will probably find that the questions are continually tweaked and adjusted over the years as new themes become relevant. For example, right now
supporting efficiency is more important than supporting innovation; in a year or two it could be the reverse.

Jerry Talley of Organizational Diagnostics
(650) 967-1444
2/20/2003

GET HELP ON 360-FEEDBACK

Call us toll-free, 877-666-2486, to discuss how we can help with your 360-degree feedback program. Or complete our contact information form to let us know how we can reach you.

Share this information

You are welcome to quote this article or any of our newsletters, or to create a link on your web site to the archived version. Please notify us of your use of this content, and include the following information:

Reprinted (or excerpted) from archives of Gainen Surveys LLC, http://gainensurveys.com.

For web publication, lease send the link where this content may be found to info@gainensurveys.com.