Outcome measurement resource network

It’s easy for communicators to measure outputs, and almost as easy to measure outtakes. But it requires more effort, imagination, and resourcefulness to measure outcomes: actual changes in people’s behavior and attitudes. Here’s a resource offered by the United Way, called the Outcome Measurement Resource Network. You’ll find more than a dozen white papers about measuring program outcomes that you can apply, with some tweaking, to your communication goals and institutional objectives.

Evaluating changes in knowledge and behavior

Communicators often measure what we do, but too often measurement is limited to outputs. What really counts is the degree to which our communications really change the way people think and behave. So how to combine the use of new communication tools like blogs, wikis, and mash-ups with traditional research methods? Here’s a brief case study written by Glenn O’Neil, Benchpoint Ltd. and International University in Geneva, Switzerland. O’Neil measured the degree of change in the knowledge, attitudes, and behavior of people who attended the LIFT06 conference.

Measures of behavior change included actions undertaken, initiatives launched, and contacts made, as a result of attending the conference. O’Neil used qualitative and quantitative methods to construct an evaluation framework, and drew from methods developed in the fields of public relations and adult learning. Data sources included content analysis of blog posts and a conference program wiki, and a participant survey.

Monitor your web presence

I’ve been using Google Alerts for some time to track the online visibility of faculty and researchers here at WCER and other topics of interest to me professionally and personally. I like Alerts because you have lots of options, including tracking online news stories, or blogs, or news groups, or a comprehensive search of all the above. You get a notification email once a day, once a week, or as it happens, depending on your choice. You register for a free Google account and start keying in the phrases you want Alerts to track, for example “teacher salaries” or “autism” or “educational podcast” or “educational technology”, then you get a separate email for each subject you track.
If you want to try it out, here’s the link.
Happy hunting.

Monitoring your web presence

I guess we pretty much all agree that it’s important to know what’s being said about your organization on the web. How many sources do you check? Google Alerts? Technorati? Maybe you have time to go beyond that and track 5 or 6 or 6 sources regularly? How about 33? Here’s a list of tools you can be using now to track your organization’s web presence. Thanks to Shel Holtz and Constin Basturea for their work.

Webinar: Measuring interactive media

My friend Dan Karleen and I will co-host a webinar April 3, “Measuring the Effectiveness of Interactive Media.” The 90-minute session will include an overview of new media, thinking through communications planning, assessing the success of a new media presence, and evaluating outcomes. We’ll chat with participants during and after the session.

Assessing organizational communication

Every time two people communicate, they not only exchange information but they also build, maintain, or destroy a relationship between them. Multiply each communication times the number of people in your organization and the implications are staggering. Cal W. Downs and Allyson D. Adrian view organizations as political-business arenas in which competing interest groups vie for control over resources. So it’s no surprise, they say, that a major goal of an organizational communication audit is to understand the underlying structure of an organization’s communication flow—and that flow doesn’t always resemble what the org chart suggests.

Cal W. Downs is professor or organizational communication at the U. of Kansas, and Allyson D. Adrian is adjunct assistant professor of management at the McDonough School of Business at Georgetown University. In Assessing Organizational Communication: Strategic Communication Audits (Guilford Press, 2004, 292 pp.) they say that an organizational communication audit must collect data in a well-planned and logistically defensible way; analyze data in a scientifically and pragmatically justified way; and interpret data in a qualitatively integrated manner.

The success of these steps depends on involving the client in the entire process, they say. Without client involvement, not only will many recommendations be irrelevant and misleading, but also the client will not “own” the recommendations and the analysis will become just another consultant’s report on the shelf.

Investigators have found that managers are more likely than workers to think that downward communication is taking place. There has long been a difference as well between management’s perception of what employees need to know and what employees say they need and want to know. Good downward communication is not limited to immediate work assignments; it integrates people into their environments.

Communication should be thought of as an ongoing, dynamic process – not a static, linear, finite phenomenon. What the organization’s managers need to do, and what auditors can help them do, is to understand how communication is an integral part of achieving the organization’s objectives, rather than a separate function.

When auditors design a final report it’s crucial to determine how the feedback can be given effectively to these people under these circumstances. Observations must be put into some context historically, organizationally, industrially, or economically.

As interaction patterns stabilize in an organization, they develop into structures called communication networks. Auditors investigate formal and informal networks to learn how organization members process information. Network analysis allows auditors to identify information pathways, to determine real or potential bottlenecks, to determine how communication links match the needs of the task processes, and to analyze the roles that specific people occupy.

The authors mention that new technologies affect how communication networks form and operate. As examples, they mention how the introduction of email and Lotus Notes gave people contacts in their organizations that they never had before. Of course, since this book was published (2004), social media, including blogs, news feeds, wikis, and social bookmarking have become influential in the workplace, and will complicate communications audits even further.

Communication audits for organizations

“Communication is what happens to organizations while they are busy making other plans.”

Communication audits are powerful tools that can revolutionize the way an organization communicates its employees and with its external constituents. A communication audit strips away myths and illusions about how well your organization communications and, it well done, provides an accurate diagnosis of the organization’s communications health.

The Handbook Of Communication Audits For Organizations, Edited by Owen Hargie and Dennis Tourish (Routledge, 2002, 2002, 2004), provides an exhaustive look into the many ways audits can be planned, executed, and evaluated. Articles by several contributors examine the strengths of various tools including the questionnaire, the interview, the focus group, and data collection log-sheets, among others. Several studies examine audits of a healthcare organization, a paper mill, a catholic disocese, and a major beverage company, among others.

External audits provide a rounded picture of the communication climate facing a given organization. They provide a standard against which the internal operations can be assessed; they measure outcomes that are significant to the organization; they monitor the social, political and business environment; and they allow one to compare consumer attitudes towards an organizational culture with what employees think is an ideal culture.

Whether internal or external, audit strategy should be ongoing, describing the business challenges that exist, their relationship to communication variables, and the best practices the organization is attempting to employ.

The editors note that, because technology is changing the ways organizations communicate with their publics, communicators should keep abreast of the impact of those changes. Organizations are increasingly called on to provide customers with access to information resources. The lines between internal and external organization parts are growing thinner.

But rather than trying to force communication to the top of management’s already crowded agenda, the editors advise, communication strategy should be linked to what is already dominating that agenda.

Communications audits for orgs

How can an organization evaluate its internal and external communications to improve its effectiveness?
The Handbook of Communication Audits for Organizations (Routledge) explores research, theory, and practice in the field of communication audits. The editors, Owen Hargie and Dennis Tourish, propose what they call an ‘action framework’ that integrates audits into the process of developing communication strategies in organizations. In 16 chapters, they and their many contributors expand on the following main points:
1. Transforming communication requires time and resources.
2. People generally welcome the opportunity to discuss their own communicative performance.
3. A communication strategy should focus overwhelmingly on changing the behaviors of key people.
4. Feedback is the key.
5. Persistence and fresh vision are vital.
6. Measurement is indispensable.

Books about comm measurement and audits

I’m reading a couple of books about measurement that offer theoretical and practical value to me as a communicator. Communications Research Measures: A Sourcebook (Erlbaum, 2004) brings together a variety of scales that measure a number of important communication measurement scales, instructions for administration and scoring, and information on validity and reliability. The approximately 60 scales are grouped into the categories of instructional communication, interpersonal communication, mass communication, and organizational communication. They’re drawn from the areas of interpersonal, mass, organizational, and instructional communication.

The Handbook of Communication Audits for Organizations (Routledge, 2002/2004) is aimed at students of organizational communication and communications managers. The first two chapters treat communication’s relationship to organizational success, and auditing communication to maximize performance. The sections that follow include articles on audit methodologies and several case studies. This is a gold mine. I’ll share more about this as I work my way through it.

Take the test: measure your ‘conversation gap’ and more

Use this interactive page provided by Hill & Knowlton to discover a number of interesting things about your organization’s presence in the blogosphere, including:
The conversation gap (the gap between the total number of conversations about a product category (e.g., education) and the proportion which mention a company or brand operating in the category);
Your ‘equity share’ (the ‘equities’ of a brand are those topics being mentioned in conversations about a brand, so your equity share corresponds to the frequency at which each topic is mentioned); and
Your share of the buzz (measures the share of buzz for different companies in the same sector).
The resulting charts point to areas you may want to work on as you continue to formulate your communication strategy.