Some years ago, the City of Calgary conducted a multi-million dollar effort to engage citizen’s in the development of a new transportation strategy. The Go Plan used consultations, focus groups, workshops and polls to engage citizens and gather feedback.
Shortly before the release of the final report, someone thought it might be a good idea to conduct a more scientific survey of Calgarian’s. It was an afterthought, a last minute effort to ensure i’s were dotted and t’s crossed. A professor of transportation planning at the University of Calgary was quickly contracted to conduct the survey.
The results completely contradicted what was already in the soon-to-be released Go Plan report, setting up a potential public relations disaster. Disaster was adverted by simultaneously releasing and dismissing the scientific effort as a small, insignificant part of the larger consultative process. Thus, the unbiased picture of what Calgarian’s wanted in a transportation system was ignored, while millions were wasted meeting the needs of specialized interests.
Gathering citizen feedback to improve public services has all but disappeared. Now, survey research is conducted for the public relations purpose of ensuring a positive result. This is the Billy Beane, Moneyball strategy of; “If you get the answer you’re looking for, hang-up”. People are suspicious of these surveys, but are unsure as to how things are manipulated. Well, here’s how.
Step one
Cherry pick the sample. This ensures those participating will give you the answer you’re looking for. You have to be subtle about it though. Surveying only cycling enthusiasts on bike lanes is too obvious. Instead, hold public engagement sessions on topics appealing to cycling enthusiasts like; ‘Bike lanes, should we have them?’ The meeting will be overwhelmed with those answering ‘Yes!’. This enables sponsors to claim the session was open to both positive and negative feedback while knowing only those in favour will show up.
An example is the engagement process concerning Calgary’s new central library. The process assumes that Calgarian’s want a new library, that libraries have an important role to play, and so on, ensuring that responses will be heavily biased in favour of current library users and supporters. Administrators will get the answers they’re looking for and will follow up with claims they listened using an ‘extensive’ public consultation process. (Consultations are usually described as extensive but rarely as unbiased.) Unfortunately, this public relations approach to engagement ensures that the people the library needs to hear from, to grow and remain relevant, are largely excluded.
Step two
Design the questions carefully. Again, you can’t be too obvious about this. Subtly implying something for nothing will generate positive responses. “Are you in favour of more police on the streets?” “Sure, who isn’t?’ ‘Would you like a new central library?”‘You bet.” Missing is any hint of cost.
Vague, general questions also elicit more positive responses than specific questions. “How do you feel about the quality of city services?” will yield more positive results than; “Considering your last interaction with the city, how would rate the level of service you received?”
Plus, vague questions enable what Darrell Huff, in How to Lie With Statistics, calls “the semi-attached figure”. This is measuring one thing and drawing conclusions about another. For example, food quality at Alberta extended care facilities, particularly in rural areas, has recently come under fire. Criticism has been deflected by noting that customer satisfaction in facilities is high. Perhaps, but what about customer satisfaction with the food and in rural areas?
Step three
This is the best trick of all and a trade secret. If you want to guarantee positive results on your survey, use a small, five point scale, and report the top two box scores – -where a score of four or five is recorded as “positive” or “satisfied”. Where’s the trick?
It’s in the phenomena called positive response bias. People tend to answer to the positive, even when their attitudes are neutral or slightly negative, answering four when three better describes it. A tally of “satisfied” respondents, therefore, includes people ‘neutral’ or “slightly dissatisfied” ensuring an inflated positive result.
Check this out for yourself. Download a citizen satisfaction report and see if it doesn’t rely on a top two box score summary on a five point scale. By the way, the smaller the scale, the stronger the bias. Citizen satisfaction skyrockets with four point scales.
So there you have it. You’re satisfied with public services and now you know why – the survey says so and it was designed to say so from the start.
By Robert Gerst, Columnist, Troy Media
