Policy Analysis Methodologies

Let me start by saying I am a little confused. First, I think that Chuck believes that it is not possible to be objective and empirical in the social sciences (I agree!). Then, Chuck says to explain how I will objectively and empirically analyze the issue I am proposing for my final issue brief. Ok … am confustigated. Anyway, let me talk about policy analysis techniques in the context of the readings and stop worrying about this oxymoron.

I already know from taking Evaluation of Networked Information Systems that Chuck is a big fan of outcomes measurement, and I see that Anderson (2005) agrees with him about its utility. Yes, it is important to identify and measure policy outcomes, or the consequences for society, but as Anderson notes, this is also really difficult. The consequences of a policy for society are hard to operationalize and hard to measure, never mind the inestimable number of years you might have to wait to see what the consequences has been for society.

Yesterday in Seminar in LIS Education, Kathy Burnett talked about how you can’t evaluate an issue while it is ongoing by looking at current literature trends. Instead, you have to wait 10, 20, or more years to see how the issue played out. (This was about the iSchool movement for those of you not taking that class.) The same can be argued about policy issues. Yes, policies affect us in the here and now, and we can certainly look at past policies and see what impacts and outcomes they have had SO FAR, but to determine the consequences a policy has had on society, we would really have to wait decades, or maybe centuries.

Causality is a whole other issue – Anderson reminds us that causality is extremely difficult to determine in social science research (2005), and we know Chuck’s view on this issue, and then I read Pawson and he’s explaining some “model of generative causal explanation” in which he combines outcome patterns, generative mechanisms, and context to determine causality (2006). Leaving aside the dubious nature of the generative mechanisms, is it really enough to combine three measures and POOF you determine causality? I kinda doubt it.

Not to brown nose, but I was much more impressed by the descriptive assessment described in McClure, Moen, and Bertot (1999). “Descriptive assessment relies primarily on existing policy and literatures, and is the process by which it is possible to describe and analyze a policy area” (p. 314). No determination of causality, just using a multi-method approach to gain a multi-viewpoint explanation of what is going on regarding a policy issue/area. Boom. Sounds simple (relatively compared to Pawson’s approach), believable, and DOABLE (really the most important). [Side note: we know for sure that Chuck approves the multi-method approach since it is also lauded in McClure & Yaeger, 2008.]

I especially appreciated the realistic (note that is REALISTIC not Pawson’s REALISM) approach to reviewing related policy instruments – comprehensive OR selective. That’s what I mean by DOABLE. But then of course, McClure et al. say that “Descriptive assessments do not replace empirical study” (p. 328, emphasis in the original) which leaves me in dire straits since I have to identify a method by which I will “empirically and objectively” study the issue I am planning to use for my issue brief. A conundrum.

In case you were confused by what I said and think I enjoyed the Pawson reading, let me clarify a little here. I was extremely suspicious of any methodology espoused by someone who names his kids Rhino and Rosebud and likens evidence-based policy to a tryst. Never before have I seen such romantic language in a methods book –

“Evidence-based policy is much like all trysts, in which hope springs eternal and often outweighs expectancy, and for which the future is uncertain as we wait to know whether the partnership will flower or pass as an infatuation.” (Pawson, 2006, p. 1)

References

Anderson, J. E. (2005). Public policy making, 6th ed. Boston: Houghton Mifflin.

McClure, C. R., Moen, W. E., & Bertot, J. C. (1999). Descriptive assessment of information policy initiatives: the Government Information Locator Service as example. Journal of the American Society for Information Science, 50, 314-330. Retrieved January 29, 2009 from Wiley InterScience database.

McClure, C. R., & Yaeger, P. T. (2008). Government information policy research: importance, approaches, and realities. Library & Information Science Research, 30, 257-264. Retrieved January 29, 2009 from ScienceDirect database.

Pawson, R. (2006). Evidence-based policy: a realist perspective. Thousand Oaks, CA: Sage.