Making change sometimes involves an elaborate public discourse and preparation of affected stakeholders and in
Ontario the discourse is towards getting people in the public sector to do more
with less. The latest target was
drawn to my attention by Alex Usher’s Higher Education Strategy Associates (HESA)
morning bulletin, which featured the preliminary report by the Higher Education
Quality Council of Ontario (HEQCO) on The Productivity of the Ontario Public
Postsecondary System.
To summarize: “Ontario universities have received increased absolute levels of funding and funding per student since 2002. Nonetheless, they are teaching more students per full‐time faculty member with less money per student than all other Canadian provinces. They also lead Canada in research profile and output. A pilot
study of four institutions suggests that full‐time faculty teach approximately three and one half courses over two semesters. On average, faculty who are not research intensive, as defined by the universities themselves, teach a little less than a semester course more than those who are research active.”
So it would appear that the post-secondary sector in Ontario is doing
more with less and is “already quite
productive” but naturally we can do better so: “further critical information is required to better assess productivity
and identify the most promising steps for improvement, including: measurement
of the quality of education, especially whether desired learning outcomes are
achieved; better information on graduation rates; more input from employers on
their satisfaction with the knowledge and skill sets of postsecondary
graduates; more detailed measurement of relevant information in the college
sector, both within Ontario and across Canada; and greater detail on the
workloads of university faculty.”
The workload issue is going to be a big one especially given that one of
the tables highlighted by Alex Usher – Table 9 – attempts to compare faculty
workloads at four institutions (Guelph, Queen’s, Laurier and York) according to
whether they are in science or social science and humanities and whether they
had research output (defined as a grant or publication in the 2010-2011
period). What they found out was
not particularly illuminating to me though Alex Usher felt it raised a question
of what all the non-research active professors were doing. (He also asks why
Quebec gets 39 percent of research council grants – that is a more interesting question but I'll let Alex Usher try and answer it at some future date).
The Ontario government would like to generate even greater future
productivity increases by changing “the design of the Ontario postsecondary
system and how it is funded. For individual institutions, the greatest
productivity opportunities may lie in greater flexibility in the distribution
and deployment of their faculty resources, particularly in the distribution of
workloads of individual faculty taking into account their relative
contributions to teaching and research.” I think the Ontario government would like to see an
increase teaching loads, but in a redistributive fashion so as not to harm its
lead in research productivity.
That is: it would like universities to raise the loads on “non-productive”
faculty and either maintain or perhaps even lower them a bit on “productive”
faculty. I am also assuming that
means more than simply saying something like all research in Ontario is going
to be done at the University of Toronto and everyone else must teach more.
However, before you can do anything, I think you seriously need to beef
up the data given what I think the ultimate purposes of the Ontario government
seems to be. You cannot just compare the average teaching loads in science and
humanities and social sciences across four universities. True, they say this is preliminary but
the fact is these preliminary tables, figures and statements have a way of
sticking around in popular memory and then become stylized facts that generate
“policy changes”. You need to do
much better than that.
So, my points specifically with respect to universities and the
measurement of productivity:
- You need to have data on teaching and research on
all the Ontario universities and not just a few. - You need to have data on teaching and research
across all disciplines and faculties at the university and not just a few. - You need to measure faculty research output for
more than just the last year – why
not the last seven as SSHRC does? - You need some type of standardized measure of
research output by institution/discipline/field as well as its impact via
citations. (Ask Alex Usher about this.
I’m sure he will recommend something based on Google Scholar. The report
uses HESA H-index numbers to compare Ontario universities to other provinces so
I’m sure it can be done). - Research grants and funds for sponsored research
are not research outputs. They are
inputs. Over the long term they
are indeed correlated with research success in those fields that require funding to
accomplish research but they are not an output. They are increasingly being considered an output measure because
universities are desperate for cash and generate revenue by taxing grants and
contracts. - Simply comparing course loads per faculty member is
misleading. You will need to
incorporate some measure of class size also. A half-course difference in teaching loads can be significant
if the class has 300 students. - Workload is not just research and teaching anymore. Just ask any anyone who has gone thorugh a "provincially mandated" quality assurance review in order to justify quality.
And remember, in the university sector, the assorted bureacratic infrastructure for measuring productivity in itself often involves increasing faculty members workloads so we can kill two birds with one stone! Now that's productivity! Enjoy!
Worthwhile Canadian Initiative 
While the workload of teaching a small versus a big course can vary, I’m not sure that institutions make that much distinction themselves. I know that my institution doesn’t; a professor is expected to teach some number of courses per year, and the size isn’t really relevant (but the large first- and second-year classes are almost invariably taught by lecturers while the professors teach the smaller third- and fourth-year ones).
Ontario colleges use a Standard Workload Formula, which takes into account class sizes, marking type (in process, automated, essay), and whether you’ve taught the course before/recently.
We’ve just been through the research assessment process in New Zealand. This is only an assessment of research; the government hands out money for teaching based on student numbers (for domestic students — international students pay fees), and institutions decide internally how to allocate funding and workload for their employees.
The research portfolio (based on six years’ work) has three categories. The major one, worth about 70%, is ‘Research Outputs’ (4 nominated research outputs to be assessed for quality, plus up to 30 more to demonstrate ‘a platform of research’). These have to be outputs: articles, conference papers, reports, patents, software, etc,etc. The two minor categories, worth about 15% each, are (research-based) ‘Peer Esteem’ and ‘Contributions to the Research Environment’. That’s where you put refereeing, being on conference committees, supervising research students, getting grants, prizes, invited presentations, and a partridge in a pear tree (provided it’s a research-based partridge).
The whole thing is then assessed by a panel assigned to a collection of related disciplines adding up to 400-800 portfolios per panel. Each portfolio is rated independently by two panel members to give two initial scores. They then agree on a second-round scores. Finally, the whole panel discusses the portfolios and assigns the definitive scores at a very long meeting.
I think the assessment of research is pretty good (disclaimer: I was on one of the panels, so I’m biased and there are lots of things I can’t say). But the process is very expensive — both in preparing the portfolios and in assessing them. Also, there’s a good case that the funding formula gives relatively too much to the top researchers, and the process doesn’t even attempt to get a fair allocation of resources to individuals, just to larger units.
Thomas, thanks for sharing those insights. It sounds like a very methodical and comprehensive effort to measure research productivity. One question: how is joint authorship on publications handled in terms of weighting. For example, is the publication weighted equally across all the authors?
Livio: someone ought to do a post on that question one day. (Not me!) The only right answer is 1/number of authors. Any other answer creates perverse incentives as well as adverse selection.
Livio: how is joint authorship on publications handled in terms of weighting.
The assessment is supposed to be primarily of research quality, not quantity. For the four nominated research outputs the researcher writes explanatory comments. These tend to include something about who was responsible for what parts of the research. At least for collaborations within New Zealand, the fact that all the portfolios go to the same panel ensures a certain level of internal consistency in the claims — you’re unlikely to get two people both saying they did nearly all the work for a particular paper.
There is no prescribed method for deciding how to handle joint authorship in the other 30 research outputs; it’s left up to the panel, who at least understand the norms for publication rates, co-authorship, and author order in their disciplines. I don’t think there is a universal answer to this question: Nick’s 1/number of authors also creates perverse incentives, just different ones.
This approach doesn’t scale well: you’d need to narrow the scope of each panel to make it feasible for Canada, let alone the USA, and so you’d lose the cross-discipline comparisons that help precisely where bibliometric approaches do very badly.
@Nick, so a physicist who spent three years of his life away from home at the LHC gets 1/3000th of a paper per publication? My last paper had ~40 authors and I am on a relatively small project.
The number of authors issue is intractable for between field comparisons without some sort of norming approach. Entire fields (like Genetic Epidemiology or High energy physics) would implode into completely unproductive with a 1/n authorship rule. It is true that the current system of counting papers disadvantages fields with high barriers to publication (e.g. Economics) and is very favorable to fields with low barriers (e.g. Electrical engineering).
My guess at the best compromise would be rank people relative to field and level, presuming that the median member of any academic field is trying to be productive. The problem here is entirely people trying to get themselves classes into a lower paper count field using items from a higher count field. My guess is that judgement could handle this but that no fully automated approach is immune to “gaming”.