Saturday, June 14, 2014

My main criteria for research quality

How does one evaluate the quality of ones own research and that of others?
First, acknowledge that this is subjective. Different people have different values, personalities, and background. Here are my values. Most are somewhat interconnected. Not only is the selection of criteria below subjective but how one evaluates each one is subjective and personal.

Validity.
Ultimately science is about truth and reality. How confident can I be that the results (but also conclusions) actually are correct? Are the results likely to be reproduced? Are they consistent will earlier work and general principles?
Of course, one can never be completely confident, but particularly with experience, once is able to weigh up the probability that results will stand the test of time. The more extra-ordinary the claims the greater the evidence must be.

Reality.
Ultimately, theories must have something to do with real materials and be experimentally testable in some broad sense. Research that claims to have technological relevance must have some chance of being actually realisable on the time scale of a decade or two and not wishful thinking and marketing [science fiction].

Concreteness.
There is a well defined problem. The research presents a well defined answer. For some papers/talks/grants I encounter it is not clear what the problem and answer are. It just seems like a bunch of measurements or a bunch of calculations. Uncontrolled approximations and poorly understood experimental techniques have some role, but do not have the same value as definitive techniques and results where one actually knows what one is doing.

Importance.
One measure of this is the breadth of application [or scientific relevance] and implication. Will this change how we think about a field? Will it make possible new measurements or new calculations? Does it solve an outstanding problem that has previously confounded excellent scientists?

Clarity.
For some work I am just never really clear what is being done, whether it is valid, or why it is important. On the one hand, this might just be because I don't have the relevant background to appreciate the work. On the other hand, I fear some scientists just don't make the effort to explain what they are really doing, the context, the assumptions, or the significance of their work.

I think that technical prowess, originality and priority are over-rated. There are exceptions. Sometimes they are very important, but they don't have intrinsic merit, and sometimes fail on the importance criteria.

Note that I do not consider funding, status [journals and institutions], or citations as good measures of research quality. They are sometimes a consequence of some of the above, but they are neither a necessary nor a sufficient condition for high quality research.

I welcome comments and discussion.

3 comments:

  1. To add to your point on Importance: my (as you rightly mention, subjective) preference has always been for problems that are largely ignored by the community; I figure that the high-impact topics are looked at by enough people that someone will find the big truths in those hot topics anyway. In that sense --- arguably a little selfish of me --- I guess I'm hoping that science will change me rather than me changing science.

    ReplyDelete
  2. I think Creativity should also be a valuable criteria. Almost all rigorous, mathematically based and important scientific results that lead to a better quality of life for all are based on the lemma 'and then a miracle happens'.

    see arxiv.org/abs/1306.0061
    or the mirror site vixra.org eg vixra.org/abs/1401.0102 on God(/el)'s theorem .

    Following Yakovenko on economic inequality as a stochastic process, and current multiverse theory ('singing in harmony') it seems that all possible empirically confirmed facts and logically consistant theorems are true, and hence everyone deserves tenure. (or at least for some period of time as everyone goes their poincare trajectory, which means sometimes you are at Harvard, Stanford Chicago, but most of the time in a service job or jail). Poincare was a conventionalist---like Max Born----the speed of light can have any value you want, and even vary. So its all relative.

    Another aspect for determining merit fior tenure could also be attractiveness and availability. (I once long ago missed an opportunity for a graduate slot because i said 'no').

    ReplyDelete
  3. I'd be interested to your reaction to the list of evaluation criteria proposed at http://opeer.org/physics -- let me know if there is an important quantifiable criterion I've left out.

    ReplyDelete