Interpreting & Documenting Research & Findings
Published by the Universities of Edinburgh, Glasgow and Strathclyde
The material from this booklet has been developed from discussion groups and interviews with the research staff of Glasgow and Strathclyde Universities
The advice and contributions of Dr Avril Davidson, Mr Keri Davies, Prof George Gordon, Mrs Janice Reid, Dr Alan Taylor and Mrs Sheila Thompson are acknowledged.
The advice of the project Steering Group: Prof Michael Anderson, University of Edinburgh; Dr Nuala Booth, University of Aberdeen; Dr Ian Carter, University of Glasgow; Ms Jean Chandler, University of Glasgow; Dr Avril Davidson, University of Glasgow; Prof George Gordon, University of Strathclyde; Prof Caroline MacDonald, University of Paisley; Prof James McGoldrick, University of Dundee; Dr Alan Runcie, University of Strathclyde; Prof Susan Shaw, University of Strathclyde; Dr Alan Taylor, University of Edinburgh; Prof Rick Trainor, University of Glasgow is also acknowledged.
The project was funded by the Scottish Higher Education Funding Council.
Other titles in Series
Gaining Funding for Research
Gathering and Evaluating Information from Secondary Sources
Preparing the Research Brief
© Universities of Edinburgh, Glasgow and Strathclyde 1999
Cartoons D. Brown & W. L. Wilson
ISBN 0 85261 688 0Printed by Universities Design and Print
This booklet is one of a series of four aimed at researchers in the early stages of their career life cycle. The comments within the booklet are based upon information collected at a series of discussion groups and interviews at Strathclyde and Glasgow Universities. The questions put to the discussion groups were based broadly upon the performance criteria and knowledge requirements identified in the report "Draft Occupational Standards in Research" (Gealy et al,1997).
The booklet is in two sections. The first section, "Interpreting Research Results and Findings" considers various aspects concerning the interpretation of results. Generally the section considers how to confirm the reliability and analysis of results, the avoidance of bias or over-interpretation of results, and the identification from the results of potential areas of future research.
Section two, "Documenting Research Results and Findings," examines methods of presenting research findings, the physical aspects of record keeping, and what should be recorded within research records both to ensure their value to the researcher and to ensure that they are legally and ethically correct.
The booklet is not intended to be read in one fell swoop, but rather to be dipped into as and when the occasion arises.
Both sections within the booklet are subdivided into subsections each of which consist of:
- Points of advice, and examples from experienced researchers to highlight these points (colour linked). Information for the second section was collected through a series of interviews and discussion groups, which were formed from lecturers, PhD students, and Contract Research Staff (CRS).
- Bullet points which highlight the main points. The bullet points refer to the points and examples preceding them.
The booklet is not intended to be exhaustive or definitive. The issues raised are those which most exercised the minds of the researchers providing the comments for its preparation. These comments do offer interesting contrasts of opinion, either because commentators disagreed about the way to approach a certain issue, or because researchers from different subjects took different approaches in their methodology. The nature of the examples provided in the booklet are a reflection of the interests of those taking part in the discussions and interviews, and possess no greater significance than that.
Interpreting Research Results and Findings. 4
How do you confirm the reliability of your results? 4
How do you avoid getting into a rut with your analytical methods? 6
How would you define interpretative methods? 8
Definitions of "Interpretative" 8
How do you recognise and avoid bias in your interpretation of your results? 9
How do you evaluate your results in the light of the objectives of your original proposal? 11
When do you think uncertainty may arise over results and their interpretation and, how do you ensure that your conclusions are fully justified by the results? 12
How do you identify potential areas of further research from the results? 14
Documenting Research Results and Findings. 17
What techniques do you use to present your findings, and possible areas of future research to other interested bodies? 17
How do you record your research and findings? Are there methods of recording that you would avoid? 18
What details do you put in your research records? What details should never be missed out of records, and why? 20
How do you confirm that your records meet all relevant legal and ethical requirements? 22
Interpreting Research Results and Findings
How do you confirm the reliability of your results?
The exact nature of what is reliable will vary from field to field. Mathematical proofs, which are unusual in that there is an absolute right, are usually developed over years. In other fields, e.g. social planning and architecture, there may be no absolute right or wrong, and the confirmation, or otherwise, may take 30 years of urban development. Communication, experimental repetition, alternate approaches, good background knowledge will all be applicable in some fields, but are unlikely to be applicable in all fields.
Points to Consider
The most important initial stage is to be aware that your results may not be reliable. Blind faith does not make for good investigative research. Results may be misleading for a wide range of reasons, e.g. an atypical sample, equipment error, or the simple vagaries of animal behaviour. The latter point is nicely summed up by the Harvard Law of Animal Behaviour:Given precisely controlled conditions, the animal will do as it damn well pleases.
During a study of prostitution habits the researcher found that it was difficult to obtain reliable data on condom use. She could ask till she was blue in the face, and in as any different ways as she could think of: one-on-one interviews, focus groups, whatever. All interviewees reported 100% condom use unless they happened to burst. Yet it was obvious to the researcher that there were women who were working without condoms.
Peer review is a basic step in checks of reliability. Asking colleagues who have a sound knowledge of the field, but have not been as close to the work as yourself, is an essential and basic check of reliability. Better to have a colleague pick up a discrepancy at an early stage rather than a paper or grant referee at a later one.
It is important to ensure that you have an adequate number of repetitions within your experimental data (allowing for events such as pseudo-replication). However, repetitions can add new variables to the process. There is inevitably a balance between the demands of the objectives and the demands of precision.
The value of repetition was emphasised by one researcher who remarked that he would not report on any data which had not been confirmed within his own laboratory. For experimentation which required statistical analysis the precise number of replications was dependent upon the expected level of variability within the measurement. In order to ensure statistical accuracy when it is not possible to run a number of replicates simultaneously, the researcher reruns the complete experiment. The precise number of repetitions depends upon the variability between trials. His recent study examining the rearing of halibut fish highlights this latter point. The experiment required four different tanks, each tank providing a different environment. Normally the researcher would aim to do these in triplicate, providing a total of 12 tanks. However, because the experiment was within a production style system, the scale of the project made simultaneous trials impossible, thereby requiring the entire experiment to be repeated. This unavoidable variability requires an increased number of repetitions beyond the average. On the other hand, the big advantage of using a production style system is the avoidance of the extra variables inherent in scaling from the very small upwards.
Using several techniques on the same sample provides an alternate form of experimental repetition. Thus the reliability of tests for genetic mutations in tumours is regularly checked by using three different techniques on the same tumour sample.
Refer to previously published work and review your results within the context of previous publications to obtain a feel for general trends. They are some trends which may be expected to emerge. You must ensure adequate quality controls to avoid bias, i.e., inadvertently creating the result expected from ‘trend’. Bear in mind when checking reliability in the light of previous trends that many breakthroughs in science have at first been regarded as completely implausible. Plausibility is determined by present knowledge.
It is important to be thoroughly familiar with the background and content of the project. This is especially important where moving into new fields, where some less than obvious fact may pass unnoticed.
Whilst out collecting crabs a postgraduate researcher observed that some crabs reacted to other individuals of the same species by rearing up and attacking. Lower shore crabs were more likely to be aggressive than upper shore crabs. Several years later the researcher discovered that there were actually two species of crab on those shores, but that the two species were virtually identical. Fortunately the researcher had not published the study, and learnt a valuable lesson cheaply.
One engineer suggests the following summary for his own speciality:
a) Derive from first principles to establish ‘plausibility’. This would help to highlight erroneous results.
b) Meticulous calibration.
c) Error analysis. (Error analysis being the system used to measure the parameter will consist of different parts, each with an associated uncertainty. When this uncertainty can be obtained from calibration,then the uncertainty of the whole should be quantifiable.)
When working with human subjects it is essential to ensure that the sample is as representative as possible in order to check for a variety of different responses. One method of checking the accuracy of responses is to rephrase the question and then compare the new response with the answer to the earlier question. It is important to ensure that the analysis of the data is as inclusive of the varied responses as possible. One technique by which this can be done is the inductive procedure of deviant case analysis.
Deviant case analysis proceeds through examination of the universe of responses provided to a certain topic. If exploring the question of condom use, a basic hypothesis may be that prostitutes would encourage their clients to use condoms to ensure their own protection against HIV. However, there might be women who do not articulate their use in these terms at all, but refer to other reasons (e.g. they form a means of distinguishing between the sex they have with their private partners and the sex they provide to clients). The overall explanation for condom use as a barrier would still fit but the argument would have to be modified to incorporate the broader spectrum of responses. If, for example it was observed that most women report other reasons for condom use which do not fit within the barrier explanation then the original argument must either be modified, or discounted entirely.
- Bear in mind that your results may not be as you need them.
- Check all results thoroughly.
- Use alternative techniques to check results.
- Examine your results in the light of other work.
- Know your background information well.
How do you avoid getting into a rut with your analytical methods?
The best way to avoid becoming stuck in a rut is to remind yourself regularly of the risk of staying there. Most researchers will develop favoured techniques, and it is always easier to fall back on well used comfortable techniques than to seek out new and novel approaches which require the additional effort of getting up to speed. Communication and keeping up with the literature (not just in your own field) appear to be the best ways of remaining fresh.
Points to consider
Keep in touch with the research world around you. It takes some time for new methods/techniques to appear in the literature. As with so many aspects of research, networking is vital. Perhaps the most common reason for the development of new techniques has been where the present technique was very labour intensive. In these circumstances it is worth asking around to see what other investigators are doing.
Look to other fields for inspiration.
One research team studying the prostitute population of the red light area in a large city decided to adopt the biological technique of ‘capture/mark/recapture’. The study required identifiers, so the team ‘tagged’ each individually ‘captured’ subject with a unique ID, then used these identifiers to model changes in the prostitution population over a period of time. This is a particularly elegant example as the technique originated from an 1800’s study in Paris in which the number of priests was used to estimate the total population of the city. Then the technique was adopted by ecologists to model animal population dynamics. In this instance the technique has moved from social science research to biological research and back to social science.
Discuss what you are about to do with your colleagues who may well contribute some good ideas. Discussion can help you avoid becoming enmeshed in minutiae and missing the bigger picture. It may be helpful to brainstorm with a group of your peers on a regular basis.
One researcher remarked that she had been experimenting for months with a new technique to identify differentially expressed genes, all the while going nowhere. Then after several months she discovered that throughout that period a colleague in the group had been arguing the case for an alternative technique. Greater efforts at communication would have saved her several wasted months.
For some fields, such as English Literature, opening a line of communication with the author you are studying may provide a useful insight into their work. For other fields, contacting authors of publications will allow you to discuss new techniques being developed, or perhaps highlight publications which you may have inadvertently missed.
Look for weakness in the methodology, for instance, is the present technique of a lower sensitivity than required, can it be improved? Regular reappraisal of the techniques used, and consideration of their less satisfactory points, should help avoid complacency.
A research group was interested in measuring virus-specific immune responses. When the researcher joined the group there were a number of techniques available to measure antibody responses to the virus. However, there were no reliable techniques available to measure the T-cell response. Unlike antibodies, T-cells recognise virus infected cells, or tumour cells, and kill them. The researcher’s first task was to develop such a technique. He succeeded in developing a technique which was then adopted globally. There remained concern that the technique was underestimating the true magnitude of the host T-cell response. The team is now in the process of designing novel assays to measure virus-specific T-cells. Using these assays it will not only be possible to verify the data that they (and other laboratories) have obtained, but the improved sensitivity afforded by the new techniques will allow detection of T-cells in circumstances which would otherwise have been overlooked.
There can be problems in securing funding for completely new approaches.. However attractive a new methodology may appear, it is important to ensure that the methodology will not discourage funding bodies if it is included within your grant application. This can be a "catch 22". You want to be adventurous, but cannot move forward because funding bodies or collaborators will be cautious of the ‘excessive’ novelty of your new idea. On the other hand, in the highly competitive world of research funding, you may need that bit of novelty as an added attraction. If in doubt it is well worth contacting your prospective funding bodies in advance. Some funding bodies run schemes to promote "blue skies" research such as the Research Council, Realising our Potential Awards (ROPA’s). Though original and novel are not one and the same, a pilot run will move "novel to original" and help you convince the more sceptical reviewer.
- Keep up with the literature.
- Networking is essential.
- Look to other fields for inspiration.
- Cast a critical eye over your methodologies, identify the weak points, seek alternatives which ameliorate them.
How would you define interpretative methods?
It became obvious during the discussions and interviews used in the creation of this booklet that the definition of ‘interpretative’ was not consistent. The question "how would you define interpretative methods?" was put to participants to try to gain some idea of the definitions of different fields. In order to avoid interpreting the interpretations of "interpretative" and inadvertently shifting the definitions towards a biologist’s view of the world, this section has been kept in the form of the original quotations.
Definitions of "Interpretative"
"This would partly be related to the way that the experiments have been set up - you set up experiments with defined objectives, the interpretation of which would initially be based on that background information. You analyse results by plotting them in various ways, carrying out statistical analysis and comparing them with your expected views from the experiment."
"As an architect you interpret your model of parts of cities against a set of criteria which you hope are generally agreed upon. There is a huge amount of literature on what a sustainable city should look like, although there is also huge disagreement. However, there are certain consistent demands upon a city, e.g. public transport, low degrees of pollution and eliminating congestion. You can set these as the targets for your models and test your models to see how they influence these criterion, and to what degree. But as the models are not real, there is no actual physical proof. Thus there are two sets of interpretations, what criterion should be used to judge the model and, in the absence of physical proof, how accurately does the model reflect what would happen in reality. The difficulty in our field is that everybody has his or her own set of criteria to judge against, and so we never agree."
"I think if something meets with your understanding of the subject. In my own research I have interpretative methods that would anticipate my critics. It can be very objective in the sense that you can interpret according to the aim of what you are trying to do. Thus in terms of interpretative methods researchers need to be aware of how the results would be interpreted, by the media, peers in research and the community at large. This is important when the research involves some controversial subject. A further advantage of attempting to foresee how some arguments will be interpreted is that a prior response can be prepared."
"Interpretation in English Literature often possesses the implicit danger of interpreting things along the lines of your own preconceived notions. In my view, that has happened too frequently, with theories transposed onto (and into) texts, and the resulting criticism has been not so much a criticism of the text but an expression of the critic’s own opinions. So interpretation becomes too greatly bound up with opinion. I think covering yourself to anticipate your critics is necessary to a certain extent, in the sense that your thesis must be as logical and consistent as possible, but this should not be at the expense of your being totally inflexible, and blinding yourself to any shortcomings in your thesis. By all means go into your project armed with notions which will challenge the received wisdom in your chosen area, but be aware of dealing with gaps in your own argument too!"
How do you recognise and avoid bias in your interpretation of your results?
One researcher remarked that he wanted a particular solution because he was sure that it was the correct solution, when in fact it was the wrong solution. It was the interpretation of a brief for a housing development scheme. He tried to test what was actually meant by the relationship of the different functional elements within the scheme. In this instance he thought he knew the answer because he had worked on similar schemes previously. Thus, when told by a colleague that there was a mistake in his interpretation he failed to check it. The housing scheme was developed to the full, and then collapsed because of that mistake. It was, he remarked, a painful exercise often remembered, never to be repeated!
Few of us can claim to be completely free of such bias, the following section attempts to identify some of the areas where bias commonly arises, and outline some techniques for its recognition.
Points to Consider
Bias can arise in the construction of the experiment rather than in the interpretation. It is important to ensure that the experimental design, or the behaviour of the researcher, does not introduce bias long before the interpretative stages are reached.
A team of ethologists were attempting to breed a more intelligent strain of rat, intelligence being measured by maze learning abilities. As the project proceeded it appeared that a superior strain of intellect had been bred. At least that was the conclusion until the techniques of the workers involved were more closely examined. The researchers stroked the more ‘intelligent’ rats before introducing them to the maze, but did not stroke the supposedly less intelligent rats. Improved learning was not a function of superior breeding, but rather of more pleasant handling conditions.
Discuss your results and interpretation of these results with colleagues. It is especially helpful to seek out colleagues from different backgrounds and experience. It is important to ensure that the review of your conclusions will be genuinely critical, there is little value to be gained from seeking excessively polite or friendly colleagues.
One researcher remarked that a recently retired professor tended to think in a different way to most of his colleagues. There were a few colleagues who, when a document was put in front of them, would react in a predictable manner. But this professor tended to throw up quite different points from the document. He had a very different background from the rest of the group, as well as having a wider range of experience. He had worked in industry for a number of years, and had had a lot of experience in vaccine development and trials, and marine biology. The areas in which the researcher was interested, but from a quite a different perspective.
In interview-based research, the perception of the interviewer and his/her experience of life, has considerable potential to radically colour his/her interpretation of events around them.
The researcher who had been conducting a research project studying prostitutes on the streets of Glasgow initially assumed that they would be afraid of the police, when many did not care about the police at all. The researcher’s interpretation of how they would react to being apprehended by the police was influenced by her own background. She stood to lose a lot by being prosecuted for anything, the prostitutes, on the other hand, felt that they had little to lose.
Reflexivity is at the core of interviewing. To some extent interviewing an individual is like looking in a mirror. There is a strong tendency for the interviewee’s response to be coloured by how they perceive the interviewer. It is not only how you perceive the experimental subject, but how they perceive you.
When interviewing drug addicts about needle sharing, the addicts rightly construe the interviewer as being someone who thinks that needle sharing is not a great idea. As a result, they will tend to deny sharing needles because they do not wish to give a poor impression to the researcher. Drug users require, in their search for drugs, good manipulatory skills, and become very skilled social actors. They will often only give an interviewer as much information as they estimate he/she already possesses. As the researcher entered more deeply into the field she gained a relatively deeper understanding of what was going on. In becoming more aware of the tendency to present selective information, the researcher established ways of getting beyond the surface presentation of the facts.
Imagine that you are presenting your conclusions to your worst enemy, where would they pick flaws, and how would you defend yourself against their arguments? Anticipate your critics.
Bear in mind that an absolutely objective truth may be unobtainable, perhaps only another kind of truth will be possible. The focus of your research is complex, the interpretation more so.
Bias can appear through the unconditional acceptance of previous work. It is in circumstances such as those described in the example below which make networking and a broad awareness of the background vital. Researchers should never accept blindly the bias of the past, if you think something is wrong - perhaps it is.
During a PhD viva the examiner asked about a certain result which the research student had quoted from a classic mathematics textbook, remarking, ‘Of course, you know the proof is wrong’. The researcher recalled having struggled with this proof. He had never managed to follow the proof’s logic, but as it was cited in a classic textbook, accepted the result as true, and quoted it. It was not a big issue at the time, but there is an element in mathematics (as in most subjects) where a theorem can be correct almost through folklore. Even if you go to the source of the proof you will find reference to some other source, or simply a statement of the result but no proof. More often than not these results are correct, but there have been occasions when results have made it into the folklore of mathematics. It raises the issue of how far back you should go before you accept that a result is true without having to iterate the entire proof yourself.
Seek out the counter arguments to your own interpretations, and consider whether or not you have given the alternates a fair hearing.
- Take advice on your interpretation, especially from colleagues with a different view of things to yourself.
- Be aware that your perception of the results is coloured by your life experiences and expectations.
- Equally you must be aware that just as your interpretation is biased by your experiences, so can the interviewee’s responses be biased by their perception of the interviewer.
- Do not accept historical wisdoms blindly.
How do you evaluate your results in the light of the objectives of your original proposal?
Adaptation of new methodological techniques was identified as a frequent source of problems. Either the methodology proves unable to deliver all that it promised, or it takes too long to train the staff/yourself in the use of the new techniques. The next most likely cause of failure was simply setting one’s sights too high, a common temptation with the increasing competitiveness of winning grant funding.
Points to Consider
The use of untested methodologies can result in a lower than expected level of success.
The researchers were examining the genetic basis of the production of particular types of toxin by bacteria. This was of interest as these were toxins which were generally considered to be produced by algae, but not by bacteria. The methods that the group chose to use were not as well developed as they had anticipated. They found that the sensitivity of the methods was not good enough to allow the screening of bacterial toxin production, although the method worked well for algae, which produce much more toxin. To effectively screen bacteria the project required a screening system which could run 1,000 samples rather than the 50-60 needed for algae, a demand beyond the capabilities of the technique. In retrospect, overconfidence in an untried methodology resulted in the project’s failure.
It can take longer than anticipated to train or re-train and provide the necessary level of new experience required for staff (and yourself) to change fields. Time must be allowed within your proposal for training, failure to do so can result in projects being less successful than anticipated.
Were the objectives unattainable? This can occur when relying on claims made for techniques without any definite evidence that these claims are accurate. This may occur for a range of reasons e.g., extrapolation beyond the reasonable range of the original results, altered conditions, attempting to push the system beyond its capacity.
The research team had been working on the microbiology of turbot larvae, which are susceptible to very large losses when they are at the first feeding stage. A new project was proposed which was to be run in collaboration with a European company. The funding was granted on the grounds that the commercial company claimed to possess an improved technique for preparing water to allow the rearing of larvae in defined conditions. However, major collaborative trials using the methodology failed to reach the standards which the company had claimed were possible in their publications. The reason for the failure was probably that the research team had inadvertently exceeded the capacity of the system and extrapolation of the technique to larger systems was impossible.
- How certain are you that the claims made by others are accurate?
- Ensure an adequate allowance of time for retraining and developing untested techniques.
- Consider that you may not have failed, perhaps the results were not positive, but that in itself need not constitute failure.
- Unexpected results may result in the research progressing along a different route.
When do you think uncertainty may arise over results and their interpretation and, how do you ensure that your conclusions are fully justified by the results?
There are a variety of reasons for which interpretation and results may become confused or over-interpreted. These range from decisions made regarding modification of the data, to allowing the expectations for the results to blind the researcher to their actuality. This section offers various suggestions on how to avoid such problems arising and, as with so many of the discussions in these booklets, emphasises that consultation with colleagues is vital.
Points to Consider
Avoid overstating the result: inconclusive results should not result in conclusive statements.
Present your results and conclusions to colleagues and ask them for comments. Make certain they understand that you are looking for constructive criticism, and not just a pat on the head.
A good rule of thumb may be to go for the simplest explanation.
One method of avoiding confusion between interpretation and results is to leave the work alone for six months. After that period when you have forgotten some of the background, and are free of any unhelpful habits of thinking you may have inadvertently fallen into, then you should be better able to spot any confusion between interpretation and results, or for that matter bias in interpretation.
When repeated trials provide conflicting results any decisions taken regarding the relative reliability or accuracy of the various results is an interpretation of the results, and should be noted as such.
Clearly differentiate between the results per se and your extrapolation/interpretation of them.
A common error is to confuse a correlation between two variables and an actual cause and effect. The magnitude, significance and direction of the correlation is the result: conclusions regarding cause and effect are interpretation.
It is important not to let your expectations of results predetermine your view of them. Firmly drawn conclusions should be sustainable by the data alone, and not reliant on the theories of previous work.
A researcher collated a large set of data on school children’s use of drugs and alcohol. This was initially analysed using logistic regression. The results appeared to fit with ideas which she had previously expressed in a literature review. Both results and discussion supported the researcher’s interpretation of the literature. She later re-analysed the data using a different series of tests. The explanation of the data she provided on the basis of these tests, although informed by the literature review, was not dictated by it. The re-interpretation was much closer to the data. For example, peer formation is significant in any behaviour but particularly in relation to drug use. Previously the researcher had been enthusiastic about theories that suggested that core family background would allow predictions of negative peer engagement. The literature had provided fairly clear evidence that this was the case, and although she had not claimed that the data demonstrated this, she had used the theory as a possible explanation of the data. However, her re-analysis suggested that the original conclusions had been an over-interpretation of the data.
Perhaps the most common reason for conclusions and discussion not being justified by results occurs when the discussion is extrapolated well beyond the limitation of the results.
A paper described research involving sampling maturing fish over the course of a year or more and looked at changes in the amount of fat stored. The paper recorded differences between maturing and immature fish, especially in the patterns of depletion and re-building of mesenteric fat stores. The discussion section of the paper considers this in the light of ecological implications of fasting on maturation rates. However, more than half of the lengthy discussion proposed a model for the hormonal control of maturation. Hormonal control had not been mentioned prior to the discussion, and the research was only tangentially related to it, leaving the reader with the impression that the authors had failed to complete a sufficiently comprehensive review article, and simply added an incomplete review at the end of the paper.
- Avoid any confusion in data recording.
- Correlation is not proof of causality.
- Decisions made regarding the reliability of the results should be labelled as interpretations not results.
- Do not allow your expectations to predetermine your conclusions.
How do you identify potential areas of further research from the results?
Opportunism is one watch word in the identification of potential areas of new research. Opportunism takes many forms, e.g. capitalising on current trends and fashions, and spotting weakness in present methodologies. The second key is to keep an eye on the long term objective. The ability to avoid becoming lost in the woods can be improved by a regular interchange of ideas with colleagues.
Points to Consider
Going away to a major conference can be a good way of focusing on the results of the past year. It both allows you escape the distractions of your usual routine and, in conversation with others, identify the direction in which your field is heading.
"At the conference last week, one area of interest was in a particular fish disease and I had gained quite a lot of information on that. When you tie it in with existing information you can spot the areas which are obvious for development. For example, this particular organism is a rickettsial infection. Rickettsiae are bacteria which must replicate within a eukaryotic cell so they are rather unusual. This is a fish disease which is particularly important in Chile but has been reported in Northern Europe as well and what we are interested in is identifying particular components of the outer membrane of the bacteria which stimulate an immune response, so from the meeting I got a clear idea of which antigens to concentrate on."
There are short term objectives and long term objectives. It is the examination of the latter which is most likely to indicate the potential for further research.
Be prepared to shift the focus of your research as the political, social and scientific priorities of the wider community move. It is not unheard of for projects which have been rejected for funding to become, at a later date, greatly sought after by the funding bodies.
A researcher’s first project included an examination of the behaviour of drug injectors. At the time this was just a small part of the project, but with increasing HIV awareness injecting behaviour gained a much higher profile politically. It became obvious that this was an area in which research funding would become available.
The identification of weakness in present techniques, e.g. high cost of production, or the potential to develop a more efficient system, will often provide new avenues of research.
The majority of currently available vaccines are based on either inactivated virus or bacteria, or comprise a synthetic or "recombinant" protein which has been produced in bacteria. The production costs for these types of vaccine are high, and some improperly inactivated vaccines have been responsible for outbreaks of disease. These problems have encouraged researchers to evaluate the potential of nucleic acid or DNA vaccination as an alternative. Using this technique the DNA is injected directly into the animal or person, there is no risk of infection since the whole virus or bacteria is not used, and there is no costly production and purification of recombinant protein. To optimise the immune responses produced following vaccination a chemical called an "adjuvant" is often included. Recently, the team has pioneered the use of genetic adjuvants in veterinary medicine. The results have now opened up a whole new area of research, not only in the application of this technology to other infectious agents of man and animals, but also in improving our knowledge of the way the genetic adjuvant is exerting its effect.
- Keep an eye focused on the longer term objectives of your project.
- Monitor the shift in public and political priorities, the timing of a proposal can be vital.
- Look for weakness in the existing and preferred techniques.
- Keep discussing your work, its progress and its potential with your colleagues.
Documenting Research Results and Findings
What techniques do you use to present your findings, and possible areas of future research to other interested bodies?
Other interested bodies are a varied group, both in their understanding of your subject, and their specific interest in it. In order to achieve maximum impact it is important to vary your approach according to interest and understanding. The following section considers when it is appropriate to take different approaches to presenting research findings, and offers suggestions as to what these alternates may involve.
Points to Consider
Industrial workshops can be a useful way of putting your message across to potential funders as workshops often allow a much freer exchange of information than conferences. This occurs, at least in part, because the presentation is to potential funding bodies, whereas at conference the presentation is to potential competitors and less recent results are often presented. You would have more opportunity to describe your capabilities and past achievements. However, when visiting industrial workshops it may be wise to take advice on intellectual property rights before making your presentation. Consult your Research Support staff who can advise on the institutional policies in this area.
When seeking research funding it is important to make your objectives absolutely clear. A good technique is to provide a very succinct list of aims and objectives. When presenting to potential industrial collaborators, funders or users of your research, a one page A4 summary (in bullet point form) of your objectives, and the commitments that you require from the industrialist, can be very helpful.
If you are talking to a group of people who know little of your subject then it becomes especially important to avoid jargon. Use clear, plain English. Get a non-expert to review your presentation or paper.
How you dress may be important, the more casual dress code common in academia will certainly be less acceptable to potential funders from industry.
Try to target your audience’s interests, tailor your presentation accordingly. Talking to members of the audience will give some idea of the sort of language they use, what they are likely to be interested in, and what they will understand.
When a researcher presented data on her project to her Co-operative Award in Science and Engineering (CASE) funding partner, a fish farming company, she altered the emphasis of the presentations. The presentations were more or less the same as those she gave at scientific conference, but with one significant difference. Scientists think in terms of the length of fish, whereas fish farmers think in terms of the weight. Thus for presentations to fish farmers she re-analysed her data to take account of the difference in approach.
For a larger audience one researcher remarked that he would use Powerpoint and a slide projector, but for a smaller more informal audience, a board and a pen or overheads. If the lights are on you can better gauge if your audience is interested and enjoying your presentation. Standing and writing also has the further advantage/disadvantage of adding to the informality of the proceedings.
As all who have written a thesis or a major report will know, most people will never read them in their totality. One solution may be to present an executive summary of the research. This increases the likelihood of its being read by focusing all of the ideas into a short and concise section, but of course it leaves out all the proof, evidence, arguments and counter arguments. Multi-media productions offer considerable potential in this area. Although more complex and expensive to produce, they allow readers to look through your research and pick out what interests them, by jumping from one point to the another.
When seeking funding consider emphasising the ‘benefits’ rather than the ‘features’. Thus instead of a fully integrated software package which is easy to use, highlight the benefits, e.g. minimal training required, financial savings.
- Know your audience
- Target the interests of your audience and be prepared to vary your approach according to those interests.
- Attend informal workshops set up by relevant industries.
- Always produce executive summaries of large reports.
- When presenting to industrial and other end users keep it simple and straight to the point.
- Avoid jargon.
How do you record your research and findings? Are there methods of recording that you would avoid?
The emphasis of this section is centred upon the physical aspects of record keeping. The second half of the discussion considers the importance of duplicates, accessibility and longevity of the records, and when records can be discarded.
Points to Consider
Records must possess longevity. Use good quality paper, which should last at least 30 years. Do not use pencils or strange coloured inks, the ink must not be water soluble or solvent reactive, it should not smear and should be light stable (BTG plc)
Research records should be kept in a form which ensures that their authenticity can be appropriately defended. Claims of originality and scientific priority are best supported by records whose provenance and date are beyond reasonable doubt. This is especially important for the protection of Intellectual Property Rights (IPR) when negotiating contracts for the exploitation of research results, and seeking to establish ownership of background IPR. To fulfil such obligations to maintain accountable and dependable records, best practice suggests that all experimental data should be meticulously and permanently recorded, in a bound notebook with numbered pages, with all entries dated, signed and witnessed. Computer printouts and instrumental data printouts should be incorporated permanently into the notebook. Where these are pasted in, the witness should sign and date across the join.
Arrangements should be made to keep duplicates of all irreplaceable data records. Important material stored on computer should be systematically backed up, ideally there should always be at least three copies, one of which is off site. Loss of experimental records, data, grant applications, and drafts of publications in fires, floods, or other disasters can vary in effect from extremely frustrating to catastrophic.
Loose leaf laboratory records can be very useful if the data contains lots of ancillary documents (e.g. photographic plates, spectrophotometer printouts or sample interview sheets). Such records are always difficult to file and a loose leaf folders can serve as a good supplementary, or replacement, to a laboratory notebook, though a loose leaf folder is less convincing evidence in any IPR disputes. The indexing of ancillary data, such as that described above, is critical. At the very least, each item should be annotated with the data and location in the notebook of the corresponding experiment. Many items of computer-controlled equipment provide printouts of instrument settings, as well as date and time. Ensure that the clock is correctly set, it may be important in future IPR debates. Make certain that you, or the computer, compensate for leap years and seasonal time changes(Beynon, 1993).
Keep duplicate records, if you are using electronic records make sure that you have off-site as well as on-site records. Ideally all computer records should be in triplicate, the hard disc, one floppy on-site, and one floppy off-site. Remember though that floppies, if unused for long periods, can cease to function properly, and data may be lost as a result. It should be borne in mind that because dates can easily be altered on electronic records they are poor evidence in the event of IPR disputes.
"It is ironic that many laboratories seem to give more consideration to the storage of reprints, which are copies of existing literature, than to notebooks, which are irreplaceable originals" (Beynon, 1993).
Make sure your records are well labelled. It may seem obvious now, but a year or so down the line the chances of remembering what the data columns represent are slim.
Consider building your research records like a tree, allowing connecting ideas to follow through a particular branch of the tree. Once you have discarded the other branches over the years, the remainder can be discarded. On the other hand, archiving old material, even if you do not believe you will return to it, may allow you to refer back to a solution to a problem which you have had to deal with previously.
Make sure that your record system is accessible, it is of little use if you have to walk through half the building to access it, or if you have one type of computer system at home and another in the office. Similarly, try not to put it on some obscure computing system that is likely to vanish within the next few years.
Take care of where and how you record your list of ‘things to do’. Consider mentioning your objectives to others at coffee time - in six months time they may remind you.
One researcher admitted that on moving office recently she found a list of ‘things to do’ dated four years earlier - none of which she had done.
- Results and methods should be recorded in a manner which can leave no doubt as to their authenticity.
- All records should be signed, dated, and witnessed.
- Keep duplicates.
- Record labelling must withstand the test.
- Records must possess physical longevity.
- That list of ‘things to do’ must be high profile and visible.
What details do you put in your research records? What details should never be missed out of records, and why?
Consistency is an important aspect of record keeping. Records should be kept in a consistent manner regardless of the experiment. Failure to do so is likely to result in records becoming incomprehensible a few years down the line. Records, if they are to be of real value, are to be kept over a period of years. A consistent style of record keeping will reduce the risk of the records appearing incomprehensible if there is a need to examine them after a prolonged period of neglect. Exactly what should be incorporated within the records depends primarily upon their end purpose. Ideally, your main record collection should allow you to repeat any course of work, be able to secure your IPR and help defend you against unjust accusations. For further explanations on material on patent law, etc. please refer to THEROS: Technology Ventures - Intellectual Property Guidelines (1998). On the whole it is better to err on the side of caution when deciding what to leave in and what to leave out.
Points to Consider
US and UK patent laws are not identical. Thus in the US evidence of the date of conception of an invention, and proof of diligence in its reduction to practice is required for patenting.
"Errors and mistakes should not be erased or obliterated beyond recognition. Neither should liquid paper be used. Simply crossing out an error so that it is apparent what the error was should be adequate. Explain all errors and mistakes as they occur and initial them. Never remove pages from the notebook." (BTG plc)
Record novel concepts and ideas relating to the work though avoid the expression of opinions (BTG plc).
Many of the researchers in our discussion group kept a day book in which they recorded everything they had done that day. The book might include the chemicals used and in what quantity, and anything that had gone wrong. Any results which come from a printer should be put into that day book (if these are perishable printouts, copies must be made). Tables of results would go into a separate folder.
When recording data it is self-evident which data set is which. Five years down the line and it is highly likely that you will have forgotten which data set is which. It is vital to sort and clearly label computer held data from day one, especially if the computer records will include earlier and later versions of the same data set. Records should be in a form which can readily be understood by everybody. This is necessary partly because in debate over patent rights it is vital that the records should be easily understood, and partly for the reasons above - you will feel silly if at sometime in the future you admit you cannot understand your own records.
Log the incidental. There may be contextual events or activities which affect the data, climatic conditions, on-going political/newspaper campaign, delay in sampling (Brownet al., 1995).
Although the comments made in the example below were valid, the potential for the records to serve more than one purpose should be borne in mind. The main aims were to ensure that the work was repeatable, and that the details were adequate for protecting intellectual property rights (however, see "Diligence"). In this light the criticism of research assistants including irrelevant details was valid. The records would allow the researcher to point out that while they accepted that a particular objective was not met, nonetheless, it was not their responsibility.
Include anything which is remotely likely to be required or useful. You come to recognise through experience (which does not take long to acquire) when you have failed to record points that are going to be needed in as much detail as possible. However, it is interesting to see the irrelevant details that some people have recorded. Huge amounts of irrelevant information, for example, I was away on holiday, or something has not yet arrived. The critical issues are the date, and a couple of lines on the objective and methods used. In terms of the details, the experimental method (especially if it deviated in any way from the standard protocol), and the results, should go straight into the book which is the day-to-day record - in the book not just on any piece of paper.
However, BTG would tend to support the researcher’s method of recording. "Diligence in the reduction to practice of an invention means that, as far as possible, generally steady, uninterrupted and constant work occurred following the conception of an invention. In an interference action (where IPR is challenged) periods of inactivity could lose the case, especially in a situation where each day is critical. All activities must be logged, even if it is only to note that you were waiting for, say, sample analysis that resulted in delay in the proceedings," (BTG plc).
When recording the results of pilot studies be careful not to be more lax than is normal in terms of the quality of the information recorded.
Data must be recorded carefully. Resist the temptation to record the data in rough form and transpose it to your notebook at a later date - this provides an extra opportunity for the introduction of errors (assuming you get round to it in the first place). When recording data from instruments, note the settings on the instrument panel.
(i)A fluorimeter value of ‘10.4 units’ is meaningless and cannot be rechecked without notes on scale widths, scale expansion factors, wavelengths and all other machine settings.
(ii)During electrophoresis note the current and voltage; this will allow you to calculate the resistance of the gel, and spot a buffer of incorrect conductivity.
(iii)In a chromatography run, note flow rate, column back pressure, detector settings, column type and, if there is more than one column of that type in the laboratory, the serial number of the column (Beynon, 1993).
- Do not exclude data which you only think may be significant in the future, err on the side of caution.
- Keep record keeping consistent.
- Remember that you must be able to recognise data files not just next year but in three or four years time.
- You can use your records not only to record experimental details, but also to cover yourself against future unfair accusations.
How do you confirm that your records meet all relevant legal and ethical requirements?
The ethical problems of working with human subjects are considered not only from the perspective of their rights, but whether or not they understand these rights. The final section of the discussion considers where sources of ethical advice may be located.
Points to Consider
In ethical terms make certain that your volunteers understand the ethical promises you have made them. Although your explanation may seem clear to you, ask the subjects some questions on what you have promised them, they may not have understood after all.
One researcher working on prostitution and HIV explained to all her subjects that a ‘double blind’ system was being used. This meant that the results of the HIV test could not be identified with the person from whom the samples had been taken. Despite this, the researcher regularly received requests from her subjects as to their HIV status, thus it became obvious that many of the volunteers had not understood the ethical commitments that the research team had made.
Research upon human beings can carry the added complication of political overtones. Data, results and conclusions should not be modified for political purposes, but neither can researchers deny that their conclusions are liable to be used for such purposes. Especially where vulnerable groups are involved, consideration should be given as to how the project will be presented.
Prostitutes are a stigmatised and vulnerable group. The group the research team tested for HIV prevalence came up with a relatively low percentage, about 3%. However, had that percentage been 50% then that would have raised a completely different set of ethical issues.
Animal experimentation is a continuous process. Once begun the animals (even during non experimental periods) require constant supervision. This must be taken into account when planning the project. Appropriate experimental records to meet the requirements for the annual Home Office returns must be kept. Organisations such as the Ministry for Agriculture and Fisheries (MAFF)will want to know whether you have conformed to standards such as ISO 2000 or Good Laboratory Practice (GLP), ISO being a standard for experimental procedure and the recording of data. GLP again has particular requirements. These often tend to be in the form of standard checks. Thus results may have to be confirmed by a superior who may have to initial a page in a notebook to say that they had read and checked the records. If there are Home Office requirements then there is a very clear line of responsibility and there will generally be someone in your Department responsible for Home Office requirements. Your institution will have staff who are responsible for such issues. Be sure to seek out their advice at an early stage of project planning.
It is not unheard of for data to be destroyed when they have not provided the expected results, or in order to avoid a closer scrutiny of conclusions. This is unethical. Your institution will have a policy on scientific conduct. Make yourself familiar with this document. Your Research Support staff will be able to advise you on such issues.
Plagiarising the work of others is unethical.
One researcher recently published her dissertation - a German dissertation - but when she went to the examination in Germany discovered that her work had been published by a member of staff.
You must obtain written permission from respondents to cite extracts from interviews in publications (even if they have been anonymised).
If you are concerned with legal aspects of animal research your institution will have an office designated to answer such enquiries. If they cannot help, you must contact the Home Office. They are very helpful and will usually give you an answer immediately.
When the researcher was working on a fish pathogen (rickettsiae) he telephoned the Scottish Office to clarify the situation regarding the import and export of these bacteria and whether or not a licence is required, because it is also the provider of such licences.
For information on safety requirements there will always be someone within your Department who will be able to provide you with the necessary information. Your institution may have a central Health and Safety Office
In microbiology the pathogenic category of the material you are using should be identified before the project commences. The pathogenicity will have been categorised one, two, three and so on. Most institutions will have little difficulty for categories one and two, but categories three and upwards require special facilities. These categories tend to be reasonably virulent organisms which require special conditions for safely recording growth. It is important that you clarify the position before you begin. This applies to most high risk material, e.g. for ionising radiation, someone within your Department will be delegated to look after radiation matters, but your institution will probably have a Radiation Protection Officer.
If your subject is to be allowed to make an informed decision on whether they wish to participate in the experiment then you should respect participant/research subject autonomy, i.e.
doing what you said you would do, and nothing more or less.
The following example from the experiences of Stephen Waters (Bell, 1993) provides interesting insights into the difficulties of carrying out research on one’s own colleagues. Although the example is predominately one of gathering the basic data, nonetheless, there are points to be learnt regarding what happens to that data after they become secondary published data.
Stephen Waters was a teacher who decided, as part of an Open University course, to investigate the role of his own Head of Department. He went through a fairly prolonged negotiation period to reassure colleagues as to his trustworthiness before embarking on his programme of research. Of interest here are some of the comments he later made regarding the guarantees he had given prior to undertaking the research. He had promised all participants an opportunity to verify statements prior to production of the final report. This proved almost impossible, most participants not having time to read the entire manuscript. Therefore, lacking time to identify all their comments within the manuscript. He had further promised all participants a copy of the final report, which ultimately cost rather more than anticipated. The ethical agreement he reached with contributors was only made verbally. This created problems at a later stage, when it transpired that none of the contributors could precisely recall the conditions agreed upon. In retrospect he regretted not providing them with a written copy of the agreement. However, it was in seeking to publish the data that the greatest problems arose. All contributors had been promised anonymity. A promise which could be met externally, nobody outwith the school could identify the contributors. However, as all the contributors came from the same school, it proved impossible to provide anonymity internally.
Careful consideration should be given to any possible conflict of interest, or the appearance of such. If this problem arises during the course of your research, experienced advice should be sought.
Independent work can bring researchers into conflict with their institution. This may occur if the independent work utilises the results of research which the institution may regard as being part of its intellectual property. Conflicts of interest may arise when a person involved in a research project has the opportunity to influence institutional funding decisions impinging upon that project. In any of these types of situation it is essential to get advice from your own Research Support staff.
- Make certain that human subjects understand the ethical conditions under which you are operating.
- You have a duty to the people you are working upon, consider how your results will affect their lives.
- Keep a record of any agreements made.
- Seek out the person within your department who is responsible for safety issues.
- Do not plagiarise.
- Remember that legal and ethical are not one and the same, and that the absence of a written code does not excuse an absence of ethical behaviour.
- If there are doubts about the legal aspects of animal experimentation then the Home Office is the place to go.
- All researchers will have a range of sources of advice available to them e.g., funding bodies, hospital/institution/ professional bodies ethical committees.
- Finally ask yourself if the standards you practice are those by which you would like to be treated?
Bell, J.(1993) Doing Your Research Project: A Guide for First-Time Researchers in Education and Social Science. Open University Press, Buckingham. 176 pp.
Beynon, R.J.(1993) Postgraduate study in the biological sciences: A researcher’s companion. Portland Press, London 151 pp.
Brown, S., McDowell, E., and Race, P.(1995) 500 Tips for Research Students. Kogan Page Ltd, London. 127 pp.
BTG plcKeeping a Laboratory Notebook. Gulph Mills, USA. 12pp.
Gealy, N. and Clarke, D.(1998) Development of an Interim Workplan for the Researcher’s Lead Body. Maloney and Gealy,24-26 Mossbury Rd. London. 30 pp.
Gealy, N., Westlake, D., & Clarke, D.(1997) Draft Occupational Standards In Research. Maloney and Gealy,24-26 Mossbury Rd. London. 59 pp.
Skelton, F. and Walker, L.(1995) Pilot Study to Assess the Benefits of Gathering Evidence of Research Competencies for PhD Students to Improve Their Subsequent Employability. Glasgow University. 21pp.
THEROS: Technology Ventures - Intellectual Property Guidelines(1998)