Written with co-author Dr Christine Bottrell. Pre-service teacher education students, in most courses, are required to undertake some study of education research methods, to provide them with skills to read and make use of education research. However, the field of education research is a complex and difficult area. Perhaps focusing upon the most frequently used research methods could be a useful starting point. So, what type of research methods are most favoured in education? This article describes the authors’ attempts to answer this question, and the unexpected outcomes of the quest.
Introduction
Understanding research and being able to critically read research reports in education would seem to be an important skill for teachers to acquire, and many initial teacher education programs, in countries such as Australia, require students to study a research method subject. In the case of pre-service teachers, there are important questions for a lecturer to consider. What should be the content and focus of an introductory course in research methods for pre-service education students? What are the priorities? Where to start? The starting point is important because, for many of these pre-service teachers, this may be the only study they will ever undertake in the area of education research. A pre-service teacher who is provided with a sound foundation in research methods is more likely to be a productive user of education research as an education practitioner.
In the limited time available for an introductory course in research methods, decisions have to be made regarding what to teach and what to leave out; what topics are considered to be more important than others and why. The field of education research is complex and, for students, the area can be overwhelming. In the experience of the authors, who have taught research methods at both under-graduate and post-graduate level over many years, students consistently describe their confusion and frustration at the sheer scope of the area and, in some cases, this acts as a disincentive.
Perhaps a useful starting point would be to focus upon the type of research that is most prevalent in education, on the assumption that students would be more likely to come across examples in the journals they read. If students are cognizant with the methods used in the research that they most frequently encounter then surely confidence would be increased. As journals are readily accessible for students, an investigation of relevant education journals would be a useful source in order to determine if certain types of research are published more frequently than others.
An investigation of this nature may reveal a profile of education research that could have implications for education researchers as well as assisting teachers and pre-service teachers. This article describes the authors’ endeavor to profile selected education research journals and the unexpected surprises encountered along the way; in particular, difficulties in the development of a suitable ‘mapping tool’. There may be implications for education researchers as well as teachers of research.
The Nature of Educational Research
Educational research is undertaken by a range of stakeholders including government departments and non-government organisations, but the majority of educational research, as with most disciplines, is undertaken by academics in universities. Educational research covers a broad range of topics such as curriculum and pedagogy, education systems (encompassing early childhood, primary, secondary education) and various specialist studies, including areas such as assessment, leadership, technology and gender.
Research needs of stakeholders vary. Education departments use research to inform teaching and curriculum practice, devise professional learning activities, target resources and improve system requirements. Non-government organisations may use research to develop teaching resources or provide information to improve services to a range of clients. Research that underpins the teaching and learning process is of particular importance to inform teacher practice. Universities usually require students to engage with the education research literature, whereby students undertake a unit in research methods or read educational research. With the growth of pre-service teacher education courses offered at the Master degree level in countries such as Australia, the requirement for research skills has escalated.
Research in education encompasses many different naturalistic, interpretative, hypothesis generating models as well as hypothesis testing models. A rich resource of text books is available for those studying the theory and practice of educational research: Burke & Christensen (2012), Punch (2009), O’Toole & Beckett (2013), Wiersma and Jurs (2009) and Yin (2012), to name a few. Due to the nature of research in an educational setting the majority of research utilises a hypothesis generation approach with a predominance of verbal qualitative data gathering.
The reporting of educational research is usually presented in a range of publications such as academic journals, including online journals, professional magazines and books. Academic journals are a pathway that allows for the results of research to be released quickly into the public space. The content of academic journals also contains opinion papers, book reviews and editorial pieces; however, in some journals, the distinction between position/opinion papers and reports of research are left to the reader to discover, which can be a problem for students and inexperienced researchers. Nevertheless, articles in journals are a readily accessible starting place for students of research methods.
Several studies have attempted to map the type of research methodology used in various educational research; for example: Murray, Nuttall & Mitchell (2008), Nuttall, Murray, Seddon & Mitchell (2006), and Tuinamuana (2012). However, Burns (2000) contends that in general, most educational research tends to be classified as “case study research”, which has become an “over-arching” term to describe educational research that does not fit with experimental, historical or descriptive research methods.
Barriers exist regarding classification of different types of research methodology; in particular, where there is not a shared understanding of categories, such as method, data source, data gathering, and data analysis. The wide-spread use of general terms, such as “qualitative research” and “quantitative research”, and the term “mixed method research”, that largely refers to the use of both verbal data and numerical data in a research study, can cause confusion. The education research field is broad and interrelated so that students, novice researchers, and teachers new to reading research are often overwhelmed and unsure where to start.
Where should the novice begin?
The question of where to start the journey into the ‘research methodology forest’ would be answered in numerous ways depending upon the preferences or individual expertise of a lecturer. Pre-service teachers, and those commencing research for the first time, often seek advice regarding the ‘best’ method, or the ‘most useful’ approach, but it is not that simple. Students themselves bring to the situation their own experience and knowledge of research, both formal and informal. As teachers of research methods to pre-service teachers and early career researchers, over many years, questions to the authors, such as “where do I start?”, “it is difficult to know who to believe when one lecturer talks about the same term in a completely different way” and “what research method is most useful for teachers?”, were often followed by complaints about the daunting size of the task and difficulty in reading research reports in education journals.
For the novice some knowledge of research methods would be essential for reading and understanding research reports in order to make a judgement of the usefulness of the findings to their situation. The absence of information about the research process deters understanding no matter what level of research expertise the reader brings to the task. Indeed, education doctoral students attending a recent conference session, given by one of the authors, expressed concern with inadequate information provided in research journal articles about the methods used, data gathering techniques and subsequent data analysis. Comments such as “it is often not clear what is being reported when components, such as how the data were collected, are missing” and “I expect to read details on the data source or data gathering but sometimes this information is just not there”, as well as comments about the difficulties encountered by students in “identifying the type of research methodology used in educational research” (Knipe& Bottrell, 2013). It seems that pre-service teachers are not alone in their concerns about reading and understanding education research.
If particular types of research methodology are more frequently used by educational researchers, such as case study as claimed by Burns (2000), then there could be justification in placing an initial emphasis on case study methodology as a starting point in teaching research methods. As it is more likely that students and early career researchers would encounter this method in educational research journals, they would have a useful starting point for reading research and designing a research study. From the confidence gained through knowledge of one method of research, students could be encouraged to use that knowledge as a springboard into other research methodology.
Developing a “Mapping Tool”
Methods of classifying research into various categories and the development of instruments used have been reported in many disciplines, from early classifications by Cooper (1984) in social science to more recent classifications in areas such as Sports Science (Williams & Kendall, 2007) and Marketing (Ensign 2006). In categorizing educational research methods, an early attempt by Barr etal (1931) identified eight areas and, more recently, Isaac & Michael (1995) designated nine categories. Books on research are mostly organised by chapters that address the various aspects of research and tend not to be arranged by methodological classifications.
A review of categories used in books on research, including text books on education research, showed that some text books are structured according to particular designated research methodologies, such as ethnography, case study, phenomenology, descriptive and experimental, including extensive description and detailed features on each research method. In other text books, research methodologies, approaches to data gathering and analysis are addressed as separate entities. Concepts such as ‘research paradigms’ are often dealt with as a category separate from research methods.
Some text books on research methods have titles relating to “qualitative” research that focused upon the characteristics of naturalistic enquiry prevalent in education research, together with an emphasis on gathering verbal data from subject and/or researcher. By comparison, very few texts were found with the title “quantitative research”, but some authors such as Burns (2000) designate a section in their book titled “quantitative research”. ‘Qualitative’ and ‘quantitative’ are terms used to describe a group of research methods, types of data gathered, and data analysis techniques. For the purpose of this project and for reasons of clarity, the terms ‘qualitative’ and ‘quantitative’ were not used as categories in the development of a ‘mapping tool’.
In classifying educational research into mutually exclusive categories, the focus was upon research methodologies, separate from data gathered, sources of data, and data analysis techniques. Four categories were designated as follows:
Source of Data (e.g. teachers/students/school administrators/parents/non-school personnel etc)
Data Gathering Technique (e.g. interview/observation/survey/existing data etc)
Data Analysis Techniques (e.g. Categories/Themes/Open and axial coding/statistical analysis etc).
Research Methods (e.g. Case Study/Action research/Field study/ Quasi-experimental, Developmental, Historical etc).
A category for “sampling methods” was included together with a category for the reporting of reliability/dependability and validity/authenticity of the data gathering tools. The ‘mapping tool’ was called the Journal Article Research Analysis (JARA) Schedule and details of the complete categories, definitions for items, and results of the use of the JARA Schedule in a research project, will be presented elsewhere.
Application of the JARA Schedule
For early ‘test runs’ of the JARA Schedule, journals selected for review varied from one complete issue of a journal to analysing all issues for one year. Journals, national and international, were drawn from four major areas in education: namely, Educational and Developmental Psychology, Education Research, Teacher Education and Education Administration.
Generally, research journals contain reports of research together with position statements or opinion and book reviews. However, in selecting education research journals to use for the ‘test run’, it became apparent that there was a far greater proportion of position/opinion papers than expected - in some cases up to half of all articles were found to be opinion/position papers. Indeed, some of the opinion papers were presented as ‘research’ but closer scrutiny revealed that the paper was merely an informal report with no evidence of systematic investigation. The ratio of research to non-research articles in a significant number of journals purporting to be research journals was an unexpected discovery.
The absence of information regarding methodology, data gathering, and/or data analysis was another surprise. One would expect that a research article in a journal designated as having a research focus would include details of the research process, as well as reference to sampling methods, reliability and validity. In some instances, an author may have claimed to have used a particular method (for example, case study) but the description of the procedures followed did not meet the criteria for case study research, according to widely accepted definitions contained in text books on research methods.
For further ‘test runs’ of the JARA Schedule, the authors sought the assistance of two very experienced researchers both of whom had taught research methods for many years at the Master and Doctoral level. The team of four, independently, applied the JARA Schedule to a selection of eight articles. Discussion prior to using the JARA Schedule clarified definitions of the categories to the satisfaction of everyone in the team. After the scoring was complete discrepancies in scoring were discussed. It became clear that a major problem was with research reports that failed to include information about how data was gathered and analysed, or where the information provided, regarding procedures followed, was inadequate. In some cases, reports did not include a description of methodology, leaving the team to provide their own interpretation. The JARA Schedule team considered such interpretation to be unsatisfactory so a scoring descriptor of “not included /not clear” was added to the coding.
The category of ‘research methods’ stimulated much discussion and all members of the team were surprised that, despite our experience in the area of education research, we were not as clear in our understanding of definitions as we would like to have been. The discussion that followed as the team clarified and refined definitions of research methods, with extensive consultation of text books on research methods, served to strengthen their understanding and expertise in the research area. In this sense, the JARA Schedule, became a professional learning tool for the research team. Later “test runs” with nine other academics revealed similar lack of certainty in definitions. This seems to suggest that definitions and understanding of research methods may have been taken for granted, and the authors connected this with the confusion expressed by the first-time research methods students.
The team reviewed the JARA Schedule categories and the definitions with reference to explanations provided in research methods texts, expanding the definitions and providing supporting explanations where necessary. The revised JARA Schedule was trialed with a group of 44 students at the post-graduate level undertaking an introductory research methods unit as part of a Master of Teaching degree. Results showed a high level of agreement for all categories except the category of ‘research methods’. However, the feedback from pre-service teachers indicated a high degree of satisfaction with using the JARA Schedule because the process provided a comprehensive overview of the field that served to expand their understanding of research methods in general. The following comments illustrate reactions from pre-service teachers;
“we were given an understanding of a wide range of methods and procedures, and that has given us confidence”
“this has been one of the most valuable aspects of the course, offering a clearly organised method of making sense of the research maze, over a short face-to- face period. I know I will be much more competent in deciding what is useful research for my daily practice”
“I get it now, I am much more able to see through the big words and evaluate the important things that go to make good quality research.”
“I am still confused but I see how important it is for me to look carefully at what people say they are doing, is it useable research and what is actually there”
The JARA Schedule emerged from the problems faced by two university lecturers as they attempted to negotiate their way around the plethora of ‘research’ on offer in education, in order to make research interesting and comprehensible to pre-service teachers. The initial task was to identify research methods most frequently published in journals. However, the most useful and unexpected outcome from the process of developing the JARA Schedule has been the creation of a viable learning tool for teaching research methods. The JARA Schedule could be used as a teaching tool for students undertaking research methods classes, as well as a resource for teachers to enable professional learning options that provide experienced teachers with the skills to critically evaluate research studies relevant to their role as a teacher and to classroom practice.
Bibliography
Barr, A. S. ; Almack, John C. ; Ayer, Fred C. ; Dashiell, J. F. ; Gates, Arthur I. ; Good, Carter V. ; Johnson, Palmer O. ; Kelley, Truman L. ; Mccall, William A. ; Ruch, G. M. ; Symonds, Percival M. ; Toops, Herbert A. ; Trabue, M. R. ; Whitney, F. L. ; Woody, Clifford ; Kilpatrick, William H. ; Henmon, V. A. C. ; Freeman, Frank N. (1931). Symposium on the Classification of Education research. Journal of Education Research, 23(5), 353-382.
Burns, R. (2000). Introduction to Research Methods. 4th edn. Longman Frenchs Forest, NSW.
Cooper, H. (1984). The Integrative Research Review. A Systematic Approach. Applied Social Science Research methods Series Volume 2. Sage Publications.
Ensign, P (2006). Ensign International Channels of Distribution: A Classification System for Analyzing Research Studies. The Multinational Business Review, 14 (3), 95-120 Emerald Group Publishing Limited.
Huberman, A. & Miles, M (2002). The qualitative researcher’s companion. Thousand Oaks, California. Journal of Educational Research, 23(5), 353-382.
Isaac, S. & Michael, W. (1995). Handbook in research and evaluation: a collection of principles, methods, and strategies useful in the planning, design, and evaluation of studies in education and the behavioral sciences. 3rd edn San Diego, Calif.: EDITS Publishers.
Knipe, S. & Bottrell, C, (2013) Barriers to Reading and Interpreting Research. 2013 European Conference for Educational Research. Turkey.
Murray, S. Nuttall, J. & Mitchell, J. (2008). Research into initial teacher education in Australia: A survey of literature 1995-2004. Teaching and Teacher Education, 24 (1), 225-239.
Nuttall, J. Murray, S. Seddon, T. & Mitchell, J. (2006). Changing Research Contexts in Teacher Education in Australia: Charting new directions. Asia-Pacific Journal of Teacher Education. 34 (3), 321-332.
O’Toole, J & Beckett, D (2010). Educational research: Creative Thinking & Doing. 2nd edn South Melbourne, Victoria, Oxford University Press.
Punch, K (2009). Introduction to Social Research, Quantitative & Qualitative Approaches. Thousand Oaks, California, Sage Publications.
Tuinamuana, K. (2012). What do ATEA people research and why? Conference Presentation Australian Teacher Education Conference, Melbourne, 2012.
Wiersma W. & Jurs S. (2009). Research methods in education: an introduction 9th edn. Boston: Pearson/Allyn and Bacon.
Williams, J & Kendall, L. (2007). A profile of sports science research (1983-2003) Journal of Science and Medicine in Sport 10, 193-200.
Yin, R. (2012). Applications of case study research. 3rd edn Thousand Oaks, Califorina, Sage Publications.
