In Part 1 of Microfinance Impact Assessment we discussed what impact assessment is and it’s perception in the Micro finance Industry.- Read here.
In Part 2 we discuss how the to go about the Impact Assessment Process in a Microfinance Company
Impact Assessment Process
Impact assessment is a comprehensive exercise and should be systematically done. The broad phases in carrying out an IA exercise are;
D. Analysis and report writing
The first step in microfinance impact assessment is planning. Planning has to be done jointly with the stakeholders which may include MFI representatives, financial supporters etc. With these stakeholders together it is first necessary to understand the context of the MFI and set clear objectives of the study and to understand the use of IA results. Once there is clarity about these questions, the research approach and method are built around it.
There is no single purpose for conducting an IA exercise it is necessary to have clarity on why IA is being conducted and what do we want to know from the exercise. This sets the objectives of the study and the probable outcomes. Therefore, an MFI planning an IA must first answer some basic questions like;
• What it wants to know through IA?
• Why it wants to know and measure?
• How will the outcomes of IA help or how will the outcomes be utilized?
Clarity on objective will help in deciding the focus of the study, methodology and intensity. Objectives have implications not only for the audience, but also influence whether the assessments will address questions that are more theoretical or practical and whether the findings can be generalized or are context-specific. The objectives and usage of the study will decide how accurate the findings need to be and accordingly methodology can be chosen. Whether it will be simple approach, moderate approach or complex approach.
Understanding Programme and Context
Before the IA exercise is initiated it is necessary to understand the context and mission of. the MFI. Carrying out IA in isolation, without knowledge of the context and mission of the MFI, will be a meaningless exercise. In the context it is necessary to see the physical environment, socio-cultural settings, economic and livelihood scenario. Understanding the major sources of income, cash flows and poverty profile of the clients. Context has influences on the operations of the MFI such as its average loan size, operating costs and eventually sustainability.
Other factors that have to be seen are presence of other institutions, government programmes/polices in the region, environmental factors/natural calamities and migrations. It is also necessary to understand the mission of the MFI and relate it to the context. Mission is what an MFI wants to achieve and the impact it is seeking through its programme. Hence, IA exercise has to be aligned with it.
Once the mission of the organization and context is clear, the overall framework of the IA study is built around it. Understanding of context and mission of the organisation is important as without prior knowledge of these the methodology and the objectives of the study cannot be properly designed. Eg and MFI which has the mission of improving the livelihood through increasing the household income. Such an MFI may be giving loans to main earner of the family, who could be a man. However, if this context is not known and an impact assessment is focused on women than it can give misleading results.
Similarly, if an MFI is operating in an area which has no literacy or the area is completely tribal with very little investment opportunities then impact has to be seen in this light. Another example could be of Islamic countries. In Islamic countries many people do not take loan as interest collection is not considered good in Islam. Hence, credit off-tale could be low if product is not suitably designed. Such socio-religious context is definitely important to know before conducting any impact assessment exercise.
The objectives of the study are made specific through a number of key questions that need to be explore during the IA. These would basically flow from the broader purpose of the study such as to what is the information required from study? Who will use the information? How will the results be used? From such broader questions specific questions can be formed such as: Has microfinance improved decision-making capacity of women? Has it improved asset base of clients? Has income of clients increased? Such questions will make the focus of the sharper and designing the study, analysis and results from the study will become more objective.
Once the planning is over, the researchers have clear ideas about the outcomes expected from the study and context of the MFI. Based on this information, the researchers have to decide the exact methodology of the study. Researchers have to decide how specific questions sought from the study will be approached. The methodology is decided based on the following factors:
A study can be good enough for certain purposes with being moderately accurate while for some purposes high level of accuracy is required. The higher the level of accuracy required, the more will be the time, effort and money required for the study. Amore intensive exercise will use greater resources. While conducting the IA study, it is necessary to decide the level of accuracy required based on the utilization of results from the study. The objectives of study and the key questions sought will guide the researchers in deciding the level of accuracy needed and thereby deciding the methodology.
Human Resource Availability
Practical availability of human resource is a major factor. One may not get people in required number or of required skill set. While designing the study and methodology it is necessary to see the quantity and quality of human resource available. The methods have to be designed keeping this constraint in mind.
Resources:Time and Money
Time and money are very important resources. Sometimes agencies conducting IA want results within a specific time period. So if time is a constraint one obviously cannot apply long elaborate methods, which will take time to yield results. Similarly money is a major factor. Some of the IA studies can be very expensive. Methodology has to be decided according to the funds available. Basically, it is a matter of trade off between accuracy, scope of study and these constraints. The higher the accuracy, the higher will be resources required. If ample resources are unavailable the accuracy has to be compromised to certain degree.
Once the objective of the study and resources available are known, one has to work on deciding the framework of the study. The first thing is to decide whether the study will have a simple, moderate or complex approach (‘Impact Assessment Methodologies For Microfinance: Theory, Experience And Better Practice’, David Hulme). This is decided on the basis of objectives of study, level of accuracy required and resources available as mentioned above.
Simple approach will basically provide quick results with moderate accuracy based mostly on triangulation. Triangulation is simply gathering information from multiple sources and cross verifying. The resources utilized are less here and methodology, analysis and interpretations are simple. Sample size and survey are of small scale. The clients can be compared with non-clients or client in different categories. Results are cross-checked by participatory appraisal, in which community is involved.
If a baseline study, that is initial data before the intervention (here microfinance) is made, is not available then a recall methodology is used, in which people are asked to explain the situation as it was in past. The study completely focuses on deriving answers for the key questions set for the study. If the answers can be derived through interviews in focused grouped discussions (focused group discussions are discussion in group on a specific topic) then even surveys can be dropped. Basically, different methods can be used here to get the answer of the key questions. These studies are used by microfinance practitioners to improve their programmes or donors to get some idea of impact of microfinance programme.
The moderate approach involves substantially higher costs than the simple approach but also yields higher levels of reliability. Here conclusions are made through statistical analysis rather than simple triangulation of information. Such studies may take long time up to two to three years sometimes. Such studies are of importance to policy-makers, donors for deciding their strategies or to some extent MFIs. The approach uses more sophisticated methodologies and analysis. Sample is carefully picked, methodical surveys are conducted and comparisons are made with carefully matched control groups, rather than triangulation.
The survey would involve more than one visit to clients with a significant interval between two visits to compare the status. The study uses a mix of quantitative and qualitative analysis to substantiate results. Quantitative analysis means mathematical and statistical analysis while qualitative mainly relies on deriving conclusions through illustrating observations, anecdotes and incidences.
The complex approach generally is a long-term research, which can range between 4-6 years. These are called longitudinal study and start with collection of baseline data for the client sample as well as for the carefully selected control groups. The samples are then visited at least three to four times and changes are recorded. Such studies are mainly used by policy makers and researchers. In this approach generally large sample size is carefully chosen to represent all key features of the client population.
This is compared against a carefully selected control selected control group, so that the number of households surveyed is likely to be between 750 and 1500 or even more. A much wider set of variable is measured and focus is on high accuracy. The analysis of data is rigorous using advanced statistical and econometric models. Needless to say, such studies are cost intensive as the time taken in the study is long involving higher numbers of working hours of researches, logistic cost are high and also services of high cost statistician or econometricians are require.
These are the three major approaches for the study. Once the approach has been decided key research questions are addressed. These questions are; what should be the size of the sample? Who should form the control group? What should be the size of control group? What should be the location of the study? Whether the study will be qualitative or quantitative? Generally, most of the researchers prefer to have a mix of two. So sample for qualitative research has to be decided. Planners have to decide the number of case studies, focused group discussions or number of interviews to conduct.
Regarding the location, one may choose multiple locations or a particular location depending on the outcomes required. If a representative sample of entire operational area of the MFI is required then multiple locations can be picked, however if certain specific observations or impact of some specific phenomenon/event is required then one specific location could be chosen. While choosing location planners also have to see logistical aspects such as transport facility, safety, staying arrangement, likely support from MFI staff, etc.
Deciding the Team
During the Design phase a team has to be finalized. It is important to carefully pick the team and assign specific responsibilities. For collection of data, it is generally good if one can find a team of associates who have done similar work earlier and can understand research. It is also important that the team is given a thorough training on the research and on their specific responsibilities.
The questionnaire that needs to be filled should be clearly explained to the team. It is always better that 5-10 such questionnaires are filled by the researchers as a practice before starting with the actual work. Each question, table etc. should be absolutely clear to the associates responsible for data collection; then only they can administer the questionnaire correctly. Apart from the questionnaire the researchers should be trained on the code of conduct with those they interact with.
The purpose of research, intended use of data, confidentiality of responders, the fact that responses will not affect their relation with the MFI and freedom to refuse to certain question or not participate in the research at all, should be made clear to all the people approached. Last but not least, the associated has to be fluent of the mother tongue of the people interviewed. Ideally, the questionnaire should be in that language also. If that is not the case the training of the associate needs to ensure that the native-language-explanations accurately reflect the (English) questions.
During the implementation phase actual Impact Assessment exercise is conducted on the fields. It is important that those leading the research should be closely involved in the client interaction and data collection during the initial phase. It is generally better to first have a pilot of at least one week. This can immediately help in judging whether the methodology and questionnaire designed are working fine.
It can also bring out the fact how everything is working out logistically in the filed. Practical issues such as communication problem, understanding of associates and client responses will be understood. Based on the outcome of pilot, changes can be made in questionnaire, or if any further training of associates is required and fine-tuning other aspects. Pilot will also confidence to the associates and first hand experience of field and collecting data before going for actual research.
During the actual data collection process care should be taken that the staff of MFI involved should not be able to influence the responses. If it is observed that even mere presence of MFI staff is influencing the responses of the clients then the staff should be explained to stay way from responders. Care should be taken be taken to ensure that even the persons recording the information are not biased or guided by their own experiences while recording the data. The notes on all observations should be properly taken within 12 hours of data collection.
In fact every evening after the field work the notes on observations, interviews, case studies should be properly taken when they are still fresh in the memory. The notes should be taken clearly and as illustratively as possible, as they may be used much later in time and may be used by different analysts who need to understand the data clearly.
Notes should capture finer details like who dominated the discussion, inter-relationships, past events – which were important in context of the responses etc. It is always better that the research team sits in the evening and discusses the day and share their experiences, observations before writing the notes. Even after writing the notes one may share it with other fellow researchers to see if it is understandable.
D. Analysis and Report Writing
After the field work is concluded, the challenge is to analyse the data to provide maximum output for addressing the key questions set in the planning phase and to present the findings in a coherent manner through a report.
The data analysis can be qualitative, quantitative or a mix of two. However, for a good IA study it is always better to have a mix analysis so that one can objectively substantiate the findings through quantitative means. When a mixed method is used, the qualitative and quantitative analysis should be used effectively to emphasize a finding, proving a hypothesis, expanding a discussion or making an argument. The two methods should complement each other rather than appear as separate components of the report.
The information gathered has to be analysed to address the key questions and reasons for the study. Quantitative models of data analysis provide ways of examining, comparing and contrasting and interpreting meaningful patterns. However, the analysis and the meaningfulness of conclusions have to be seen in context of the key questions set for the study. Qualitative analysis such as adding a case study or observation can substantiate findings and make the study interesting.
While writing the report, care has to be taken that only relevant information is put. During the study a lot of information may have been collected but it has to be pruned and simplified to make relevance for the report. Unwanted information should be avoided as it may confuse the reader. The text of the report should flow in a coherent manner and should not appear disjointed. The text should be guided by the outcomes of the analysis and the observations in the field rather than by the impressions of the writer or researcher.
It is very important to present the data in the most informative manner. Graphs and tables are two most important ways of representing information or to show trends. If properly used tables and graphs can make complex observations easy and explicit to represent. At good report should be structured well and should have at least these following sections.
• Executive Summary
• Methodology followed and its reason
• Constraints of the study
• Body of the study having observations, analysis and findings in detail
• Inferences and conclusion