Mama Global Meplan Final All

Global Monitoring and Evaluation

Framework
July 2012

This publication was made possible by the generous support of the American people through the United
States Agency for International Development (USAID), under the terms of the Leader with Associates
Cooperative Agreement GHS-‐A-‐00-‐08-‐00002-‐00. The contents are the responsibility of the Maternal and
Child Health Integrated Program (MCHIP) and do not necessarily reflect the views of USAID or the
United States Government.
The Maternal and Child Health Integrated Program (MCHIP) is the USAID Bureau for Global Health’s
flagship maternal, neonatal and child health (MNCH) program. MCHIP supports programming in
maternal, newborn and child health, immunization, family planning, malaria, nutrition and HIV/AIDS,
and strongly encourages opportunities for integration. Cross-‐cutting technical areas include water,
sanitation, hygiene, urban health and health systems strengthening.
TABLE OF CONTENTS
Acknowledgements ...................................................................................................................................... ii
List of Acronyms .......................................................................................................................................... iii
Preface ........................................................................................................................................................ iv
I. Introduction ........................................................................................................................................... 1
A. MAMA Theory of Change Model ..................................................................................................... 2
B. Core Constraints, Facilitating Factors, and Critical Assumptions .................................................... 7
II. M&E Framework .................................................................................................................................... 8
A. Features .......................................................................................................................................... 8
B. Key M&E Questions ....................................................................................................................... 10
C. Operationalizing the MAMA M&E Framework ............................................................................. 11
III. Resources and Timeline ....................................................................................................................... 27
A. M&E Resources and Organization ................................................................................................. 27
B. Data Sources and Data Collection Tools ........................................................................................ 28
C. Portfolio Review ............................................................................................................................ 28
D. Plan for Data Analysis, Review, Reporting, and Feedback ............................................................ 28
E. Assessing Data Quality .................................................................................................................. 29
F. Reviewing and Updating the M&E Framework ............................................................................. 30
Appendix A: Sample Performance Indicator Reference Sheets ................................................................. 31

LIST OF TABLES AND FIGURES
Table 1: Key Target Audiences of the MAMA Initiative ............................................................................. 2
Figure 1: MAMA Theory of Change Model ................................................................................................. 4
Table 2: Levels of Data Collection and Relevant Audiences ...................................................................... 9
Table 3: MAMA M&E Framework (Activities, Outputs) ........................................................................... 13
Table 4: Indicators for Monitoring and Evaluation .................................................................................. 15
Table 5: Overview of M&E Activities in Country ...................................................................................... 27

MAMA M&E Plan June 2012 i

ACKNOWLEDGEMENTS
This M&E framework was written by Soumya Alva of MCHIP/ICF International. It incorporates input from
presentations and discussions at the MAMA M&E brainstorming meeting of technical staff of the Bureau
of Global Health USAID, MAMA South Africa, MCHIP, Johnson & Johnson, Baby Center, and United
Nations Foundation in Washington, D.C. on August 3, 2011 and from discussions with implementation
teams in Bangladesh and South Africa. It includes feedback from the stakeholders represented at this
meeting, in-‐country implementing partners, and technical experts identified by the MAMA partners to
finalize the choice of indicators that measure progress toward project goals, and guidance on data
collection for each indicator.
We would like to thank the MAMA global and country partners for numerous discussions and reviews of
drafts:
Erin Boardman—formerly with USAID, who coordinated discussions, phone calls, and meetings
Alice Lin Fabiano—Johnson & Johnson
Kirsten Gagnaire— MAMA
Patricia Mechael — mHealth Alliance
Daphne Metland—Baby Center
Cathryn Meurn—mHealth Alliance
William Philbrick—mHealth Alliance consultant
Jennifer Potts—previously with mHealth Alliance
Ananya Raihan—DNet, Bangladesh
Sandhya Rao—USAID, Bureau of Global Health
Joanne Stevens—MAMA

Others who made important contributions are:
Kai-‐Lik Foh—GSMA
Craig Friderichs—GSMA
Kanta Jamil—USAID, Bangladesh
Elizabeth Jordan—Johns Hopkins University
Alain Labrique—Johns Hopkins University
Lauren Marks—USAID, South Africa
Pamela Riley—SHOPS, Abt Associates
Jon Stross—formerly with Baby Center
Meena Umme—USAID, Bangladesh

MCHIP staff members who contributed to this document are:
Samantha Herrera, Matt Holtman, Sarah Jones, Barbara Rawlings, Leo Ryan, and Kirsten Unfried.
MAMA M&E Plan June 2012 ii

LIST OF ACRONYMS
CHWs Community Health Workers
HWs Health Workers
ICTs Information and Communication Technologies
IP Implementing Partner
M&E Monitoring and Evaluation
MAMA Mobile Alliance for Maternal Action
MCH Maternal and Child Health
MCHIP Maternal and Child Health Integrated Program
MDG Millennium Development Goal
MNC Maternal and Newborn Care
MoTeCH Mobile Technology for Community Health
QED Quasi-‐Experimental Design
RCT Randomized Control Trial
MAMA M&E Plan June 2012 iii

PREFACE
In 2010, 287,000 maternal deaths occurred worldwide.1 Despite a 47 % decline in maternal deaths
between 1990 and 2010, developing countries accounted for 99 % of these deaths, mostly in Sub-‐
Saharan Africa (56%) and South Asia (29%). In addition, 2.6 million stillbirths occur every year, with
neonatal deaths in equal numbers.2 As the target date of 2015 approaches, efforts to achieve the United
Nations Millennium Development Goals (MDG) broaden and intensify. Two MDGs focus on maternal
and child health (MCH): Goal 4 is reduction of the under-‐five mortality rate by two-‐thirds between 1990
and 2015; Goal 5 is reduction of the maternal mortality ratio by three-‐quarters between 1990 and 2015
and universal access to reproductive health services by 2015.3 Global efforts such as “Every Woman
Every Child” led by the UN Secretary General seek to intensify global action to improve the health of
women and children globally by working with leaders from governments, multilateral organizations, the
private sector, and civil society.4
In order to measure and advance the pace of progress pursuant to these goals, the Countdown to 2015
Initiative, a collaborative effort by concerned individuals and partner organizations, tracks coverage
levels for health interventions proven to reduce maternal, newborn, and child mortality.5 It calls on
governments and development partners to be accountable, identifies knowledge gaps, and proposes
new strategies for MDGs 4 and 5.
THE ROLE OF MOBILE TECHNOLOGY IN IMPROVING MCH OUTCOMES

In the context of slow progress towards achievement of the MDGs amidst funding constraints, the
international health community is identifying new tools and other creative opportunities to improve
MCH outcomes. eHealth is increasingly being used in developing countries to address these issues.
When used appropriately, eHealth can catalyze, support and monitor health improvements at scale and
accelerate achievement of national and global development goals, including the MDGs.6
Since the first World Telecommunication Development Conference in 1994, the use of information and
communication technologies (ICTs) to improve access to health care services in developing countries has
also grown considerably. Today, close to 90 % of the world’s population has access to mobile networks.
Because many of these 5.9 billion mobile phone subscribers are in developing countries, mobile phones
also offer another opportunity to improve access to health.7 Because of improved mobile infrastructure
and near ubiquitous access to mobile phones by individuals in developing countries, mobile technologies
are now being used to reduce maternal and newborn mortality and morbidity by providing better access
to maternal and infant health services, improving efficiency of health service delivery, enhancing

1
WHO. 2012. Trends in Maternal Mortality: 1990 to 2010: WHO, UNICEF, UNFPA and The World Bank estimates
http://whqlibdoc.who.int/publications/2010/9789241500265_eng.pdf.
2
Lancet Stillbirth series. http://www.lancet.com/series/stillbirth.
3
United Nations. Official List of MDG Indicators
http://unstats.un.org/unsd/mdg/Host.aspx?Content=Indicators/OfficialList.htm.
4
United Nations. Every Woman Every Child http://www.everywomaneverychild.org/.
5
Countdown to 2015. http://www.countdown2015mnch.org/.
6
The Bellagio eHealth Evaluation Group. 2011. Call to Action on Global eHealth Evaluation. Consensus Statement
of the WHO Global eHealth Evaluation Meeting, Bellagio, September 2011.
7
International Telecommunication Union. 2011. The World in 2011. ICT Facts and Figures. http://www.itu.int/ITU-‐
D/ict/facts/2011/material/ICTFactsFigures2011.pdf.
MAMA M&E Plan June 2012 iv

collection of health information to refine public health interventions, and improving monitoring and
evaluation of health system activities for enhanced planning and decision-‐making.8
In many countries, Malawi for example, these activities are aligned with country government national
priorities and health strategies and are being implemented under the direct supervision of the country
ministries of health supporting the government’s desire to improve health with technology.9 According
to the United Nations, more than 100 countries are now exploring the use of mobile phones to achieve
better health.10
Many mHealth projects such as “Mobile Technology for Community Health” (MoTeCH), a Ghana Health
Service project funded by the Bill and Melinda Gates Foundation, are currently underway. By providing
information to mothers and community health workers via mobile phones, MoTeCH uses a mobile
health information technology system to improve the quality and increase the availability of maternal
health care, with a focus on ensuring the continuum of care.
EVIDENCE BASE FOR THE MHEALTH-‐MCH RELATIONSHIP

“To improve health and reduce health inequalities, rigorous evaluation of eHealth is necessary to generate
evidence and promote the appropriate integration and use of technologies. To ensure effective and
appropriate use of eHealth systems, implementation must be guided by evidence from evaluations at all
design and scale-‐up stages.”

The 2011 Consensus Statement of the WHO Global eHealth
Evaluation Meeting of The Bellagio eHealth Evaluation Group
Even though there is general consensus that implementing mHealth and eHealth strategies has the
potential to improve maternal and newborn health, there is insufficient evidence to measure or quantify
the impact of these strategies on improved health outcomes. Most research in this emerging domain
has been qualitative, focusing on the user experience or the technical performance of mHealth
systems.11 Other research findings have been observational; the natural introduction of mobile
technology into populations under epidemiologic study has allowed the study of mHealth in the absence
of formal programs.
One example from the JiVitA population research site in Bangladesh documented the use of mobile
phones in obstetric and neonatal emergencies, even in remote, resource-‐limited settings. Over 50% of
women experiencing a near-‐miss delivery event reported that a mobile phone was used to contact a
provider or an ambulance. This study found that families with access to mobile phones, irrespective of

8
mHealth in the Millennium Villages Project.
http://cghed.ei.columbia.edu/sitefiles/file/Mobile_Health_within_MVP%20(1).pdf.
9
US Global Health Initiative Malawi. http://www.ghi.gov/country/malawi/index.htm.
10
United Nations Secretary General. 2010. Global Strategy for Women’s and Children’s Health.
http://www.un.org/sg/hf/Global_StategyEN.pdf.
11
GSMA. 2011. Improving the Evidence for Mobile Health. White paper prepared by AT Kearney
http://www.gsma.com/connectedliving/wp-‐
content/uploads/2012/03/atkearneyevidenceformobilehealthwhitepaper.pdf.
MAMA M&E Plan June 2012 v

household phone ownership, used them in emergencies and could find phones within their
community.12
Other research and observational studies are actively underway, examining the efficacy of mHealth
strategies to improve adherence to treatments, reduce harmful behaviors (smoking, drug use), and
manage chronic illnesses. With the near ubiquity of mobile technology, opportunities to study and
assess the impact on health outcomes have increased. Despite a growing evidence base for mHealth in
recent years, (most of the published data on mHealth has emerged in less than a three-‐year period),
robust research on the benefits and risks of implementing these technologies and on their
cost-‐effectiveness is still lacking.13
Since most of the work involving mHealth has been at the pilot or proof-‐of-‐concept stage, few of which
have been scaled to larger programs, predicting how investments in mHealth projects could be
beneficial at a national scale in a given country remains difficult. 14 Furthermore, the challenges inherent
in scaling these programs are difficult to calculate without adequate, well-‐documented experiences
taking such technologies to scale. A recent literature review of mHealth programs suggested a promising
role for mHealth strategies in facilitating preventive services including health education, antenatal care,
expediting emergency obstetric referrals and enabling health workers to collaborate and improve
delivery of care. However, the review also highlighted the lack of project evaluations and studies with
quantitative research designs to demonstrate a statistically significant impact.15
A recent review of barriers and gaps affecting mHealth “identified significant gaps in mHealth
knowledge stemming from the limited scale and scope of mHealth implementation and evaluation, a
policy environment that does not link health objectives and related metrics to available mHealth tools
and systems, and little investment in cost-‐benefit studies to assess mHealth value and health outcomes
research to assess success factors that weed out poor investments. The current mHealth evidence base,
mostly in the field of computer science (not health), is not sufficient to inform and influence
governments and industry partners to invest resources in nationally scaled mHealth initiatives.”16 There
is little evidence that adequately reflects how mHealth addresses current health system shortcomings to
help governments achieve national goals. There is a critical need for robust evidence to make the health
and investment case for scale-‐up and integration of mHealth programs.

12
Labrique, AB, S Sikder, L Wu, N Jahan, P Christian, R Klemm, and KP West, Jr. 2001. Mobile phone ownership and
use during emergency obstetric crises in rural Bangladesh. Seamlessly Mobile ICA Pre-‐Conference, May,
Cambridge, MA, USA.
13
GSMA. 2011. Building a Robust Evidence Base for Mobile Health. Preliminary Phase: Review of Current State.
14
Noordam, AC, BM Kuepper, J Stekelenburg, and A Milen. 2011. Improvement of maternal health services
through the use of mobile phones. Tropical Medicine & International Health, 16(5): 622–626.
15
Tamrat, T, and S Kachnowski. 2011. Special delivery: An analysis of mHealth in maternal and newborn health
programs and their outcomes around the world. Maternal and Child Health Journal. DOI: 10.1007/s10995-‐011-‐
0836-‐3.
16
Mechael, P, H Batavia, N Kaonga, S Searle, A Kwan, A Goldberger, L Fu, and J Ossman. 2010. Barriers and Gaps
Affecting mHealth in Low and Middle Income Countries: Policy White Paper. The Earth Institute Columbia
University; mHealth Alliance. p. 79. http://www.globalproblems-‐globalsolutions-‐
files.org/pdfs/mHealth_Barriers_White_Paper.pdf.
MAMA M&E Plan June 2012 vi

I. INTRODUCTION
In order for MAMA to achieve its overall goals of improving maternal, newborn and child health among
the target population in developing countries, there is need for an appropriate tool for planning and
management and monitoring and evaluation. The key objective of this M&E (Monitoring and Evaluation)
framework is to ensure that MAMA program outputs meet the needs of the target population. The
indicators presented in this framework can be used to monitor and report on progress in the
implementation of the various MAMA initiative components in countries.

A brief history of the Mobile Alliance for Maternal Action (MAMA)
Mobile Alliance for Maternal Action (MAMA), an innovative partnership developed in collaboration with the
White House Office of Science and Technology Policy and the US Department of State, was launched in May
2011. At the global level, MAMA seeks to achieve “scale, sustainability and impact” by creating a replicable
model for reaching low-‐income mothers and household decision-‐makers (husbands, mothers-‐in-‐law) by
increasing the impact of current mHealth programs, providing technical assistance to new mHealth models, and
improving methods of applying mobile technology to protect maternal health. MAMA’s three-‐year, $10 million
investment aims to create and strengthen programs in three countries – Bangladesh, India and South Africa –
and to enhance global capability of new and existing mobile health information programs for mothers in those
countries and beyond.

MAMA is a partnership between the U.S. Agency for International Development (USAID), Johnson & Johnson,
the United Nations Foundation, and BabyCenter and operates through a secretariat hosted by the mHealth
Alliance. It uses technology to improve health and nutrition outcomes among pregnant women and new
mothers and their infants in resource-‐poor settings. Given the explosion in mobile market penetration and
handset ownership in developing countries, MAMA empowers women in low-‐resource settings to improve and
protect their own health and that of their children and families. Using mobile technology, MAMA aims to deliver
time-‐sensitive, stage-‐based information on critical health issues directly to expectant and new mothers. These
messages will be consistent with national and state behavior change and communication strategies and will
aim to be endorsed by host country governments. Health topics covered include pregnancy advice and tips, local
health services, antenatal care, nutrition, birth preparedness, skilled attendance at birth, safe delivery,
breastfeeding, family planning, birth spacing, immunizations, child diseases, and other care-‐seeking behavior.

The initiative is expected to foster collaborations among similar initiatives in other countries to accelerate
efforts to reach millions of women who have mobile phone access around the world with critical health
information. At a later stage, MAMA could link to other services to ensure direct impact on health outcomes
beyond awareness and behavior change.

This M&E framework is therefore a resource with important implications for MAMA to—
• Identify performance gaps and setting goals for their improvement.

• Improve the quality of programs.
• Assess implementation of the initiative in line with overall priorities, particularly providing
health information to improve the health status of women and young children.
Through a strong global M&E framework and locally adapted M&E plans in the countries where it
operates, MAMA will build the evidence base on the effective application of mobile technology to
improve maternal health for future applicability in other countries. In accordance with USAID’s new
MAMA M&E Plan June 2012 1

evaluation policy,17 this document provides guidance on the M&E of programmatic responses to global
health challenges through MAMA by outlining M&E for the overall objective and identifying indicators
useful for improving program management by monitoring and evaluating progress.
The intended audience for the M&E framework will be project implementers in countries using this
document to guide the development of national M&E plans. It is possible that the experience from the
pilot phase in the three countries may result in changes to the global M&E framework and associated
indicators at a later stage.
As Table 1 indicates, MAMA engages multiple stakeholders working together to meet the overall health
objective.
Table 1: Key Target Audiences of the MAMA Initiative

Primary Targets of MAMA
1. Beneficiaries—Pregnant women, new mothers and household decision-‐makers
Key Stakeholders
2. Global MAMA Partners—USAID, Johnson & Johnson, Baby Center, mHealth Alliance, United Nations Foundation
3. In-‐country partnership of implementing organizations
4. Country governments
5. Mobile network operators
6. Corporate advertisers and
7. Future donors
Note: Other potential stakeholders could include health workers and health NGOs working on related issues.
In addition to the global MAMA partnership, an in-‐country partnership of an implementing organization
working in collaboration with country governments’ mobile network operators to deliver health
messages via mobile phones, corporate advertisers (who publicize the program and offer means of
subsidizing operating costs) and future donors play a critical role in successful implementation.
While this framework is applicable globally without a specific country context in mind, the in-‐country
partnership implementing MAMA will need to tailor it to the specific context in each country. The
changes in the framework in each country could be based on the nature of the partnership, business
model, target population identified, and localized content of mHealth messages, and type of mHealth
messages (data, voice, text).
A. MAMA THEORY OF CHANGE MODEL

MAMA’s Theory of Change model hypothesizes that a small group of international partners can work
together to—
• Catalyze locally driven, sustainable country partnerships.

• Create a global learning agenda that builds capacity among mHealth organizations by drawing
upon the country partnership experience to inform the larger mHealth community, and drawing
the best practices from the larger mHealth community to inform MAMA country programs.
• Focus on the outcomes of improved care-‐seeking and preventive behaviors for pregnant
women, new mothers, and their families in resource-‐poor settings.
• Develop a more robust set of resources and evidence base to support future mHealth initiatives.

17
USAID. 2011. USAID Evaluation Policy. EVALUATION Learning from Experience
http://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf.

This section outlines the processes involved in MAMA implementation for each country in which the
program has been introduced and its link to the MAMA Global Partnership and the global health
community at large. The successful implementation of MAMA is expected to contribute to a reduction in
maternal and neonatal/infant mortality in the long term. This long-‐term goal of mortality decline is a
global direction for the program, but progress towards this goal will be difficult to measure without a
carefully designed evaluation which takes into account that other programmatic efforts to reduce
maternal mortality may also contribute to changes in this indicator.
Other than mortality, however, important outcomes related to improved preventive behaviors and care-‐
seeking among pregnant women, new mothers, and decision-‐makers are measureable and within the
realm of influence of the MAMA initiative. The MAMA theory of change is presented in Figure 1 with the
expectation that if the program is able to achieve these intermediate or proxy outcomes in countries,
there is high likelihood that this intervention strategy will contribute to more distal mortality impacts. It
is noteworthy that although expectant and new mothers in low-‐resource setting are the primary end-‐
users of MAMA, depending on the country context, health messages could be sent to the mobile phones
of key household decision-‐makers.
MAMA is implemented in each country based on the availability of financial and technical input and the
necessary mobile network infrastructure. In each country, once the target population and key focus of
mHealth messages are identified, a sequence of activities to achieve these outcomes is envisioned.
Establishing in-‐country partnerships, business models and mHealth platforms is the first step in
developing a MAMA program in a country and involves working with various country partners including
mobile network operators, corporate advertisers, health foundations, and software companies, for
example, to ensure that—
• An effective country partnership is formed and mobile operators providing the service are
recruited.
• A business model funded through (for example) subscriber charges, subsidies by mobile
operators, or funds obtained from corporate advertisers, for example, is developed.
• A suitable software provider for an mHealth platform is identified.
• Such a platform is developed to suit the specific needs of the country program.


Figure 1: MAMA Theory of Change Model

Note: The above sequence of activities takes place within the policy and socio-‐cultural context within the country, enabling the various activities to be
performed as part of MAMA.

Since the target population for MAMA programs is based in low-‐resource settings, it is critical for a
program to be established with the objective of long-‐term sustainability. Once the program is set up in a
country, the following steps are expected to ensure the increase in knowledge of preventive and care-‐
seeking behavior in the household.
1. Appropriate localized content: At this stage, content for mHealth messages need to be location
specific and available in the language appropriate to the specific target population in each
country. Although the content broadly covers health information targeted to pregnant women
and new mothers, the focus of the messages may vary across countries. For instance, in some
countries messages will incorporate information on Preventing Mother to Child Transmission of
HIV. These messages will be sent in a timely manner to the target population. A set of adaptable
messages developed through the MAMA partnership is available for countries to access and
adapt at http://www.healthunbound.org/mama.
2. Increasing access to health information: With the distribution of mHealth messages, it is
expected that target women will have access to quality health information and will be aware of
necessary preventive and care-‐seeking health behaviors. This information could either directly
reach the women concerned, or be shared by other household members who are subscribers
and have access to mHealth messages or through informal communication within the
community.
3. Increased knowledge and engagement: Engagement has been defined as “actions individuals
must take to obtain the greatest benefit from the health care services available to them.”18 With
access to quality health information through the methods mentioned above, women are
expected to be more aware of their health needs and available health services. They are also
likely to be engaged and seeking health information through mHealth messages and possibly
disseminating this information to other women.
With this knowledge, it is expected that women (or their household members) will want to change their
behavior patterns towards improved preventive or care-‐seeking behavior based on their stage of
pregnancy or motherhood. For a majority of women, this intent will likely translate to changes in
behavioral outcomes as well.
Therefore, effective health communication is the means to ensure improvements in health-‐related
practices and in turn, health status.19 This series of processes follows the pathways proposed by
behavior change and communication theories over the last 2-‐3 decades, which have contributed to
substantial improvements in the health status of children in the developing world.
These communication processes play a key role in influencing health behavior change according to
major theories and conceptual frameworks of health behavior change such as the “health belief
model” and the “theories of rational behavior and planned behavior.”20 21 22 According to the health

18
Center for Advancing Health. 2010. A New Definition of Patient Engagement: What is Engagement and Why is it
Important? White paper. http://www.cfah.org/pdfs/CFAH_Engagement_Behavior_Framework_current.pdf.
19
Graeff, JA, JP Elder, and EM Booth. 1993. Communication for Health and Behavior Changes: A Developing
Country Perspective. San Francisco: Jossey-‐Bass Publishers.
20
Rosenstock, IM. 1966. Why people use health services. Milbank Memorial Fund Quarterly, 44 (3): 94–127.
21
Becker, MH, SM Radius, and IM Rosenstock. 1978. Compliance with a medical regimen for asthma: A test of the
health belief model. Public Health Reports, 93, 268-‐77.
22
Ajzen, I. 1985. From intentions to actions: A theory of planned behavior. In J. Kuhl & J. Beckman (Eds.), Action-‐
control: From cognition to behavior (p. 11-‐39). Heidelberg: Springer.

belief model, individuals will be more likely to change their health-‐related behavior if they recognize the
existence of a health risk or condition, if they feel they are susceptible to the risk or condition, and if
they regard the benefits of change to be greater than the barriers to change. Communication strategies
play a key role in influencing these perceptions. Health promotion through methods such as mass media
and peer education play an important role in translating readiness into action.23
According to the theories of reasoned action and planned behavior, people’s attitudes about a specific
behavior and their notions of what important reference groups think about that behavior influence the
intention to change, which in turn further influence behavior. Effective communication is critical to bring
about these processes in the country-‐level program which must take place within the socio-‐economic
and cultural context of each country, enabled by supportive government policies and actions that
facilitate cooperation of country partners and make use of appropriate mobile technology to
disseminate health information to the target population. It is therefore critical that to ensure that
country governments are aware of and support this initiative at all stages.
Although it is expected that this sequence of activities will be the norm for effective program
implementation in all countries, it is important to note that the actual choice of partners, target
population, methods of disseminating information (i.e. Web, data, voice or text messages), ability of the
target population to pay for services, types of business models, and subscriber charges will vary across
countries. Based on the mobile technology in use, methods of obtaining feedback from the target
population will also vary. Moreover, the socio-‐economic and cultural context of the countries will also
determine whether women have mobile phones or whether mHealth information will be transmitted to
the target population through other household decision-‐makers particularly men or older women in the
household.
These country-‐level processes take place under the purview of the MAMA global partnership. The two-‐
way global learning process between the MAMA global partnership and country programs facilitates
establishment and implementation of country programs which in turn contribute to experience-‐based
global learning. This global learning component also helps accelerate organizational performance by
focusing on organizations pursuing mHealth solutions to extend health information to mothers and build
competencies using existing expertise from MAMA partners, in-‐country programs, and colleague
organizations. The global learning component focuses on building the various competencies of
organizations engaged in mobile messaging to mothers using expertise from the MAMA partners, in-‐
country programs, and colleague organizations. The methods include:
Information exchange via an online community on mHealth Alliance HUB
•
(http://www.healthunbound.org/)
• Dissemination of online learning modules (on topics such as content localization, managing
partnerships, and securing specialized business services) and global tools (e.g., M&E framework
with core indicators, adaptable health messages)
• Possibly an “accelerator" initiative to help scale promising mHealth for MCH platforms.

The M&E framework and indicators for this element will be outlined in a separate document.

23
Murphy, EM. 2005. Promoting Healthy Behavior. Health Bulletin 2, Population Reference Bureau.
http://www.prb.org/pdf05/promotinghealthybehavior_eng.pdf.

B. CORE CONSTRAINTS, FACILITATING FACTORS, AND CRITICAL ASSUMPTIONS
Several external factors are critical to the success of MAMA in countries, but are not explicitly addressed
in this M&E framework, although each can potentially influence (either positively or negatively) the
success of the overall program. Some of these are listed below:
• The partnership established in the country. The development of sustainable public/private
partnerships is an inexact science, and the appropriate mixture will vary from country to
country. This M&E framework assumes that a strong partnership exists for implementing the
program
• Funding from corporate advertisers and the contributions of mobile network providers and
their subscription plans. This could vary considerably across countries and have a great impact
on the success of the project.
• Strong relationships with country governments and their support of MAMA Initiative
priorities. It is expected that the health care-‐seeking behavior of women will increase when
there is easy access to quality health services and trained health personnel.
• The role played by community health workers, women’s groups and other informal
community networks. This facilitates the comprehension of health messages and the transfer of
health information to women in the community.
• Available network infrastructure. Although mobile technology has a key role to play in mHealth
programs, its efficiency and effectiveness could be compromised by the available network
infrastructure in the country and the cost of transmitting information through mobile networks.
• Subscriber loyalty. Although beyond the scope of the program, it is possible that subscribers
could change their mobile network provider for various reasons unrelated to MAMA. For
example, incentives offered by other mobile network providers not associated with MAMA
could potentially encourage subscribers to leave the program.

II. M&E FRAMEWORK
A. FEATURES
The MAMA M&E framework includes performance metrics, indicator definition and reference sheets,
and guidelines for data collection and program evaluation (including in-‐depth studies and analyses to
answer specific questions). It is intended to be a living, functional guide for those implementing the
MAMA Initiative to assist with expansion of the program to other countries. It presents indicators for
routine monitoring of processes and outputs based on the MAMA Theory of Change.
This framework includes guidance for both monitoring and evaluating country programs, and these two
terms are briefly clarified below and distinguished further in Table 2.
• Monitoring is a periodic recurring task that begins at the planning stages of the project and has
the objective of improving project design and functioning. It involves ongoing data collection
and analysis, provides indications of progress and achievement of goals at regular intervals, and
measures project outputs. Monitoring data is routinely collected by the implementing partner
(IP), and used to inform decisions about the direction of the program on an ongoing basis and to
report general progress to stakeholders.

A core set of indicators are presented later in this document that can be used for cross-‐national
comparison across currently piloted MAMA countries and for program expansion in the longer
term. It is however important to note the differences in the tools, methods and approaches
when making these comparisons. The performance indicator reference sheets presented in
Appendix A will guide the development of data collection tools to ensure that data collection is
standardized across project areas using specific sources and that collected data are reported
appropriately and consistently. The final selection of indicators will take into account the
specific context of each country program, the timing and content of mHealth messages, burden
of data collection and reporting as well as their use in decision-‐making and performance
management in each country.

• Evaluation is a systemic collection and analysis of data to assess the conceptualization, design,
implementation, and utility of programs and requires data collection at multiple points in time
(at a minimum, baseline and end of project) in order to demonstrate change. Evaluation takes
place within a slightly longer timeframe, typically the end of the project.24 By examining longer-‐
term results, and identifying how and why activities succeeded, failed, or changed, evaluation
informs the design of future projects. The above definition applies best to outcome or impact
evaluations, which assess program achievements and effects, but evaluations could also include
formative evaluations on a small group of persons in the early stages of the program to test run
the activities and processes proposed in the program. Evaluation data may include some routine
monitoring data collected by the IP but normally require a higher degree of rigor to collect and
are gathered by an external evaluator to the implementing partnership.

Table 2 presents information on the different levels of data collection and audiences for this document.
While this M&E framework focuses primarily on identifying indicators and establishing processes for

24
Rossi, PH, and HE Freeman. 1993. Evaluation: A systematic approach (5th ed.). Newbury Park, CA: Sage
Publications, Inc.

routine monitoring, it also specifies research questions and methods for more specific evaluation studies
including the collection of qualitative information. MAMA monitoring activities will concentrate on
measuring change in MAMA process and output indicators on a regular basis and evaluation studies will
examine the effects of certain processes in the formative stage to improve design of the program and
focus on more long-‐term effects on outputs and health outcomes of the program. Other specific
analytical and operations research studies25 covering topics such as cost effectiveness, financial viability
and programmatic implementation issues may also need to be conducted.
Table 2: Levels of Data Collection and Relevant Audiences
Monitoring Evaluation Operations Research

Purpose Tracking progress, Evaluation of outcomes, Answers specific
program management/ success of program questions regarding
decision-‐making, program implementation
improvement
Who performs the In-‐country implementing Formative evaluation (IPs), Either in-‐country IPs or
activity partners (IPs) outcome/impact evaluation evaluation partners
(external organization)
Rigor Less rigorous than Very thorough, including Very thorough
evaluation, data collected on systematic collection of
a routine basis according to baseline and final data using
data collection and reporting sample survey methods
systems and processes
established
Examples of At each point in time, what is What is the impact of MAMA Financial viability
questions answered the extent of women’s on the increase in the sustainability
engagement as measured by demand/use of health analysis/cost-‐benefit
participation in the program, services by women? studies/cost
and level of knowledge on effectiveness of alternate
care-‐giving and care-‐ technologies/
seeking? approaches
Who most uses the In-‐country IPs Resource document for in-‐ To be done by external
results country IPs for formative organizations. This M&E
evaluation and evaluation plan does not address
partner for outcome/impact operations research
evaluation issues
Focus of this Information on core set of Guidance for countries on No
document key process and output development of appropriate
indicators, data sources and methodology, design and
data collection methods evaluation to assess program
performance focusing on
selected output or outcome
indicators

25
Operations research is “a process, a way of identifying and solving program problems. The goal of operations
research is to increase the efficiency, effectiveness, and quality of services delivered by providers, and the
availability, accessibility, and acceptability of services desired by users.” (Fisher, A. and J. Foreit. 2012. Designing
HIV/AIDS Intervention Studies: An Operations Research Handbook, Washington, DC: Population Council). It uses
qualitative and quantitative methods to examine issues related to program implementation to achieve the best
results.

This framework includes specific ideas and guidelines for analytical studies including impact evaluations
with information on data collection, and on methodologies, particularly qualitative ones. Other
information included in the framework guides the development of the M&E system in countries
providing specification for data collection, data analysis, review, reporting, and feedback to identify
performance gaps and improve the program. This framework seeks to clarify which data and indicators
are most relevant to program monitoring—and should therefore be included in the monitoring and
evaluation systems of implementing partners—which data and indicators are most useful for program
evaluation, which would normally be carried out by an external group to the implementing partnership,
and which data and indicators might apply to both.
B. KEY M&E QUESTIONS

At the country program level, based on the Theory of Change presented in Figure 1, the MAMA M&E
framework seeks to answer key questions about the process of implementing the program and its
expected health outcomes. These questions can be categorized according to both the perspectives of
the target beneficiaries and the program (partnership) itself.
Questions to be considered from the perspective of effects on women/beneficiaries
1. Has content for dissemination of appropriate evidence-‐based (maternal and infant) priority
health information been developed?
2. Has there been sufficient coverage of the target population? Are subscribers being retained? Is
information being delivered on a timely basis through mHealth applications? Has women’s
access to quality health information increased?
3. Has there been sufficient engagement of the target population through mHealth messages? Has
their awareness of health needs and available health services increased? Do they express an
intention to adopt healthy behaviors? Has there been an increase in their knowledge on MCH
over time?
4. Has there been an increase in the demand for health services among women? How is this
demand translated to greater uptake of services, both use of preventive health as well as care-‐
seeking behavior at the household level?
Questions to be considered from the perspective of the program/partnership effects on the in-‐country
program
1. Have a successful partnership, business model and functioning mHealth software platform been
developed?
2. Has the policy environment in the country improved?
3. Is there increased knowledge exchange within the community?
4. What is the extent of scale-‐up of the program within the country and expansion to other
countries?
5. Are other organizations (outside of the core implementer group) actively endorsing and
promoting the service?
In addressing all of the above questions, another way of examining the success of the MAMA program in
a country is by subdividing program considerations into three main types:
• Scale—Is the program reaching a significant number of target women expected?

• Sustainability—Is the program sustainable in the long run? Does it have a well-‐developed
partnership and operate in a supportive policy and socio-‐economic environment that
encourages expansion?
• Impact/effect—Has the program affected the envisioned increase in women’s care-‐giving and
care-‐seeking behavior? (This could also occur through the behavior of other household decision-‐
makers).
This document addresses all these three types of considerations, though not equally. Sustainability will
be addressed in more depth in special operations research studies on cost effectiveness and financial
viability.
Changes in health outcomes might also be evaluated through routine monitoring, but significant change
is more perceptible in the longer term. Impact is more appropriate to assess through long term
evaluation.
C. OPERATIONALIZING THE MAMA M&E FRAMEWORK

The process of contributing to the MAMA objective is presented in the M&E framework in Table 2. With
MAMA engaging several key stakeholders who play significantly different roles, this framework displays
the activities that need to be performed in order to obtain necessary outputs and outcomes. The
framework is aligned with the Theory of Change in Figure 1 and M&E questions specified earlier in
Section B.
The main MAMA project inputs at the country level include:
• Funding and technical assistance from the Global MAMA partnership to focus countries.
• Mobile network infrastructure.
• Inputs from the IP in each country.
• Inputs from other stakeholders, including corporate advertisers.
The key IP in country coordinates activities with in-‐country partners. These include:
• Coordination with the country government to ensure that MAMA plans are aligned with the
overall country health policy and objectives, and that content has been vetted and approved.
• Coordination with corporate advertisers, mobile network operators and other donors who could
provide funding or in-‐kind resources (e.g. mobile inventory) to the program in country in order
to develop a business model.
• Coordination with mobile network operators and software developers who provide the
infrastructure and technology for timely delivery of mHealth messages.
• Coordination with the content provider to provide localized content in the appropriate language
and ensure sufficient user testing.
• Coordination with governments and local organizations to support training of community health
workers.
• Coordination with branding and marketing organizations to develop a local brand, logo and
marketing strategy.
• Coordination with legal team to develop Terms of Service, privacy policy, and principles of data
use that align with any national regulations.

Typically, routine monitoring efforts in the various components of the MAMA program implementation
in countries presented in Table 3 cover processes, implementation activities, and outputs. Given the
short time period since the MAMA initiative piloted in selected countries and launched globally, it will
be difficult to discern sufficient change in health outcomes. As a result, a substantial number of
indicators identified in this framework focus primarily on processes, implementation activities and
health outputs. Health outcomes are examined further in a subsequent section.
From the program perspective, it is important to examine the process of “Initial setup” of the MAMA
program in a country, which includes obtaining information about the development of a coalition
partnership, a business model, and the development of an appropriate mHealth software platform.
Some of these activities will take place only when MAMA is initiated in a country, therefore routine
monitoring may not be appropriate. It is also possible that some of the above mentioned activities may
be a slow iterative process that may develop as the partnership evolves in each country. Similarly, the
number of donors could also increase with greater evidence of usage and the success of the program.
Therefore, although developments in these areas are necessary to examine progress, less frequent
monitoring may be sufficient. Furthermore, it is important to note that the types of partnerships and
standards and benchmarks for what is appropriate may be very country specific.
Section K focuses on long-‐term sustainability of the program, program expansion within the country,
and sharing of knowledge between countries. The 3 year period envisioned in the MAMA’s pilot phase is
unlikely to be long enough to substantiate a detailed analysis.
The analysis of the indicators specified in this framework also needs to be understood in the context
that the mobile network infrastructure in countries and the role played by country governments
regarding health service delivery are outside the scope of MAMA. However, they can influence the
achievement of MAMA’s objective and the overall impact of contributing to declines in maternal,
neonatal and infant mortality. For instance, it is possible that increased demand for health services
through a program like MAMA could in turn generate the need for better health services to meet the
existing demand.
Indicators
Building on the program activities and outputs listed in Table 3, a list of suggested indicators, including
health outcomes as measured by change in women’s care-‐giving or care-‐seeking health behavior is
presented in Table 4. These indicators are organized so they can be mapped easily to the Theory of
Change presented in Figure 1 and the activities presented in Table 3. Depending on the nature of the
program in each country, countries will likely use a shorter list of key indicators for M&E purposes to suit
their needs. A core list of indicators for cross-‐country comparison across MAMA country programs are
bolded and italicized in Table 4. Selection of these indicators is based on criteria including the
maximization of existing data, minimizing the burden on country programs, monitoring over the life of
the program, and contribution to the MAMA Global learning agenda. Country programs may exclude a
few of these indicators based on the focus of the program and timing and content of mHealth messages.
Key indicators on initial set-‐up and content development are also highlighted in Table 4, as they are
relevant in the initial stages of program development. More specific details on these indicators,
suggested questions to obtain consistent information, data collection methods and M&E procedures are
specified in Appendix A.

Table 3: MAMA M&E Framework (Activities, Outputs)
Impact: Reduction in maternal and newborn/infant mortality in resource-‐poor settings
Outcome: Improved household preventive health and care-‐seeking behavior
Activities Outputs
Initial Setup:
Coalition partnership
Ø Identify implementing partner(s) Ø In-‐country partnerships finalized
Ø Discussions with mobile network operators
Ø Discussions with corporate advertisers
Ø Discussions with country governments and target communities

Business model Ø Target population identified
Ø Identify target population Ø Business model selected
Ø Explore business models Ø Low cost of health information dissemination with development of client
base
mHealth Software platform
Ø Develop and adapt mHealth software platform Ø Functional mHealth software platform developed
Ø Ease of use
Ø Expanded platform of services (if feasible)
Develop appropriate localized content:

Branding and developing a subscriber base
Ø Media and communications strategy developed
Ø Brand development
Ø Program logo well recognized
Ø Brand promotion

Content development

Ø Global content development
Ø Generic, global messages developed
Ø Localization, formatting and usability testing of content
Ø Locally appropriate messages developed with high user comprehension
Ø Content development beyond MNC/initial health messages (if feasible
and satisfaction
and applicable to local context)
Ø Greater number of topics included in mHealth system (if feasible)
Ø Design of quizzes/polls (if applicable)

Ø Moderation of user-‐generated content (if applicable)

Access to health information:
Reach of target population/Access to mHealth information
Ø Program promotion Ø Improved coverage
Ø Program enrollment Ø Timely delivery of mHealth messages
Ø Delivery of mHealth messages Ø Ease of use by target population

Other support to subscribers
Ø Use of health outreach partners Ø Target population enrolled/supported by health outreach partners
Ø Monthly supervision and support provided to CHWs, women’s groups in Ø Target population supported on the field by CHW household visits/other
the community women’s groups
Increased knowledge and engagement of women/household members
Participation in the program Ø Increase in women’s comprehension of messages
Ø Active involvement in receipt of mHealth messages Ø Women’s interest in receiving information through mHealth messages
Ø Dissemination of information to non-‐subscribers Ø Interaction and active engagement with messages
Ø Informal communication among women and mothers through other
networks including recommendations

Greater knowledge Ø Women’s greater knowledge of health needs
Ø Awareness of health needs Ø Intention to adopt healthy behaviors and implement health advice
Ø Women’s increased confidence regarding health issues
Notes:
• Some of the activities included in the section on initial setup may stretch through the course of program implementation. For instance, the number of
donors could increase with greater evidence of usage and the success of the program. Similarly, the establishment of a business model is a slow
process that may develop as the partnership evolves in each country.
• Activities related to training and role of community health workers may vary by country.

Table 4: Indicators for Monitoring and Evaluation
• Greater use of preventive health and care-‐seeking behavior at the
Outcomes household level
• Women’s/Household’s increased uptake of health services
Domain Suggested Indicators Illustrative Data Source
Behavior change: Ø Percent women/households who followed Ø Subscriber interactive phone or Web surveys conducted by
Increased Uptake/Use of healthy behaviors (example – Implementing partner, compilation of information from health
Services breastfeeding, family planning, nutrition, records/vouchers sent via mobile for redemption at health clinic on a
Ø Increase in care-‐giving newborn care) quarterly basis, triangulation with clinical data if available
and care-‐seeking and Ø Percent women who utilized follow-‐up
healthy behavior services offered by the health system
(nutrition, newborn (example -‐ IPT, ANC, SBA, facility birth,
care) postnatal care, diarrhea, fever/ARI
treatment/PMTCT)
Program Activities and Outputs
Initial setup
Domain Suggested Indicators Illustrative Data Source
Coalition partnership and Ø Percent of partners retained Ø Implementing partner
business model Ø Return on investment (upfront investment/operating cost)* Ø Mobile operators
Ø Viability of the Ø Average cost per user Ø Corporate advertisers
partnership Ø Average revenue per user Ø Implementing partner
Ø Monthly maintenance costs Ø Subscriber interactive phone or
Ø Monthly income raised Web surveys conducted by
Ø Diversity of income streams implementing partner on a
Ø Average expenditure on advertising/user quarterly basis
Ø Percent contribution of global partner/each country partner/other
donors/subscribers, revenue-‐sharing with other mServices*
Ø Assessment of target population’s willingness to pay
mHealth Software
platform Ø Platform development Ø Software developers or mobile
Ø Functional mHealth Ø Interoperable, locally adapted platform operators
platform Ø Platform cost per user (including technical maintenance cost)
Ø Ease of use (based on usability assessment)
Ø Percentage of message non-‐delivery and other technical complaints
Ø Percentage of time when platform is nonoperational
Ø Link to other mServices

Develop appropriate localized content
Branding and developing Ø Recognition of logo Ø Implementing partner
a subscriber base Ø Branding partnerships developed Ø Qualitative survey

Content development Ø Percentage of messages vetted by experts and follow WHO guidelines Ø Implementing partner or Baby
Ø Percentage of messages adapted for local use and easy understanding by user by Center
MCH area/stage of pregnancy, motherhood*
Ø Percentage of messages translated to local language
Ø Number of health topics expanded beyond MNC
Women’s increased access to health information
Timely delivery of priority MCH information through mHealth applications
Reach of target Ø Percentage of target population covered Ø Mobile operator subscriber records,
population/Access to Ø Percentage of target population enrolled who are women or household member Implementing partner
mHealth information subscribers (by socio-‐economic characteristics/geographic location and
through timely delivery representation)
of priority MCH Ø Total number of subscribers
information Ø Percentage of subscribers retained (from pregnancy to childbirth)
Ø Level of coverage/ Ø Percentage of new subscribers
Penetration Ø Percentage of subscribers who opt out of the program
Ø Loyalty to program Ø Number of messages accessed or delivered per user per week
Ø Percentage of messages delivered on time
Ø For Web-‐based services—page views, monthly unique visitors, monthly repeat
visitors

Ø Number of health outreach partners
Ø Percentage of women enrolled through health outreach partners Ø Implementing partner or health
Other support to Ø Percentage of women who received support visits or information from other outreach partner records
subscribers subscribers, CHWs, HWs, or other community, women’s group members
Ø Percentage of CHWs and HWs who provided support to subscribers

Increased knowledge and engagement of women
Engagement of target Ø Percentage of messages read and conveyed by CHW, HW or household member Ø Implementing partner records
population Ø Percentage of messages recalled/understood by target population
Ø Increased motivation Ø Percentage of messages shared with women if subscriber is a household member Ø Subscriber-‐interactive phone or
to participate in the Ø Positive feedback on message clarity, relevance, and appeal Web surveys conducted by
program Ø Percentage of subscribers satisfied with the service Implementing partner on a
Ø Percentage of messages subscribers did not delete quarterly basis
Ø Women’s rating of ease of access to health messages
Ø Number of peer referrals outside the household
Ø For Web-‐based services – number of user comments, time spent logged in
Greater knowledge of
health needs and available
health services Ø Percentage of subscribers with increased knowledge of care-‐giving and care-‐ Ø Subscriber-‐interactive phone or
seeking behavior Web surveys conducted by
Ø Percentage of subscribers who intend to adopt appropriate care-‐giving and implementing partner on a
care-‐seeking behavior quarterly basis
Notes:
• See Appendix A for detailed performance indicator reference sheets for the highlighted core indicators (in bold italics). Additional key indicators
related to initial setup and content development are highlighted in the table with an asterisk.
• Some of the indicators listed in the above table especially during the initial stages of the program relating to the establishment of the partnership are
Yes/No indicators that may not need to be compiled as the program progresses in countries. Furthermore, a study on the nature of the partnership
(breadth of partners, channels of communication, division of roles and responsibilities) and sustainability of the business model is more appropriate as
an Operations Research study rather than routine monitoring.
• Although some of the indicators refer to women, they could be easily applied to non-‐deciding household members and other household decision-‐
makers who have access to phones in the household.


Long-‐Term Sustainability and Program Expansion

As mentioned earlier, the primary purpose of this document is to review the MAMA program in
countries on a regular basis to measure performance. It also provides some guidance on evaluation
studies conducted to examine the effects of the program. This work will build the evidence base for the
association between mHealth messaging and greater access to health information resulting in improved
care-‐giving and care-‐seeking behavior among women. By providing critical information on the success of
the program in terms of reaching beneficiaries and improving their health behavior, results from routine
monitoring and evaluation studies have a key role in influencing long-‐term sustainability of the MAMA
program in various ways:
• Providing incentives to expand the program in-‐country or in new areas thus providing
economies of scale
• Increasing government support to the partnership
• Possibly encouraging increased provision of health services based on an increase in demand
• Influencing the IT policy in the country regarding mobile services
• Increasing the role of local communities including other NGOs in mHealth messaging
• Improving government policy environment in countries
Expanding the program nationally will also provide economies of scale with implications for the long
term sustainability of the program.
Evaluation Studies
Bellagio eHealth Evaluation Principles
Each of these principles is essential in the design, deployment, and application of eHealth evaluation.
1. Core principles underlie the structure, content, and delivery of an eHealth system independent of
the rapidly changing technology used.
2. High quality data collection, communication, and use are central to the benefits of eHealth systems.
3. Evaluating eHealth both demonstrates its impact and fosters a culture that values evidence and
uses it to inform improvements in eHealth deployments.
4. To ensure the greatest benefit from eHealth and enhance sustainability and scale, eHealth
evaluations should recognize and address the needs of all key constituencies.
5. Evidence is needed to demonstrate costs and benefits of eHealth implementations, and maximize
eHealth’s beneficial impact on health system performance and population health.
6. The value of a complete evaluation program is enhanced through research that is attuned to the
differing requirements throughout the life-‐course of the project, whether at needs assessment,
pilot-‐, facility level-‐, regional and national scale-‐up stages.
7. Independent and objective outcome-‐focused evaluation represents the ideal of impact evaluation.
8. Country alignment and commitment to a clear eHealth vision, plan, and evaluation strategy is
essential.
9. Improving the eHealth evidence base requires m ore than increased numbers of studies but also
improved quality of eHealth research studies.
The Bellagio eHealth Evaluation Group. 2011. Call to Action on Global eHealth Evaluation. Consensus
Statement of the WHO Global eHealth Evaluation Meeting, Bellagio, September

These principles are very applicable in the mHealth context, especially for pilot programs. Performance
indicators typically indicate progress and cannot be used to determine why a certain result occurs.
Therefore, there is a need to perform a thorough evaluation of the MAMA program in countries to
determine its effectiveness in improving health outcomes for pregnant women and their babies. These

outcomes include women’s behavior in relation to family planning, nutrition and self-‐care and care-‐
giving for newborns or routine care-‐seeking behavior such as antenatal, postnatal care or vaccinations
for pregnant women or infants; or care due to sickness.
Subscriber databases and information from mobile network operators easily provide information on the
number and characteristics of subscribers and the type of messages delivered. Other methods such as
interactive phone quizzes or small surveys of a sample of the target population need to be conducted by
the in-‐country IP to gather information on behavior change by women (see Table 4).
As part of the performance monitoring process, the in-‐country IP will collect such data regularly on a
small sample of women. However, as part of the program evaluation, a comparison of health behavior
patterns of women who are in MAMA-‐subscribing and non-‐subscribing households is also needed.
Other studies can be undertaken to better understand the implementation process, the successes and
challenges of the program, the extent to which women are engaged through the use of mHealth
messaging and how much of the information provided through MAMA is read, understood or used by
women. Qualitative methods can provide more detailed information on these topics. Furthermore,
qualitative data also provide the context to better understand numbers on various indicators obtained
through routine quantitative data collection.
The proposed evaluation studies cover the following areas:
• Impact evaluation using a pre-‐post study or quasi-‐experimental design to evaluate the impact of
mHealth messaging on women’s health-‐related behavior
• Formative evaluation studies to plan the implementation of the program, such as identifying the
best methods to reach the entire target population including other household members who act
as gatekeepers in each context.26
• Process evaluation to better understand the implementation of the program and provide
feedback during the course of the program
Some specific ideas and guidelines for analytical studies with information on data collection, including
qualitative methods are listed below. These guidelines can be used by countries to gather information
on the success of their programs as well as information on lessons learned that could be useful for
program expansion. Specific topics of interest may vary to some extent across countries based on the
nature of their program and may be developed specifically for each country to be included in the
country-‐specific M&E plan.
Impact Evaluation
An impact evaluation will estimate the effect of MAMA program activities on women’s behavioral health
outcomes, including breastfeeding, newborn care, nutrition, family planning, vaccination, treatment for
diarrhea, fever or ARI, adherence to recommended antenatal and postnatal care regimes, and skilled
birth attendants or facility birth. Specifically, such an evaluation will be designed to answer the following
research questions:
1. Does the MAMA program have a positive impact on a mother’s antenatal and postnatal care
regimens?

26
Many studies on healthcare decision-‐making have been conducted across various populations over the past four
decades. These could be reviewed in addition to consulting with other implementing organizations.

2. For those women who adhere to the MAMA recommendations, does it have a positive impact
on their health and the health of their child?
3. What aspects of the messaging are most effective in prompting women who are enrolled to take
action?
In order to answer these research questions, appropriate evaluation designs will need to be developed
for each country context in which the program is being rolled out. Below, we describe a general
approach to impact evaluation design that lays out a number of feasible approaches to measuring the
effectiveness of the MAMA program.
Designing an Impact Evaluation

The central problem in any impact evaluation is to establish a “counterfactual,” or a belief about what
would have happened to the program’s beneficiaries in the absence of the program intervention. An
impact evaluation is about identifying the difference in outcomes for those like-‐individuals who
participate in a particular program, compared to those who don’t. An appropriate counterfactual
revolves around identifying a control group that is as comparable as possible to the intervention group
except that the intervention group receives program services and the control group does not, so that
outcomes may be compared across both groups. This allows permits ruling out any other potential
influences over participant outcomes, so that any difference in outcome can be attributed to the
intervention. In order to make strong inferences about whether and how the MAMA program improves
women’s health-‐related behaviors and outcomes, it is necessary to find a way to compare MAMAs
subscribers to people who do not receive the service but are comparable in every other way (i.e., age,
socio-‐economic status, geographic location, education level, motivation, etc.).
This is difficult because women who subscribe to MAMA are a self-‐selected group. That is, MAMA
participants are not recruited. The program uses advertising to raise awareness and create demand for
the service, but signing up for it is voluntary. This means that subscribers may differ from the general
population in at least two ways:
• Having mobile phones

• Being proactive enough to sign up for the service and possibly more likely than other people to
take better care of their own MCH needs.
Because of these factors, any comparison of subscribers to the general population would likely
overestimate the program’s efficacy. Study-‐design options to mitigate them, including Randomized
Control Trial (RCT) and Quasi-‐Experimental Design (QED) methods are presented below.
RCT Design Option #1: Subscriber versus Non-‐subscriber

An appropriate methodology would be to randomize a known “target” demographic of the appropriate
sample size, most likely using a cluster-‐randomized design as a result of which a selected group will
receive mHealth information through the messaging service while others not receiving services are
assigned to a control group.27 The program would then not be accessible to anyone except those
“randomized” to receive the intervention.

27
See Labrique, AB, RW Klemm, M Rashid, AA Shamim, A Massie, J Katz, P Christian, and KP West Jr. 2011. A
cluster-‐randomized maternal vitamin A or beta-‐carotene supplementation trial in Bangladesh: Design and
methods. Trials, 12:102.

Before this selection process, a baseline demographic survey will need to be administered to collect
baseline data about the health status of the women enrolling. This will help measure whether there is
balance in the randomization process. Both groups of participants will then be followed throughout the
course of the program at specific intervals to see whether or not there are differences in their health
outcomes and care-‐seeking behavior. Additional information regarding period and frequency of data
collection and data analysis types are presented below.
When the sampling frame is well constructed, this design shows definitively whether or not the MAMA
program has an impact on maternal health outcomes and antenatal and postnatal care. This model,
however, would not identify particular content in the messaging that influenced women’s behavior and
health outcomes. To overcome these challenges, a second design is presented below that will provide
more insight into how the impact of messaging without withholding messaging from any women.
RCT Design Option #2: Treatment A vs. Treatment B

The MAMA program is based on the premise that timely reminders and health information delivered via
text message will help women remember and stay motivated to seek appropriate health services as
their needs arise, as well as providing them with information regarding their babies’ development.
Because the messages play this dual role, it is unclear exactly which aspects of the intervention influence
women to seek care. Is it specific content of the messages or the fact that participants receive periodic
reminders? Would any of these interventions be effective on their own, or must they be delivered in
combination?
This second design option provides a way to examine which aspects of the messaging may impact the
behavior of the women who receive them. In this approach, rather than randomly assigning applicants
into treatment/no-‐treatment categories as described above, individuals are assigned into two treatment
groups. In one of the groups, subscribers receive phone messages that provide health information about
their babies’ development and targeted messages at specific points in the pregnancy to seek specific
medical care based on their stage of pregnancy or motherhood. In the second group, the subscribers
receive less targeted messaging, at regular intervals either reminding them to stay on top of their care
or providing basic information about their babies’ development.
The hypothesis is that the women who receive the more detailed messages will better understand and
more actively pursue their own medical care, and that these women and their babies, will have better
health outcomes. However, if it is found that the more basic form of messaging has an equivalent
impact, then this has implications for the investment. For example, this may imply that it is only or
primarily increased awareness that inspires women to seek medical care rather than the actual content
of messaging. If this is the case, a much larger population of women who don’t have access to mobile
phones may be reached through other media.
The main disadvantage of this design is that it does not allow for the comparison between those who
receive messages and those who receive none. This technique could certainly be combined with Option
#1. However, this would again mean withholding messaging from a single group of women. An
extension of this approach would be to have an additional “control” group”, who do not have access to
either kind of messaging to examine the effect of the lack of messaging on women’s health information-‐
and care-‐seeking behavior.
Other Design Options

A QED is another alternative, though it may be prone to capturing the effect of secular improvements in
population health. Any research design that departs from a randomized trial and instead uses statistical

adjustment to reduce selection bias falls into the category of QED. In order to make it work, a large
study sample with sufficient data on factors likely to influence the study outcomes other than exposure
to the program intervention is needed, so that treatment and control comparisons can be made
statistically. The various models available for this kind of research include propensity score matching
and difference-‐in-‐difference estimation. These approaches can overcome sample selection bias, but
they remain vulnerable to omitted variable bias and are particularly prone to problems due to model
specification error. This approach is therefore more difficult, and it can require large samples.
Another challenge of QED is locating individuals who otherwise may be challenging to locate, such as
those in the control group who may not have mobile phones. If some evaluation study participants
cannot be located for follow-‐up, the remaining sample may be different in fundamentally important
ways from the initial study cohort. This will lead to errors in our estimates of program effects. Ideally,
attrition problems will be manageable because we will be able to maintain contact with study
participants through their cell phones. But this requires that both the treatment and control groups
subscribe to a cell phone service.
A stepped-‐wedge randomized trial design is a variation of the RCT design where the timing of receiving
the intervention is randomized across clusters, so that at the end of the study, an equal amount of
person-‐time is accrued in the treatment as well as in the control group. In such a design, the
intervention is sequentially rolled out to participants who could be either individuals or clusters over a
number of time periods.28Each randomization unit first receives the control condition and then crosses
over to receive the intervention.29 It is important to note that the order in which beneficiaries receive
the intervention is random. The stepped wedge design is particularly useful when it is not feasible to
provide the intervention to everyone or every community at once, and for evaluating the effectiveness
of interventions that have proven efficacious in a more limited research setting and are now being
scaled up to the community level. This design is also useful for evaluating temporal changes in the
intervention effect.
This method uses beneficiaries already chosen to participate in a project at a later stage as the
comparison group. The assumption is that because they have been selected to receive the intervention
in the future they are similar to the treatment group, and therefore comparable in terms of outcome
variables of interest. For example, in the case of MAMA implementation, a modified version of this
approach where the implementation of the program/implementation is staggered could be applied. In
such a case the control group could be selected from a district or region where the MAMA program has
not been launched but will be launched in the near future. Comparisons could potentially be made
between those who were not subscribers and subscribers when the program is launched. However, in
practice, this approach could be problematic mainly stemming from the fact that the underlying
assumptions of population comparability are violated. In such an approach, it cannot be guaranteed that
treatment and comparison groups are comparable and some method of matching will need to be
applied to verify comparability.

28
Brown, CA, and RJ Lilford. 2006. The stepped wedge trial design: A systematic review. BMC Medical Research
Methodology, 6:54.
29
Hughes, JP. 2008. Stepped Wedge Design. Wiley Encyclopedia of Clinical Trials. 1–8.

Data Collection
Sample Definition and Power Analysis
The first stage of the sample design is to define the eligibility criteria for participation in the program.
For example, the MAMA program presumably targets women of reproductive age of a certain education
and income level who live in specific geographic areas of the target countries. These criteria should be
made as clear as possible prior to the start of the study, and the same criteria should be applied to both
the treatment and the control group. In the context of an RCT, sample selection is done after initial
screening, which ensures that this comparability will be achieved. Any characteristics likely to affect
outcomes should be used to “block” sample selection in order to achieve balance across those factors
during study enrollment. This means that selection into the treatment or control group is done within
blocks.
In a QED, sample selection for the control and treatment groups may occur separately, which is why it is
important to articulate the screening criteria up front. Furthermore, some mechanism needs to be
developed for identifying and recruiting women for the treatment group, since they won’t be captured
when signing up for the MAMA’s service. This would likely involve a screening survey and substantially
increase the cost of the study.
An important consideration for any evaluation study is sample size. A power analysis should be done in
order to estimate the sample size needed to detect program effects with a reasonable level of certainty.
The power analysis can take account of various kinds of outcome indicators (such as continuous, binary,
and count measures) and the consequent statistical requirements for rejecting the null hypothesis
assuming small, moderate or large effect sizes, as appropriate. Standard power calculations can be done
using widely available software, and this will be done for each country in which an impact evaluation will
take place. The longitudinal nature of the measures being proposed may mean that larger samples are
needed in order to accommodate more independent and dependent variables, and to account for
anticipated attrition from the study cohort. Power calculations for QED models can be more difficult,
since the minimal acceptable statistical analysis is usually a regression-‐based model with a number of
predictor variables.
Timing
Regardless of the design that is selected, a critically important element of an impact evaluation is the
timing of follow-‐up data collection. Because many of the outcomes of interest take many months or
longer to manifest—for example, birth outcomes of pregnant women won’t be known until months
after enrollment in a study—it is important to be able to conduct ongoing data collection at appropriate
times. Therefore what needs to be determined is when data will be collected for the first time
(baseline), and the periodicity of follow-‐up data collection. Also, given that women will be at various
stages of pregnancy when they enroll it needs to be determined, how enrollment in the study will occur.
One of the simplest approaches is to set up an enrollment period, during which women will be enrolled
in the study. Using this approach, each site will identify the number of individuals included in the
sampling frame (both subscribers and the appropriate comparison group) and continue to enroll
individuals into the study until that sample size (N) has been met. This sets up a cohort design, in which
the individuals in that sample will be followed through the duration of the study. In this design, the
women included in the study will be at various stages in their pregnancies. This makes it likely that a
larger number of study participants will be needed in order for all of the stages of pregnancy to be well
represented (i.e., an equal number of women in their first, second and third trimesters) in both the
treatment and control groups. However, if targeting women who are all in the same stage of pregnancy,

then it may take longer to reach an appropriate N, which would increase the cost of baseline data
collection.
This is an important tradeoff that should be considered at the start of study design.
Baseline Data Collection

Baseline data collection will gather information on demographic and socio-‐economic characteristics of
women/households as well as basic information on health outcomes based on their stage of pregnancy
or newborn care. If opting to use an RCT design, then it is important that the baseline information is
collected prior to randomizing women into either the treatment or control group. The purpose of the
baseline data collection is twofold. First, it is to assess whether or not the women in the treatment and
control group are comparable. Second, it will provide insight into their health, behavioral patterns, and
other demographic information prior to enrolling.
Follow-‐up Data Collection

Follow-‐up data collection will be conducted every 3-‐6 months of the program for both groups. Each site
can decide if the same frequency of data collection should occur after the women have given birth. In
the initial stages of MAMA program implementation, it may be useful to have a follow-‐up at 3 months
especially to a selected sample which can then be increased to a 6-‐ or 12-‐month follow-‐up. More
frequent data collection in the initial stages has the advantage of determining whether changes to the
program need to be made. This is particularly important if the same sample is used to obtain
information on some of the questions related to formative research that are proposed in the next
section.
Follow-‐up data collection instruments will include data collection on all of the outcomes of interest so
that will capture the information needed to assess their health-‐related behaviors and outcomes
regardless of each participant’s stage of pregnancy or child rearing. It is possible that some women who
are not subscribers may also get information on MAMA from other sources. Additional questions could
also be included in the data collection instrument to get information to understand the avenues through
which such transfer of information may be taking place and how it affects their behavior.
Data Analysis: Demonstrating Impact

Analyses will use dependent variables that are designed to reflect the appropriate level of measurement
for each outcome of interest. There may be essentially binary events, such as birth in a clinic. Other
events, such as number of antenatal clinic visits and number of birth complications, may be measured
with count variables. A number of regression-‐based models are available to examine these outcomes,
such as logistic regression and Poisson regression, respectively. These models can also be used in the
context of a multilevel modeling framework when necessary to account for clustering of observations.
Based on the nature of data collection, an event-‐history based cohort analysis is also possible to
demonstrate changes in behavior as women progress through stages of their pregnancy and
motherhood.
In addition to examining treatment versus control status, all analyses will control for the effects of
women’s socio-‐economic characteristics such as education, economic status, urban/rural residence and
other socio-‐economic characteristics, and baseline health status on outcomes at the various stages of
follow-‐up.

What does it mean if there is no evidence of any impact?
Some impact evaluations, regardless of how well they are designed, are unable to show any program
impact. This can happen for purely statistical reasons. For example, it is typical to set statistical power at
90 percent when designing a sample. This means that the study has a 90 percent chance of correctly
identifying a program effect of a given size. Conversely, it also implies there is a 10 percent chance the
study will miss such an effect due to random sample variability. Statistical power has to be set so that
the risk of such an error is balanced against the costs of doing a bigger study, and finding the right
balance is an inherently subjective endeavor.
An alternative possibility, if a study shows no impact, is that the comparison is flawed. This is the most
critical issue surrounding sample selection bias, and it is often underappreciated. “Bias” means that the
estimate is inaccurate, but it is hard to know whether it is too high or too low. There are many
circumstances in which a biased sample leads to the incorrect conclusion that the program had no
effect. This would happen, for example, if the control group were set up to include a disproportionally
high number of women who are knowledgeable and empowered to take good care of their prenatal
health, which would put the treatment group at a disadvantage before the program even starts.
Minimizing sample selection bias is the main reason for doing RCTs, but even a well-‐designed RCT can
break down if there is differential attrition between the treatment and the control groups.
Finally, a study could fail to show impact because there was no impact if the program did not perform as
well as expected. This is a possibility that should be entertained, especially if the study design is sound.
Note, however, that all of these issues also apply if a study does identify an impact. The “impact” may
actually be due to sampling error or bias. Thus, an evaluation is always an imperfect look at program
effectiveness, and it should be considered only as one piece of information to be interpreted in a larger
context. As more high-‐quality evaluations are done, the picture of program effectiveness will become
clearer. Therefore, examining results of the impact evaluation in conjunction with process and output
indicators obtained through routine monitoring is necessary; both monitoring and impact evaluation
processes need to take place side by side because program implementation could have an impact on the
impact of the program.
Other Analytical Studies, including Formative and Process Evaluation

In addition to determining impact, it is equally important to understand the implementation process of
the MAMA program in countries, their successes and challenges to bring about improvements in quality
of the program and learning for program improvement and expansion. Therefore, it is critical that the
impact evaluation and formative research are systematically aligned. Some examples of other analytical
studies in addition to the impact evaluation described above are proposed. While these studies can be
conducted in isolation, it may be more efficient to include some relevant questions from these studies in
the data collection instruments proposed for the impact evaluation. Since the impact evaluation has a
design involving regular follow-‐up, these modules could be included during data collection at the
appropriate time.
Some possible research questions and analytical strategies are presented below.
Research questions addressed in such analysis could gather information about—
• Woman or household’s motivation to join and stay in the program
• Ease of accessing messages

• Comprehension of information

• Assistance provided by CHWs
• Improved knowledge of preventive health information
• Ability to access health services
• Any constraints in using the health system as prescribed by the mHealth messages through
MAMA
Research question: Since many women may not directly use mobile phones and may rely on
information from husbands and relatives who are subscribers to the MAMA program, such interviews
will also provide useful data about whether they receive appropriate and timely information on healthy
and care-‐seeking behaviors as expected.
• Data collection methods: In-‐depth interviews and focus group discussions of women subscribers
and household members. An attitudinal/motivational survey can also be used to gather
information on women’s/households motivation to stay in the program. This component can
also be built into the impact evaluation proposed above.
• Analysis: Analysis of these qualitative data could be examined taking into account the socio-‐
economic characteristics of interviewed women to see if any specific group in particular faces
constraints.
Research question: With the possibility of information transfer through peers, community groups,
women’s groups or community health workers, what is the extent and nature of information transfer
through informal social networks?
• Data collection methods: Quantitative data collection can be built into the impact evaluation to
determine the process of information transfer among non-‐subscribers. The 6 month data
collection follow-‐up proposed in the impact evaluation can pick up any women who obtained
health information through MAMA subscribers at various timepoints beyond baseline data
collection.
• Analysis: In addition to quantitative analysis of these data, other qualitative analysis can be
conducted on interviews, and focus groups of sampled women.
Research question: What are other possible business models to be considered for program expansion?
What is the target population’s willingness to pay for services received?
• Data collection methods: Interviews of women (including current subscribers and non-‐
subscribers) can provide community level information on whether they would remain
subscribers if subscriber fees were increased, how much women are willing to pay for this
service, other barriers to their participation, and what they were looking for in such a program.
Although women who are MAMA subscribers and non-‐subscribers can be administered a
questionnaire for a quantitative analysis to determine women’s willingness to pay, focus group
discussions with other outreach partners can also provide information on ways in which the
community can be engaged in order to enhance the reach and benefits of the program.
• Analysis: In addition to quantitative analysis, qualitative analysis of interview responses will
provide key information for program improvement and expansion.

III. RESOURCES AND TIMELINE

A. M&E RESOURCES AND ORGANIZATION
Global M&E framework in relation to the country M&E plan: The Global M&E framework provides
guidelines and a broad set of indicators to monitor MAMA program activities. These indicators are
expected to be generally applicable across different country contexts. The main implementing
organization in each country will develop a country specific M&E plan using this broad framework with
assistance from the MAMA global partnership as needed. Other assistance including training on data
collection, ensuring data quality and analysis and reporting for program improvement will also be
provided. The country M&E plans will be more detailed including more specific indicators and studies
specific to the country context.
M&E activities: In each country, the routine M&E activities are listed in Table 5 and are the
responsibility of the MAMA in-‐country IP and its M&E team, except for the impact evaluation which falls
under the scope of the external evaluation partner in each country. To perform these activities, an M&E
system needs be developed in each country to ensure data collection and reporting of results on a
timely basis.
Table 5: Overview of M&E Activities in Country

Type of M&E Activity Timing Group Performing Activity
Collecting performance data Continuous, compile monthly In-‐country IP with information from
mobile operator records and other
partner records
Reviewing performance data Monthly meeting for routine data In-‐country IP and other partners
collection, quarterly for data from
small surveys of behavior change
Using data for program Continuous, after monthly program In-‐country IP and other partners
management meeting
Reporting data results Monthly, quarterly, annual report In-‐country IP
Assessing data quality In-‐country IP
Reviewing and updating the M&E Quarterly In-‐country IP, after discussion with
plan global MAMA partnership
Formative evaluation In the first quarter after In-‐country IP
implementation of country program
Impact evaluation and other Baseline in the first quarter after External evaluation partner, after
analytical studies implementation of country discussion with in-‐country MAMA
program, followed by 6 month partner and global MAMA
follow up data collection partnership
In-‐depth evaluation and operations research studies will be conducted by an external organization to
assess the impact of MAMA program on healthy and care-‐seeking behavior of women. Specific
methodologies for evaluating these activities will be determined once the activities have been designed,
but both quantitative and qualitative methods will be used as appropriate to triangulate and strengthen
the findings. Findings from such evaluations will show that the project has been successful in creating
effective mechanisms for transfer of information to improve the health of women in low-‐resource
settings.
In the early stages of the program, the IP will also conduct small-‐scale formative evaluations in pilot
areas to test various aspects of the program and make modifications as needed. However, conducting

evaluations and special in-‐depth studies will be the responsibility of the evaluation partner. In the long-‐
term, results of these evaluation studies will be critical to understand the impact of the program on
women’s health behavior, and the various successes and challenges in program implementation, thus
providing relevant information for program expansion and to make changes to the program if needed.
Other operations research studies will be conducted by other organizations as needed.
B. DATA SOURCES AND DATA COLLECTION TOOLS

Timely reporting of reliable information is critical for effective monitoring. Data collection on M&E
indicators will primarily be the responsibility of the MAMA IP in each country. Data will be collected on a
routine basis from multiple sources such as coverage and message delivery records from mobile
operators and other partners.
In some countries information will also be disseminated through Web sites, Web statistics there could
be monitored on a regular basis. Information on data sources for each indicator is presented in Table 4.
Specific details by indicator for the core indicators for cross-‐national comparison are presented in the
indicator sheets in Appendix A.
The output/immediate outcome indicators (implementation of project indicators) will be collected and
monitored on a regular basis throughout the project using the above-‐mentioned sources. However,
information on outcome indicators including participant responses will need to be compiled by the IP
using external sources such as periodic surveys of a small sample group of women/household member
subscribers. At a later stage of the program when it has sufficiently expanded, health service utilization
data may also be useful. Depending on the country context, other methods such as web surveys or
interactive phone quizzes or use of coupons for health services may also be methods for routine data
collection to assess the utility of mHealth information and use of health services. These data collection
methods can be conducted on small samples on a quarterly basis.
C. PORTFOLIO REVIEW
The MAMA country IP and M&E staff will monitor and analyze performance data during the course of
the year in order to determine whether targets are being met and assess whether there are
performance gaps. The team will meet on a monthly basis to discuss and review progress based on the
monthly report. Depending on the results of these reviews, the team may need to adjust programming
and activities.
Quarterly meetings will also be held to examine the results of the data collection of behavioral
outcomes based on small sample surveys and other similar methods. A more comprehensive meeting
involving the MAMA global partners to review performance will be conducted annually to share
information on program activities and progress during the year. This annual performance review will
provide the opportunity to examine the implementation of activities, the completion of milestones, and
the achievement of performance results. Small studies and site visits by the IP will also occur over the
life of the program to monitor progress in the field.
D. PLAN FOR DATA ANALYSIS, REVIEW, REPORTING, AND FEEDBACK

Data analysis: In general, analysis of routine monitoring data will be conducted by the M&E team of the
in-‐country MAMA program IP based on information gathered from mobile network operations and
other program partners, and small special surveys to assess knowledge of health behavior, intent to
change behavior, and actual behavior change. If any discrepancies in the data provided by small surveys

and program statistics are found, the team will need to perform triangulation of data to better
understand the dynamics of data disparity. Analysis of data from the evaluations and special studies will
be the responsibility of the evaluation partner.
Presentation of data: Data will be presented using a variety of tools including tables, graphs, and charts
as appropriate. Key findings will be summarized in PowerPoint presentations. The data will be presented
at the monthly and quarterly meetings in-‐country and the annual meeting with MAMA global partners.
Reporting of data: A monthly report will include results of routine data collection from IP records,
mobile operators, and other partners. All M&E results will be framed to facilitate critical programmatic
decision-‐making with opportunities for feedback and reprogramming by partners and other
stakeholders. The quarterly report will contain additional information from specific sample surveys of a
sample group of women users and household members providing greater information on activities.
These reports will be distributed to all in-‐country partners. Analysis of the changes in the indicators over
the course of the project will be analyzed and reported in the annual and end-‐of-‐project (2014) reports.
Use of data: The primary use of the data will be to inform the MAMA country program and MAMA
global program on program management and need to make changes to program activities. Particularly
in light of long-‐term sustainability of the program beyond the pilot phase, the regular analysis of these
data will provide useful information on about the program, target population reached and nature of the
partnership as it evolves to incorporate new means of funding and associated activities.
The country M&E plan will serve as an essential management tool to monitor the progress of activities in
each country and ensure that activities are being implemented as planned. The results from M&E
activities, including the evaluation, will also help inform plans for program expansion in countries as well
as targeting new populations in this process.
For the global MAMA partnership, comparison of program activities and results from country programs
and results of the evaluation serve multiple purposes:
• Learning about the progress in program activities in each country and developing the evidence
base for mHealth.
• Presenting a good picture of whether the target population is being reached and the effect the
program in improving care-‐giving and care-‐seeking behavior among women.
• Making cross-‐national comparisons of country program activities and results.
• Disseminating information on country programs, activities. and results to other countries with
MAMA programs as a form of capacity building. Structured information exchange through
various media as well as documentation of best practices and challenges will be useful to
programs in other countries
• Reporting results by MAMA global partners to the partner organizations
• External dissemination and communication of program results to the general public
E. ASSESSING DATA QUALITY

Known data limitations and significance: Indicator-‐specific data limitations will be noted in the
performance indicator reference sheets as applicable, and actions taken or planned to address these
limitations will be outlined.

Data quality assessment procedures: The in-‐country IP M&E team will review program results at the
monthly meetings. Random checks of data from mobile operators and other partners will also be
conducted. These will be examined with the data on behavior change from the small surveys which will
be conducted on a quarterly basis. If unusual patterns are observed, the data will be triangulated on a
quarterly basis to understand reasons for data disparity. These could include the use of different tools
such as data checklists, site visits and interviews. A data quality checklist of items measuring validity,
reliability, timeliness, and precision will also be developed and implemented. Any known limitations
regarding the quality of the data will be noted. Based on these findings, changes to the indicator or
method of collection will be made. If data limitations have a strong influence on data quality, a decision
may need to be made not to review the particular indicator and alternate data sources or indicators may
need to be developed.
F. REVIEWING AND UPDATING THE M&E FRAMEWORK

The global M&E framework is a living document; MAMA partners will update it as more information is
obtained on country program activities. At the country level as well, updates to the plan will also be
needed. Program activities will be reviewed in country at quarterly meetings and updates will be made
based on decisions at that time after consultations with the MAMA global partnership. At the global
level, the M&E framework will be reviewed and updated at the annual meetings. Updates will typically
be based on any changes in strategy and activities in countries.
In conducting this review and update, the key questions that need to be answered are:
• Are the performance indicators measuring the intended result?

• Are the performance indicators providing the information needed?
• How can the M&E framework be improved?
If the MAMA country team makes significant changes to the country M&E plan regarding indicators or
data sources then the rationale for adjustments will need to be documented. For changes in minor M&E
plan elements, such as indicator definition or responsible individual, the plan can be updated to reflect
the changes, but without the rationale.

APPENDIX A:
SAMPLE PERFORMANCE INDICATOR REFERENCE SHEETS

Healthy Behavior: Family planning

MAMA Domain:
Behavior Change: Increase in care-‐giving and care-‐seeking and healthy behavior at the household level
Indicator:
Percentage of women subscribers that are currently using any modern method of family planning
Suggested Questions:
Are you currently doing something or using any method to delay or avoid getting pregnant?
Which method are you using?
Date Established:
When did relevant parties agree on the reporting of this indicator?
Date Last Reviewed:
When did relevant parties last review/discuss/alter the indicator?
A. DESCRIPTION
Precise Definition & Method of Calculation:
Numerator: Number of women subscribers (new mothers) who are currently using any modern method of
family planning
Denominator: Total number of women subscribers (new mothers) surveyed
Modern method of family planning: Modern family planning methods include the pill, IUD, condoms,
implants, injectables, female sterilization
This question is only asked to new mothers.
Unit of Measure:
Percentage (%)
Disaggregated by:
Urban/rural, region, socio-‐economic status
B. PLAN FOR DATA COLLECTION
Data Sources:
Subscriber interactive phone/web quizzes/survey conducted by implementing partner
Timing/Frequency of Data Collection:
3 months
Responsible Organization/Individual(s):
Implementing partner
Location of Data Storage:
With Implementing partner
C. PLAN FOR DATA ANALYSIS, REPORTING, AND REVIEW (SCHEDULE, METHODOLOGY, RESPONSIBILITY)
Data Analysis:
Comparison of data every 3 months
Presentation of Data:
Indicator values will be presented in tables and/or as a graph.
Review of Data:
Data will be reviewed every 3 months
Reporting of Data:
Every 3 months
D. DATA QUALITY ISSUES:
Known Data Limitations and Significance (if any):
(Reliability Concerns) Depends on reliability of response from respondent.
Actions Taken or Planned to Address Data Limitations:

Healthy Behaviors: Early Newborn Care

MAMA Domain:
Indicator:
Percentage of subscribers whose most recent child was dried and wrapped with a cloth or blanket
immediately after birth
Was (NAME) dried (wiped) immediately after birth before the placenta was delivered?
Was (NAME) wrapped in a warm cloth or blanket immediately after birth before the placenta was
delivered?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of women subscribers (new mothers) whose child was dried and wrapped with a
cloth or blanket immediately after birth
Unit of Measure:
Percentage (%)
Disaggregated by:
Data Sources:
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months


Healthy behaviors: Breastfeeding

MAMA Domain:
Indicator:
Percentage of women subscribers (new mothers) whose infants under the age of six months are
exclusively breastfed by their mothers
Has (NAME) ever been breastfed?
Is he/she still being breastfed?
Did (NAME) drink any other liquid yesterday, during the day or night?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of women subscribers (new mothers) whose infants under 6 months of age are
exclusively breastfed
Denominator: Total number of women subscribers (new mothers) surveyed with infants under 6 months
of age
A selected group of questions determines exclusive breastfeeding before number of infants can be
determined.
Unit of Measure:
Percentage (%)
Disaggregated by:
Data Sources:
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months
(Reliability Concerns) Depends on reliability of response from woman

Healthy behaviors: Early initiation of breastfeeding

MAMA Domain:
Indicator:
Percentage of women subscribers (new mothers) whose infants were put to the breast within one hour of
delivery
Did you ever breastfeed (NAME)?
How long after birth did you first put (NAME) to the breast?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of women subscribers (new mothers) who infants were put to the breast within 1
hour of delivery
Unit of Measure:
Percentage (%)
Disaggregated by:
Data Sources:
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months
(Reliability Concerns) Depends on reliability of response from the respondent.


Care-‐seeking behavior: Antenatal care

MAMA Domain:
Behavior Change: Increase in uptake of health services
Indicator:
Percentage of pregnant women subscribers who delivered a child and had at least one/four antenatal
care visits for the pregnancy
During your pregnancy with (NAME), did you see anyone for antenatal care?
If Yes:
Whom did you see? Anyone else?
During your pregnancy with (NAME), where did you see receive antenatal care?
During your pregnancy with (NAME), how many times did you receive antenatal care?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of pregnant women subscribers who delivered a child and had at least one/four
antenatal care visits for the pregnancy
Denominator: Total number of pregnant women subscribers surveyed who delivered a child
Unit of Measure:
Percentage (%)
Disaggregated by:
Data Sources:
Subscriber interactive phone/web quizzes/survey conducted by implementing partner every 3 months
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months


Care-‐seeking behavior: Skilled birth attendance

MAMA Domain:
Indicator:
Percentage of pregnant women subscribers who delivered a child whose birth was attended by skilled
personnel
Who assisted with the delivery of (NAME)?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of pregnant women subscribers who delivered a child whose birth was attended by
skilled personnel
Skilled personnel: A skilled attendant who possesses the knowledge and a defined set of cognitive and
practical skills that enable the individual to provide safe and effective health care during childbirth to
women and their infants in the home, health center and hospital settings. Skilled personnel include
midwives, doctors, nurses with midwifery and life-‐saving skills.
Unit of Measure:
Percentage (%)
Disaggregated by:
Data Sources:
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months

Care-‐seeking behavior: Health facility birth

MAMA Domain:
Indicator:
Percentage of pregnant women subscribers who delivered a child in a health facility
Where did the delivery of (NAME) take place?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of pregnant women subscribers who delivered a child at a health facility
Denominator: Total number of pregnant women subscribers who delivered a child
Unit of Measure:
Percentage (%)
Disaggregated by:
Data Sources:
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months


Care-‐seeking behavior: IPT for malaria

MAMA Domain:
Indicator:
Percentage of pregnant women subscribers who received Intermittent Preventive Treatment (IPT) for
malaria during the most recent pregnancy
When you were pregnant with (NAME), did you take any drugs in order to prevent you from getting
malaria?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of pregnant women subscribers who received Intermittent Preventive Treatment
(IPT) for malaria during the most recent pregnancy
Unit of Measure:
Percentage (%)
Disaggregated by:
Data Sources:
3 months
Data Analysis:
Indicator values will be presented in tables and/or as a graph
Review of Data:
Reporting of Data:
Every 3 months


Care-‐seeking behavior: Postnatal care

MAMA Domain:
Indicator:
Percentage of women subscribers whose newborns were attended during the postnatal period by a
health care provider
After (NAME) was born, did any health care provider check on (NAME’s) health?
How many hours, days or weeks after the birth of (NAME) did the first check take place?
Who checked on (NAME’s) health at that time?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of women subscribers whose newborns were attended during the postnatal period
by a health care provider
Denominator: Number of women subscribers who recently delivered a child that was a live birth
Postnatal period: Postnatal period begins one hour after the birth of the placenta and ends 6 weeks later.
Unit of Measure:
Percentage (%)
Disaggregated by:
Stage of postnatal period, urban/rural, region, socio-‐economic status
Data Sources:
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months


Care-‐seeking behavior: Treatment for diarrhea

MAMA Domain:
Indicator:
Percentage of women subscribers whose child had diarrhea in the past two weeks for whom advice or
treatment was sought from a health facility or provider
Has (NAME) had diarrhea in the last two weeks?
Did you seek advice or treatment from any source?
Where did you seek advice or treatment?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of women subscribers (new mothers) whose child had diarrhea in the previous two
weeks for whom advice or treatment was sought from a health facility or provider
Denominator: Total number of women subscribers (new mothers) whose child had diarrhea in the past
two weeks
Note: The time frame for this indicator may need to be extended if there are insufficient cases in the
sample
Unit of Measure:
Percentage (%)
Disaggregated by:
Data Sources:
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months

Care-‐seeking behavior: Treatment for ARI/fever

MAMA Domain:
Indicator:
Percentage of women subscribers whose child had symptoms of acute respiratory infection (ARI) in the
past two weeks for whom advice or treatment was sought
Has (NAME) had an illness with a cough at any time in the last two weeks?
When (NAME) had an illness with a cough, did he/she breathe faster than usual with short, rapid breaths
or have difficulty breathing?
Was the fast or difficult breathing due to a problem in the chest or to a blocked or runny nose?
Did you seek advice or treatment for the illness from any source?
Where did you seek advice or treatment?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of women subscribers (new mothers) whose child had symptoms of acute respiratory
infection (ARI) in the past two weeks for whom advice or treatment was sought
Denominator: Total number of women subscribers (new mothers) whose child had symptoms of acute
respiratory infection (ARI) in the past two weeks
Note: The time frame for this indicator may need to be extended if there are insufficient cases in the
sample
Unit of Measure:
Percentage (%)
Disaggregated by:
Data Sources:
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months




Initial Setup: Average cost per user

MAMA Domain:
Business model
Indicator:
Average cost per user
N/A
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Total financial cost ($) of in-‐country program
Denominator: Total number of subscribers of in-‐country program
Total costs: Includes amount of financial and in-‐kind resources from implementing partner, mobile
operators, corporate advertisers and other coalition partners
Unit of Measure:
Ratio
Disaggregated by:
N/A
Data Sources:
Financial records from implementing partner, mobile operator(s), corporate advertiser(s), and other
coalition partners
Annually
Responsible Organization/Individual(s)
Implementing partners is responsible for collecting data from coalition partners (mobile operators,
corporate advertisers, others)
With implementing partner
Data Analysis:
Comparison of data on an annual basis.
Review of Data:
Data will be reviewed annually.
Reporting of Data:
Annual
Reliable on accurate financial records being made available from partners. Additionally, the calculation of
total number of users takes into consideration subscribers of the program. It will not take into account the
total number of people reached by mHealth messages through the program.


mHealth Software Platform: Non-‐delivery and technical complaints

MAMA Domain:
Functional mHealth platform
Indicator:
Percentage of message non-‐delivery and other technical complaints
N/A
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of technical complaints received on delivery of mHealth messages and other reasons
Unit of Measure:
Number/count
Disaggregated by:
Type of complaint
Data Sources:
Complaint records from mobile operator(s)/Software platform/system records
Monthly
Responsible Organization/Individual(s)
Mobile operator(s)/implementing partner
Implementing partner/implementing partner system records
Data Analysis:
Comparison of data on a monthly basis.
Review of Data:
Data will be reviewed monthly
Reporting of Data:
Every month
Only takes into consideration the number of complaints that have been reported.


Increased access to information: Target population enrolled

MAMA Evaluation Question:
Women’s increased access to quality health information/reach of target population
Indicator:
Percentage of target population enrolled who are women or household member subscribers
N/A
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Total number of women or household member subscribers
Denominator: Total number of persons within the defined target population
How to calculate the target population: Estimation of women who are pregnant and/or are new mothers
with children under the age of 1 year. Socio-‐economic status of households of women may be taken into
account in this calculation. Target population calculated based on national or other estimates.
Unit of Measure:
Percentage (%)
Disaggregated by:
Urban/rural, region, level of education, socio-‐economic status, sex
Data Sources:
Mobile operator subscriber/implementing partner records. National/other estimates for target
population
Monthly
Mobile operator/implementing partner system records
Data Analysis:
Comparison of data every month
Review of Data:
Data will be reviewed every month
Reporting of Data:
Every month


Increased access to information: Total number of subscribers

MAMA Evaluation Question:
Indicator:
Total number of subscribers
N/A
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Total number of women or household member subscribers
Unit of Measure:
Number/Count
Disaggregated by:
Urban/rural, region, socio-‐economic status, pregnant women/new mothers
Data Sources:
Mobile operator subscriber/implementing partner records
Monthly
Data Analysis:
Comparison of data every month
Review of Data:
Reporting of Data:
Every month


Increased access to information: Percentage of subscribers retained

MAMA Domain:
Indicator:
Percentage of subscribers retained (from pregnancy to childbirth)
N/A
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Number of subscribers who were previously enrolled as pregnant women till birth of child
and re-‐enrolled as new mothers in the current month
Denominator: Total number of subscribers who were previously enrolled as pregnant women till birth of
child
Unit of Measure:
Percentage (%)
Disaggregated by:
Urban/rural, region
Data Sources:
Mobile operator subscriber/implementing partner records
Monthly
Mobile Operator(s)/implementing partner
Data Analysis:
Monthly comparison of data
Review of Data:
Data will be reviewed on a monthly basis
Reporting of Data:
Every 3 months


Increased access to information: New subscribers

MAMA Domain:
Women’s increased access to quality health information; level of coverage
Indicator:
Percentage of new subscribers during the past month
N/A
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Total number of new subscribers (including pregnant women and new mothers) enrolled
within the past month
Denominator: Total number of subscribers (including pregnant women and new mothers) at the end of
the past month
Unit of Measure:
Percentage (%)
Disaggregated by:
Urban/rural, region, sex, pregnant women/new mothers
Data Sources:
Mobile operator subscriber records
Monthly
Mobile Operator(s)/implementing partner
Data Analysis:
Comparison of data on a monthly basis
Review of Data:
Reporting of Data:
Monthly


Engagement of target population: Messages recalled/understood

MAMA Domain:
Engagement of target population
Indicator:
Percentage of health messages recalled by target population (subscribers) within the past month
Did you receive any health messages on your phone during the past month?
What were the messages that you received during the past month?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Total number of health messages recalled by subscribers (pregnant women and new mothers)
within the past month
Denominator: Total number of health messages that were sent out to subscribers (pregnant women and
new mothers) during the past month
To calculate the denominator:
Multiple the number of health messages sent out to subscribers during the past month by the total
number of persons surveyed.
Unit of Measure:
Percentage (%)
Disaggregated by:
Urban/rural, region, pregnant women/new mothers
Data Sources:
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months
Reliable on respondents’ ability to recall messages during the past month.


Engagement of target population: Percentage of subscribers satisfied with the service
MAMA Domain:
Indicator:
Percentage of subscribers satisfied with the service in the last month
Were you satisfied with the content of the messages received in the past month?
Were you satisfied with the timing of the messages received in the past month?
Were you satisfied with the quality of the service in the past month?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Numerator: Total number of subscribers (pregnant women and new mothers) who were surveyed and
reported that they were satisfied with the service in the past month
Denominator: Total number of subscribers (pregnant women and new mothers) that were surveyed in the
past month
Unit of Measure:
Percentage (%)
Disaggregated by:
Urban/rural, region, pregnant women/new mothers
Data Sources:
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months
Reliable on respondents’ ability to recall messages during the past month.

Engagement of target population: Peer referrals

MAMA Domain:
Indicator:
Number of peer referrals given outside of the subscriber’s household within the past 3 months
Have you received any health-‐related messages within the past 3 months on your phone?
Have you discussed any of these messages you received with another person during the past 3 months?
With who have you discussed the health-‐related messages?
Date Established:
Date Last Reviewed:
A. DESCRIPTION
Number of referrals given to a peer outside of the household by women subscribers (pregnant women
and new mothers) surveyed during the past 3 months
Definitions:
Peer referrals: Health-‐related information or advice (e.g. information on the practice of a healthy
behavior, where to access services, etc) that is provided to a peer that is outside of the respondent’s
household.
The number of peer referrals need to be understood in relation to the number of women subscribers
surveyed.
Unit of Measure:
Number/Count
Disaggregated by:
Urban/rural, region, subscribers who were pregnant women/new mothers
Data Sources:
3 months
Data Analysis:
Review of Data:
Reporting of Data:
Every 3 months


Reliable on respondents’ ability to recall messages received and shared/discussed during the past three
months.


Knowledge of health needs and available health services

MAMA Domain:
Awareness of health needs and available health services: Greater knowledge of health needs
Indicator:
Percentage of women subscribers (pregnant women and new mothers) with increased knowledge of care-‐
giving and care-‐seeking behavior
Measured through indicators such as the following:
For pregnant women:
a. Percentage of women subscribers (pregnant women) who know the benefits of attending
antenatal care services during their pregnancy
b. Percentage of women subscribers (pregnant women) who know that the safest way to deliver a
baby is with a skilled birth attendant and/or in a health facility
c. Percentage of women subscribers (pregnant women) who know to begin breastfeeding within
the first hour after the delivery of their baby

For new mothers:
a. Percentage of women subscribers (new mothers) who know that exclusively breastfeeding for six
months provides the best nutrition for the baby
b. Percentage of women subscribers (new mothers) who know that having their infant immunized
prevents the baby from getting certain diseases/illnesses

Indicators may be added or modified based on content and timing of messages in each country
Date Established:
Date Last Reviewed:
A. DESCRIPTION
For pregnant women:
a. Numerator: Number of of women subscribers (pregnant women) who know the benefits of
attending antenatal care services during their pregnancy
Denominator: Total number of women who are pregnant surveyed

b. Numerator: Number of women subscribers (pregnant women) who know that the safest way to
deliver a baby is with a skilled birth attendant and/or in a health facility

c. Numerator: Number of women subscribers (pregnant women) who know to begin breastfeeding
within the first hour after the delivery of their baby

For new mothers:
a. Numerator: Number of women subscribers (new mothers) who know that exclusively
breastfeeding for six months provides the best nutrition for the baby
Denominator: Total number of new mothers surveyed

b. Numerator: Number of women subscribers (new mothers) who know that having their infant
immunized prevents the baby from getting certain diseases/illnesses
Denominator: Total number of new mothers surveyed

Unit of Measure:

Percentage (%)
Disaggregated by:
Urban/rural, region, education level of women, socio-‐economic status
Data Sources:
3 months
Data Analysis:
Review of Data:
Indicator values will be reviewed every 3 months
Reporting of Data:
Every 3 months
(Reliability Concerns) Depends on reliability of response from respondent. Only measures intention of
behavior. Is not able to measure whether or not the respondent carriers out the actual behavior (in the
future).

Knowledge of health needs: Intent to adopt services

MAMA Domain:
Awareness of health needs and available health services
Indicator:
Percentage of subscribers who intend to adopt appropriate care-‐giving and care-‐seeking behavior
Measured through indicators such as the following:
For pregnant women:
a. Percentage of women subscribers (pregnant women) that intend to go to at least one antenatal
care visit during their pregnancy
b. Percentage of women subscribers (pregnant women) that intend to deliver in a health facility
c. Percentage of women subscribers (pregnant women) that intend to have a skilled birth attendant
assist with their delivery
d. Percentage of women subscribers (pregnant women) that intend to put their infant to their
breast within one hour after delivery of their baby
e. Percentage of women subscribers (pregnant women) that intend to have a postnatal check-‐up
for their youngest child within the first 6 weeks of birth

For mothers of infants 0-‐6 months:
a. Percentage of women subscribers (new mothers) who intend to exclusively breastfeed their
youngest child for six months
b. Percentage of women subscribers (new mothers) who intend to have their youngest child
immunized

Note: Indicators may be added or modified based on timing and content of messages in each country
Date Established:
Date Last Reviewed:
A. DESCRIPTION
For pregnant women:
a. Numerator: Number of women subscribers (pregnant women) that intend to go to at least one
antenatal care visit during their pregnancy
Denominator: Total number of women subscribers (pregnant women) surveyed

b. Numerator: Number of women subscribers (pregnant women) that intend to deliver in a health
facility

c. Numerator: Number of women subscribers (pregnant women) that intend to have a skilled birth
attendant assist with their delivery

d. Numerator: Number of women subscribers (pregnant women) that intend to put their infant to
their breast within one hour after delivery of their baby

e. Numerator: Number of women subscribers (pregnant women) that intend to have a postnatal
check-‐up for their youngest child within the first 6 weeks of birth

For new mothers:
a. Numerator: Number of women subscribers (new mothers) who intend to exclusively breastfeed
their youngest child for six months

b. Numerator: Number of women subscribers (new mothers) who intend to have their youngest
child immunized
Unit of Measure:
Percentage (%)
Disaggregated by:
Urban/rural, region, education level of women, socio-‐economic status
Data Sources:
3 months
Data Analysis:
Review of Data:
Indicator values will be reviewed every 3 months
Reporting of Data:
Every 3 months
(Reliability Concerns) Depends on reliability of response from respondent. Only measures intention of
behavior. Is not able to measure whether or not the respondent carriers out the actual behavior (in the
future).


Mama Global Meplan Final All

Uploaded by

Copyright:

Available Formats

Mama Global Meplan Final All

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Mama Global Meplan Final All

Uploaded by

Copyright:

Available Formats

Global Monitoring and Evaluation

MAMA M&E Plan June 2012 i

MAMA M&E Plan June 2012 ii

HWs Health Workers

ICTs Information and Communication Technologies

M&E Monitoring and Evaluation

MAMA Mobile Alliance for Maternal Action

MCH Maternal and Child Health

MCHIP Maternal and Child Health Integrated Program

MDG Millennium Development Goal

MNC Maternal and Newborn Care

MoTeCH Mobile Technology for Community Health

QED Quasi-­‐Experimental Design

RCT Randomized Control Trial

MAMA M&E Plan June 2012 iii

THE ROLE OF MOBILE TECHNOLOGY IN IMPROVING MCH OUTCOMES

MAMA M&E Plan June 2012 iv

EVIDENCE BASE FOR THE MHEALTH-­‐MCH RELATIONSHIP

MAMA M&E Plan June 2012 v

MAMA M&E Plan June 2012 vi

• Identify performance gaps and setting goals for their improvement.

MAMA M&E Plan June 2012 1

Table 1: Key Target Audiences of the MAMA Initiative

A. MAMA THEORY OF CHANGE MODEL

• Catalyze locally driven, sustainable country partnerships.

MAMA M&E Plan June 2012 2

MAMA M&E Plan June 2012 3

MAMA M&E Plan June 2012 5

MAMA M&E Plan June 2012 6

MAMA M&E Plan June 2012 7

MAMA M&E Plan June 2012 8

Table 2: Levels of Data Collection and Relevant Audiences

Monitoring Evaluation Operations Research

MAMA M&E Plan June 2012 9

B. KEY M&E QUESTIONS

MAMA M&E Plan June 2012 10

C. OPERATIONALIZING THE MAMA M&E FRAMEWORK

MAMA M&E Plan June 2012 11

MAMA M&E Plan June 2012 12

Outcome: Improved household preventive health and care-­‐seeking behavior

MAMA M&E Plan June 2012 13

MAMA M&E Plan June 2012 14

MAMA M&E Plan June 2012 15

MAMA M&E Plan June 2012 16

MAMA M&E Plan June 2012 17

Long-­‐Term Sustainability and Program Expansion

MAMA M&E Plan June 2012 18

The proposed evaluation studies cover the following areas:

MAMA M&E Plan June 2012 19

Designing an Impact Evaluation

• Having mobile phones

RCT Design Option #1: Subscriber versus Non-­‐subscriber

MAMA M&E Plan June 2012 20

RCT Design Option #2: Treatment A vs. Treatment B

Other Design Options

MAMA M&E Plan June 2012 21

MAMA M&E Plan June 2012 22

MAMA M&E Plan June 2012 23

Baseline Data Collection

QED Quasi-‐Experimental Design

EVIDENCE BASE FOR THE MHEALTH-‐MCH RELATIONSHIP

Outcome: Improved household preventive health and care-‐seeking behavior

Long-‐Term Sustainability and Program Expansion

RCT Design Option #1: Subscriber versus Non-‐subscriber

Follow-‐up Data Collection

Care-‐seeking behavior: Antenatal care

Care-‐seeking behavior: Skilled birth attendance

Care-‐seeking behavior: Health facility birth

Care-‐seeking behavior: IPT for malaria

Care-‐seeking behavior: Postnatal care

Care-‐seeking behavior: Treatment for diarrhea

Care-‐seeking behavior: Treatment for ARI/fever

mHealth Software Platform: Non-‐delivery and technical complaints