Assessing Market-Based Solutions: Lessons from Evaluating a

Assessing Market-Based Solutions:

Lessons from Evaluating a

Youth Employment Initiative

Karim Harji and Edward T. Jackson

Supported by

THE

Rockefeller Foundation

MONITORING & EVALUATION OFFICE

ASSESSING MARKET-BASED SOLUTIONS: LESSONS FROM EVALUATING A YOUTH EMPLOYMENT INITIATIVE2

Generating jobs for young people has become one

of society’s most important and urgent tasks. This is

particularly true in the context of developing countries.

Oicially, the number of young people out of work

worldwide has risen past 70 million and is climbing.

Unoicially, the figure is much higher. And too often,

these youth find themselves underemployed in low-wage,

precarious and dangerous work.

Governments, foundations, and others tackling youth

unemployment confront the dual challenges of creating

quality, targeted, and sustainable opportunities while, at

the same time, delivering value for the private sector.

In parallel, these stakeholders must also measure the

impact of their programs in cost-eective and meaningful

ways. Increasing understanding of how employment

translates into broader economic and social benefits and

then communicating that understanding is complex –

but necessary.

Harnessing ICT for youth

employment

Increasingly, information and communication

technologies (ICTs) provide new opportunities for young

people. The potential of mobile phones, the most visible

example, is only beginning to be explored. ICTs are giving

rise to new approaches that prepare young people for

work, helping them find the right job and strengthening

Creating employment opportunities for youth is a priority for many countries. How can these

opportunities – increasingly situated within market-based approaches to development –

generate and sustain positive employment and social outcomes for individuals, their families

and communities? This paper reports on an evaluation of a Rockefeller Foundation initiative that

provided instructive lessons on how to assess youth employment and digital jobs programs that

embed market-based principles.

ASSESSING MARKET-BASED SOLUTIONS: LESSONS FROM EVALUATING A YOUTH EMPLOYMENT INITIATIVE

Measurement: challenges and

practical application

Tensions and issues in measurement

Impact sourcing, which operates at the intersection

of market-based approaches, ICT, and workforce

development, oers a compelling example of the challenges

of complex impact measurement. The experience of

evaluating the Foundation’s digital jobs initiative has

yielded three important measurement-related insights.

1. Clarifying the intent and users of measurement.

There is often a disconnect between what should

be, and what actually is, measured. A root cause

of this is the lack of clarity around the objectives of

measurement – whether it is for accountability (to

funders or investors), learning (for the broader field),

performance management (for the enterprise), or a

combination of these. This challenge is closely related

to the users of measurement, as well as to who pays

for these activities (and, in fact, whether they are the

same). To address these challenges, measurement

strategies should be formulated with specific users

in mind and anchored in a firm understanding of their

data and information needs.

2. Distinguishing performance at individual and

firm levels. The unit of measurement and analysis in

market-based approaches typically oscillates between

their ability to progress in their careers (Raftree,

2013). However, while the use of ICT in international

development programs has shown some promise, these

initiatives have not consistently been designed with a

view toward sustainability.

New initiatives that blend ICTs and market-based

approaches take advantage of two distinct features of

the digital economy. First, a steady stream of tasks can

be carried out from virtually any place that has computer

access and a reliable internet connection. Second, many

young people are digitally literate – comfortable with

mobile phones and computers – and often determined to

access new opportunities beyond their communities. At

the intersection of these two factors lies impact sourcing,

a promising approach that combines ICT and business

process outsourcing, and provides access to employment

opportunities for people facing employment barriers (Harji

et al., 2013).

Over the last few years, The Rockefeller Foundation

– working with other funders, industry associations,

governments and networks – has supported a portfolio

of grantees to test a range of models for training,

employing, and supporting low-income youth to carry out

digital work. The Foundation funded an evaluation for

its work in digital employment and the experience from

the initial phase - which focused on impact sourcing-

provided instrumental lessons on how one might think

about assessing youth digital employment programs that

embed market-based approaches. Reflections from this

experience are summarized below.

Impact sourcing operates at the

intersection of market-based

approaches, ICT, and workforce

development.

ASSESSING MARKET-BASED SOLUTIONS: LESSONS FROM EVALUATING A YOUTH EMPLOYMENT INITIATIVE4

3. Translating individual to household outcomes.

It is especially important for market-based approaches

to demonstrate that individual results can translate

into household- and community-level outcomes over

time. Measurement helps construct an informed

narrative around how an approach contributes to

changes, whether positive or negative. This often

involves capturing baseline household conditions,

embedding techniques for longitudinal tracking for

specific sample populations, and choosing the right

design and measures within constraints – whether

they are time, budget, access to data, cultural barriers,

or privacy issues. For digital employment in particular,

new ICT-enabled tools (further explained in the next

section) are helping researchers and practitioners

collect these data more cheaply, eiciently, and reliably.

Elements of measurement in practice

As a dynamic area of inquiry, impact sourcing is providing

useful insights in addressing key challenges with regard to

measurement. Six elements have been applied in impact

sourcing initiatives that are critical to eective impact

measurement eorts.

1. Developing and interrogating a theory of change.

Theories of change are valuable tools for monitoring

and evaluation (M&E), communications, and adaptive

management. They situate M&E questions and the

indicators in the context of a broader intervention,

articulate causal relationships, and reveal assumptions

that underpin interventions. Throughout an

intervention’s lifecycle, its theory of change can be

adapted in response to emergent learning and shifts

in context (Jackson, 2013). Maintaining a theory of

change for a digital-employment or other market-based

intervention, and adjusting and improving it over time,

can support successful implementation. Theories of

change can also be used to organize and communicate

learning within and outside of an organization.

2. Employing evaluation practices that embrace

innovation and complexity. The practice of M&E

continues to evolve in response to shifting social

sector priorities and trends. Specifically, there is

demand for agile M&E approaches that adapt to the

the individual and the firm. Naturally, there is a desire

to understand and assess changes at both levels.

In practice, however, the firm as the unit of analysis

provides some helpful simplification for data collection

and analysis. For example, data on aggregate trends

across a firm for employee performance or retention can

provide relevant benchmarking within specific sectors

but, on its own, it does not explain what is happening

below the surface. Business to Rural Technologies which

is an impact sourcing service provider in rural India,

claims impressive retention statistics compared with

averages in the business process outsourcing sector.

However, deeper examination of its model suggests

that strengthening social capital among employees has

been a key contributor to these positive results.

Dynamic Measurement Systems:

An Example from Samasource

Based in the United States, Samasource connects

youth to training and employment in the digital

economy. At the time of the evaluation in 2013,

Samasource had trained and employed over 5,500

disadvantaged youth and women. Through a

technology platform, large data projects are converted

into small tasks that can be completed at local

delivery centers by youth in developing countries

with basic training. Samasource’s theory of change

is that exposure to formal work and fair pay can be

transformative.

The platform provides sophisticated analysis of

individual worker performance and collects data on

social impact factors and changes, therein linking

individual economic and social progress. A variety

of methods and approaches is used to triangulate

data, including baseline, follow-up, and exit surveys.

Measurement at Samasource fulfills multiple

objectives, including examining employee experiences

and engagement, providing task optimization,

and supporting social impact analysis for external

communication and accountability. The organization’s

evaluation practices have continued to evolve over

time in response to dynamic needs and opportunities.

Source: Chow and Harji, 2014.

ASSESSING MARKET-BASED SOLUTIONS: LESSONS FROM EVALUATING A YOUTH EMPLOYMENT INITIATIVE

and analyzed using statistical or quantitative methods

(Bamberger et al., 2010). Quantitative approaches can

include, for example, trend analysis of existing data

sets (such as earnings and sales) as well as regression

analysis, among other methods. The appropriate

balance of approaches should be suited to the purpose

and align with the budget and timeline.

5. Focusing on contribution to results. The use of

conceptual models to describe how individuals

and households progress out of poverty is still not

well understood or widely adopted. To complicate

matters further, funders often aspire to attribute

a demonstrable impact to their programs. While

it is possible, from a methodological perspective,

to develop approaches that ascertain attributable

impact, this should rarely be the sole or dominant

motivation for investing in robust M&E. If funders are

able to move beyond a fixation with attribution and

instead focus on how their resources contribute to

particular outcomes, their evaluation resources may

be channeled more eectively.

6. Using real-time, technology-enabled M&E tools.

For monitoring activities in particular, it is desirable to

link data collection, analysis, and utilization. At each of

these stages, there may be scope to integrate tools that

make the cycle easier, faster, or more cost-eective. A

range of ICT-enabled tools – including mobile phones

to conduct surveys and real-time dashboards that

dynamic conditions within which an intervention

is implemented. There is increased emphasis on

accountability to a range of civil society stakeholders.

Finally, mixed methods, and new ICT-enabled and

big data are gaining traction (UNDP, 2013). All these

conditions represent their own challenges and

opportunities, while also underscoring the importance

of flexible, iterative, and dynamic approaches to M&E.

Developmental evaluation oers one such approach

for complex, market-based programming (Westley

et al., 2010).

3. Building an integrated model that bridges the

enterprise, individual, and household levels.

In undertaking an evaluation of a digital-employment

or other market-based initiative, it is often easiest to

track indicators at the enterprise level. Data related to

employment, including job level, type of job, time on

the job, and rate of pay, are relatively straightforward

to collect. Individual demographic data, such as

gender, age, education level, and place of residence,

are similarly collected with relative ease. Less clear,

though, is the relationship among data at the individual,

job, and enterprise levels. Equally important is the

relationship between an employed young person and

his or her household. Preliminary research shows that

youth use their earnings to contribute to household

expenses. However, there is room to learn more about

whether and how this translates into real impact

at the household level. An integrated, multi-level

measurement approach is therefore required.

4. Choosing mixed research methods for data

collection and analysis. Using both qualitative and

quantitative approaches to data collection and analysis

can strengthen the quality, relevance, and use of M&E

findings. Choosing the mix of methods should begin

with strong evaluation questions, to which appropriate

methods can be matched (rather than picking the

methods first). Qualitative approaches can provide

layers of texture around the rationale, assumptions, and

changes in the theory of change, and how they evolved

during the period examined. Qualitative data collection

approaches can include participant observation,

participatory appraisals, open-ended interviews, and

focus groups – all of which can be codified, quantified,

ASSESSING MARKET-BASED SOLUTIONS: LESSONS FROM EVALUATING A YOUTH EMPLOYMENT INITIATIVE6

1. Leveraging M&E for a variety of purposes.

A broader interpretation of the purposes, uses and

users of M&E, including a broadened definition of

stakeholders should be applied. For example, the

functions of monitoring and evaluation are too often

viewed as separate and distinct from the strategic

planning processes of implementing organizations,

foundations, or development agencies. This represents

a missed opportunity, since eective M&E can generate

valuable intelligence, insights, and models that can

inform strategy.

2. Using innovative approaches and tools to

strengthen M&E. Digital employment initiatives oer

an opportunity to harness technology in creative

ways for M&E activities. A variety of new techniques

now being applied have the potential to dramatically

reshape every component of the traditional

M&E lifecycle, and even to redefine the purpose,

processes, and use of M&E itself (UNDP, 2013). It

should be clear, however, that the technologies are

enablers – they are a means to an end, rather than a

solution in their own right. Without the right people

and processes in place, no tech-enabled tool will

succeed.

aggregate and visualize multiple data streams – can

help to support this. For example, Samasource, a

pioneer in impact sourcing, uses its technology and

microwork platform to analyze worker performance

and to collect data on social impact factors and

changes (see Box 1). Using these tools also comes

with a new set of opportunities as well as risks. For

the former, the ability to reach previously unreachable

segments of youth – for example, those in rural areas

who are now accessible via mobile or social media

– is an important driver of inclusivity. For the latter,

there are many important considerations that must

be managed, such as technical capacity, infrastructure

requirements, and user security (Thakkar et al., 2013).

Implications: measuring

better futures

It is still early days for measurement of market-based

solutions, indicating the need for further development and

testing. At present, the M&E field faces three challenges

related to impact measurement for market-based

approaches.

Mixed Methods for Multiple Purposes:

An Example from Digital Data Divide

Cambodia-based Digital Divide Data (DDD) creates

jobs for disadvantaged youth through business

process outsourcing (BPO) tasks. Founded in 2001, the

organization employed more than 1,000 people across

three countries in Africa and Asia at the time of the

evaluation in 2013. As a social enterprise, DDD intentionally

recruits youth who face barriers to employment, facilitates

access to customized skills training and personal

development, provides jobs that have fair wages and

progressive career paths, and oers scholarships to

employees to attend university. Taken together, this theory

of change integrates several components to ensure that

youth are ready and able to work, prepared to build on

their experiences, and can retain work within DDD or

elsewhere.

DDD’s social impact measurement approach incorporates

several components to collect, assess, communicate,

and utilize data. Formal and informal methods are used

to collect both qualitative and quantitative data related

to performance on specific tasks and projects, as well

as broader human resourse (HR) considerations related

to developing soft/foundational skills. At the same time,

baseline in-person assessments and follow-on surveys

assess the social and economic conditions for individuals

and their households, as well as how they change over time.

These data points are used for a variety of purposes, includi

human resource management, task/role optimization, and

social impact assessment and communication.

For more information, see digitaldividedata.org/impact.

ASSESSING MARKET-BASED SOLUTIONS: LESSONS FROM EVALUATING A YOUTH EMPLOYMENT INITIATIVE

The contents of this report are the views of the authors and do not

necessarily reflect the views or policies of The Rockefeller Foundation.

This work is licensed under a Creative Commons

Attribution-NonCommercial 4.0 License

3. Placing the household closer to the center of

poverty reduction eorts. Youths’ livelihoods are

intimately tied to the households in which they live

(James-Wilson, 2013). The household is considered

critical to poverty reduction eorts, as it is an

important entry point for analyzing the status of, and

changes in, individuals within it, as well as the nature of

interaction among them over time (Burns and Keswell,

2006). Making the link from individual to household

outcomes is especially important for the legitimacy

of digital employment. An approach that enables

an understanding of how households move out of

poverty would dierentiate the type of interventions

appropriate for them, and would integrate the unique

needs and challenges of families as they seek to build

assets and livelihoods (POWG, 2011).

Looking ahead

Innovative market-based solutions hold much promise for

those committed to promoting youth employment. The

new and growing body of work in this area has necessitated

that M&E approaches adapt and evolve. The measurement

community now finds itself confronted with the important

tasks of supporting programmatic improvement, holding

stakeholders accountable for their impact on youth, and

demonstrating the value proposition for market-based

players.

Drawn from the experience of evaluating The Rockefeller

Foundation’s market-based youth employment program,

this paper has highlighted some of the most salient

factors that M&E practitioners ought to consider when

developing a measurement approach for impact sourcing

programs. However, as youth employment programs and

market-based approaches continue to evolve, so too must

this conversation.

About E.T. Jackson & Associates

E.T. Jackson and Associates Ltd. is an

international management consulting firm

providing professional services in strategic

planning, organizational learning and performance

assessment to grant-makers and investors

in the public interest. With a track record of

award-winning work in Africa and Asia, the firm

specializes in impact investing, microfinance,

social enterprise, civil-society organizations,

gender equality, local governance, and basic and

higher education.

About The Rockefeller Foundation

For more than 100 years, The Rockefeller

Foundation’s mission has been to promote the

well-being of humanity throughout the world.

Today, The Rockefeller Foundation pursues this

mission through dual goals: advancing inclusive

economies that expand opportunities for more

broadly shared prosperity, and building resilience

by helping people, communities, and institutions

prepare for, withstand and emerge stronger from

acute shocks and chronic stresses.

Monitoring and Evaluation at

The Rockefeller Foundation

Committed to supporting learning, accountability

and performance improvements, the Foundation’s

Monitoring and Evaluation Team works with sta,

grantees and partners to monitor and evaluate

the Foundation’s pathways to impact in the short-

and long-term, and to capture lessons about what

works and what doesn’t across the Foundation’s

diverse portfolio.

www.rockefellerfoundation.org

References

Bamberger, M., Rao, V. & Woolcock, M. 2010. Using Mixed

Methods in Monitoring and Evaluation: Experiences from

International Development. The World Bank – Development

Research Group – Poverty and Inequality Team. Available at

elibrary.worldbank.org/doi/pdf/10.1596/1813-9450-5245.

Burns, J. & Keswell, M. 2006. Intra-Household Decision-

Making and Resource Allocation, Social Networks and

Social Cohesion: An Input Document in Preparation for

the National Income Dynamics Study. Available at nids.uct.

ac.za/publications/background-papers/116-intra-household-

decision-making-and-resource-allocation-social-networks-

and-social-cohesion/file.

Chow, G. & Harji K. 2014. How and Why We Measure Impact at

Samasource: Theory, Methodology, Limitations, & Challenges

(Samasource publication).

Harji, K., Best, H., Essien-Lore, E. & Troup, S. 2013. Digital Jobs:

Building Skills for the Future. The Rockefeller Foundation.

Available at rockefellerfoundation.org/blog/digital-jobs-

building-skills-future.

Jackson, E.T. 2013. Interrogating the Theory of Change:

Evaluating Impact Investing Where It Matters Most. Journal

of Sustainable Finance and Investment, 3(2), 95-110.

James-Wilson, D. 2013. Youth Livelihoods Development

Program Guide. EDC, USAID. Available at

youtheconomicopportunities.org/resource/1060/youth-

livelihoods-development-program-guide.

Poverty Outreach Working Group (POWG). 2011. Economic-

Strengthening Pathways for the Bottom Billion: Connecting

the Dots – E-Consultation Discussion Synthesis. The SEEP

Network. Available at seepnetwork.org/filebin/pdf/es4vp/

SEEP_EconStrength_web.pdf.

Raftree, L. 2013. Landscape Review: Mobiles for Youth

Workforce Development. The MasterCard Foundation,

the United States Agency for International Development

(USAID), and the mEducation Alliance. Available at

lindaraftree.com/2013/09/16/surveying-the-landscape-

mobiles-and-youth-workforce-development.

Thakkar, M., Floretta, J., Dhar, D., Wilmink, N., Sen, S., Keleher, N.,

McConnaughay, M. & Shaughnessy, L. 2013. Mobile-Based

Technology for Monitoring & Evaluation. Regional Centers

for Learning on Evaluation and Results (CLEAR). Available at

theclearinitiative.org/mobile-basedtechnology.html.

The Rockefeller Foundation. 2013. Impact Sourcing. Digital

Jobs Africa. Available at rockefellerfoundation.org/our-work/

current-work/digital-jobs-africa/impact-sourcing.

UNDP. 2013. Innovations in Monitoring & Evaluating Results.

UNDP – Knowledge, Innovation and Capacity Group.

Available at nec2013.org/documents/papers/Innovations-in-

mande.pdf.

Westley, F., Zimmerman, B. & Patton, M.Q. 2010. Getting to

Maybe: How the World is Changed. Toronto:Random

House,Canada.