SR 06-26-2018 3O
City Council
Report
City Council Meeting: June 26, 2018
Agenda Item: 3.O
1 of 4
To: Mayor and City Council
From: Karen Ginsberg, Director, Community & Cultural Services
Subject: Award RFP #169 for Outcomes Design and Data Management Solutions
Recommended Action
Staff recommends that the City Council:
1. Award RFP# 169 to Canavan Associates, a Massachusetts-based company, for
redesign of performance measures and outcomes and implementation of a data
management system for human services programs.
2. Authorize the City Manager to negotiate and execute an agreement with
Canavan Associates for a total amount not to exceed $395,000 over a 3-year
period, with future year funding contingent on Council budget approval.
Executive Summary
City funding of non-profit agencies, direct services, and community partnerships offer
support for community wellbeing and a safety net to the City’s most vulnerable
populations, including youth, families, seniors, people with disabilities, low-income
households, and people experiencing homelessness. These efforts connect to the
City’s Framework Outcome Areas and advance Strategic Goals for Homelessness,
Learn and Thrive, and Inclusive and Diverse Community. Yet while the City works to
integrate all our efforts for collective impact to improve the lives of residents, data on all
these efforts and the results we achieve remains uneven. Developing and sharing data
can increase accountability, inform efficient investments and improve service delivery,
leading to better lives for program participants. By harnessing data to drive decisions,
this project will build on the City’s momentum as a 21st century government. Staff
recommends contracting with Canavan Associates to develop a library of outcomes and
associated metrics that connect to the City’s Framework and utilize innovative
technology to capture data for Human Services programs.
Discussion
2 of 4
The City has identified seven Framework Outcome Areas for a Sustainable City of
Wellbeing and five Strategic Goals. City departments are now working to develop
performance metrics for key projects and activities which connect to the Framework.
Within the Community & Cultural Services Department (CCS), the Human Services
Division (HSD) oversees a diverse range of programs and partnerships that directly
advance goals for addressing Homelessness, enabling all members of the community to
Learn and Thrive, and maintaining an Inclusive and Diverse Community. Through the
Human Services Grants Program (HSGP), the City provides over $8 million annually to
20 non-profits offering over 40 unique services to create a stable safety net for the City’s
most vulnerable residents. In addition, a variety of direct services and community
programs are offered through Virginia Avenue Park (VAP), the Police Activities League
(PAL), and CREST out-of-school time enrichment activities.
HSD’s current approach to program evaluation requires providers to submit periodic
reports containing a substantial amount of demographic, output, and outcome data.
While this method is sufficient for individual programs, standardizing metrics and
collecting the data in a digital format will inform comparisons across programs and help
measure progress toward citywide goals. In 2017, HSD engaged with Canavan
Associates (CA) to complete a thorough assessment of performance reporting tools and
data collection procedures utilized by its programs and agency partners. The CA team
surveyed City staff and HSGP grantees and studied program plans and reporting tools
in order to generate recommended actions. The resulting Recommendations for an
Outcomes Measurement System report (Attachment 1) lays out a roadmap to align
outcomes and metrics with the Framework and Strategic Goals and to implement an
accompanying data management system. The report’s central finding was that HSD
must first invest resources to define outcomes that best measure success before
implementing a data system. To that end, the report recommends a phased process to
engage stakeholders, collect more meaningful data, and incorporate modern technology
into a system that clearly analyzes program outcomes. This new contract will bring CA’s
team of data, finance, and policy experts into partnership with City staff, including
Human Services program analysts and a new Data Science Administrator, in order to
realize this ambitious vision.
3 of 4
Vendor/Consultant Selection
On April 2, 2018, the City issued a Request for Proposals (RFP) for the redesign of
performance measures and outcomes along with implementation of a data management
system for human services programs. The RFP was posted on the City’s on-line
bidding site and notices were advertised in the Santa Monica Daily Press in accordance
with City Charter and Municipal Code provisions. 53 vendors downloaded the RFP; one
firm responded. Procurement conducted additional outreach to vendors who
downloaded the RFP to see why they did not submit a bid; responses indicated that
vendors did not feel the project was the right fit for their firm or they were unable to
participate due to competing priorities.
The response to this RFP was reviewed by a selection panel of staff from Community &
Cultural Services and the City Manager’s Office. The proposal was evaluated based on
criteria in SMMC section 2.24.073, including experience, demonstrated competence,
capacity to promptly provide services, reputation of the firm, and compliance with City
specifications. Canavan Associates has 15 years of experience working with federal,
state and local partners to help communities design comprehensive, data-driven
approaches to human services delivery. CA put these skills into practice to implement
the City of Santa Monica’s Homeless Management Information System (HMIS) and
complete the Recommendations for an Outcomes Measurement System report. Based
on these criteria, staff recommends Canavan Associates as the best qualified firm to
develop a library of outcomes and metrics and implement a data management system
for human services programs.
Financial Impacts and Budget Actions
The agreement to be awarded to Canavan Associates is for an amount not to exceed
$395,000. Funds of $275,000 are available in the FY 2017-18 Capital Improvement
Program Budget in the General Fund (Data Management System for Human Services
Grants Program project), and annual funds of $40,000 are available in the FY 2018-19
Budget in the Community and Cultural Services Department. The contract will be
4 of 4
charged to the following accounts. Future year funding is contingent on Council budget
approval.
C019170.589000 / C0107110.689000 $275,000
012621.544450 / 01400003.521010 $120,000
TOTAL: $395,000
Prepared By: Marc Amaral, Administrator
Approved
Forwarded to Council
Attachments:
A. Recommendations for an Outcomes Measurement System
B. 2018 Oaks -Canavan Associates
City of Santa Monica
Human Services Division
Recommendations for an
Outcomes Measurement System
Understand the measurable.
Quantify the meaningful.20
1
8
Santa Monica Human Services Division Recommendations 2
Introduction
The City of Santa Monica Human Services Division has invested in a robust portfolio of projects and programs to support all segments of the
community. From infants, youth, and families, to seniors, the disabled, and homeless households, the services provided by the City through projects
like VAP, PAL, and CREST support the entire community. Through recent initiatives such as Cradle to Career (C2C), the Wellbeing Project, and
SaMoStat, the City of Santa Monica’s performance management program, the City has started to use data to measure how these and other
services benefit the community. As these initiatives generate and inspire more rigorous analysis methods, the City is increasingly poised for an
ambitious and transformative evolution in its approach to delivering human services.
Starting with the FY17-FY19 budget cycle, the City implemented the Framework for a Sustainable City of Wellbeing. This approach to
budgeting maps key department activities and projects with six outcome areas based on Civic Wellbeing, the Sustainable City Plan,
and the principles of good governance. Wellbeing, for example, takes a data-driven approach to articulating and measuring the
goals of City residents in areas such as life satisfaction or daily servings of fruits and vegetables. The project brings into focus
the wide diversity and income inequality in Santa Monica; households struggling to maintain stable employment and housing
will be unable to achieve many Wellbeing goals until their basic needs are met. Initiatives like Learn & Thrive and Inclusive
Diverse Community have begun to connect the visionary efforts of the Wellbeing Project with the full scope of needs within
the entire community.
Many HSD projects target these complex households, connecting resources to efforts that most effectively meet
the needs of Santa Monica’s most vulnerable populations. Consistent with communities throughout the United
States, the City of Santa Monica endeavors to create a modern, data-driven system to accurately measure
the effectiveness, success, and true impact of this work. As demonstrated by the recent hiring of a Chief
Performance Officer, the City is committed to the development and implementation of performance
management with clearly defined goals and metrics for all departments.
The City’s increasing focus on performance management through its recently established SaMoStat
program is intended to set City-wide outcomes and goals while increasing accountability.
Currently, the City is working to map work of all departments towards these outcome areas,
including the development of input and output metrics for Framework activities that tie back to
the outcomes. HSD joins these efforts with their intention to create a data-driven system to
benefit and improve their projects.
This report proposes a concise recommendation to design and build an outcome
measurement system for HSD projects made up of a technology component and
a library of outcomes, enhance staff capacity to effectively manage it, and hire
an experienced Data Science Administrator to lead the project going forward.
Typically, a project of this scope would take two to four years, but that timeline
can be accelerated based on how long it takes the City to accomplish
each phase. The City starts this process with a deep knowledge around
the delivery of human services and will only expand its expertise by
enhancing data collection and better analyzing project outcomes.
A successful effort will significantly improve the City’s approach to
service delivery, leading to better lives for Santa Monica’s most
vulnerable populations.
Canavan Associates
84 Sherman Street,
Cambridge, MA 02140
Phone: 617.492.0562
Santa Monica Human Services Division Recommendations3
Defining the Problem
HSD’s current approach to assessing services integrates physical site visits to service providers along with
semi-annual project and fiscal status reports. These reports, from both City-funded and City-managed
providers, contain clear descriptions of services offered along with demographic breakouts of participants
served and a quantitative measurement of each service supplied. Though HSD currently collects some
outcomes, without a standardized approach comparisons across the portfolio are not possible. While a
large amount of project data is collected, the data remains on paper and is not stored in a flexible format.
This in turn makes it difficult for the City and HSD to utilize and understand the data collected and to know
how best to assist their projects for the greatest benefit of project participants.
The City of Santa Monica is invested in ensuring quality services and outcomes for their most vulnerable
community members. To do so, they need to collect more meaningful data, implement a system that clearly
analyzes the outcomes of their projects, and incorporate modern innovative technology. A successful data
collection system will include consistent data systems, universal data elements (UDE), and standardized
outcomes based on UDE’s. All of these components will assist HSD and HSD’s projects to align with the
City’s other performance management activities and expand the way they serve their community.
Although technology and data are key pieces of an effective approach, they alone are insufficient to
improve outcome measurement or generate meaningful analysis. The most critical component is an
accurate assessment of what services are connected to success and how that success is measured.
This requires broad agreement by policy-makers, service providers, and the community. Each HSD
project must be measured by its larger impact on other community resources in order to understand if
clients are truly achieving their desired outcomes. Building consensus within the community around the
proper balance of privacy and utility of data will require intentional engagement with all stakeholders.
Fortunately, excellent tools to implement a layered approach are available; these are discussed further
under Action 9: Privacy.
Communities across the country have only just begun the challenging process of combining data, project
effectiveness, and technology to deliver the best possible human services. While consensus on the best
method does not yet exist, the City of Santa Monica and several other communities are leading the effort
to set data-driven expectations for their projects to improve outcomes and better help their communities.
Each attempt builds on previous efforts to expand the knowledge base and establish best practices,
and incredible potential for success remains. To that end, the City of Santa Monica boldly embraces this
challenge and joins other municipalities, such as Santa Clara County and the City of Boston, in seeking an
appropriate strategy to engage and partner with regional, state, and national service providers.
A successful effort will result in increased accountability for the City and its grantees and better outcomes
for participants. This creates an ideal environment for benefits to accrue in a variety of areas such as
finance (cost savings from lower arrest and detention rates), productivity (increased employment), and
social support (more stable family units and relationships).
The US Department of Housing and Urban Development’s (HUD) approach to data
regarding homelessness provides a valuable example of a successful data system for human
services. Over the past decade, HUD has invested over $10 million to create universal data
standards about homeless households and the effectiveness of projects. Its investment in
collaborative systems, academic research, and quantitative reporting now allows homeless
serving projects to connect with leaders in finance and government to make new investments
that achieve better outcomes for both the household and the larger community.
Santa Monica Human Services Division Recommendations 4
The Process
In order to identify a practical solution, the City of Santa Monica contracted Canavan Associates (CA),
a firm based in Cambridge, MA with nearly 15 years of experience working at the local, state, federal,
and international level of program management and design. The CA team previously worked with HSD to
implement the City’s Homeless Management Information System (HMIS), a HUD-supported platform that
collects and analyzes information about the effectiveness of homeless services provided.
The City of Santa Monica tasked CA with making specific recommendations to support HSD with the
design and implementation of a new outcome measurement system. The proposed system is comprised of
two main elements, a conceptual framework constructed by experienced human services practitioners and
a technology component designed and supported by information technology experts. These new tools will
improve project monitoring and increase provider accountability through uniform tracking of outcomes.
This, in turn, will result in more effective services leading to better lives for project participants.
CA began by identifying and gathering necessary sources of information. In order to gain a
comprehensive understanding of current Human Services projects, CA reviewed 2015-16 project
plans and outcomes, analyzed funding sources documented in project budgets, and administered
a survey instrument to HSGP grantees. The survey sought to better understand grantees’ current data
collection methods and to identify any potential benefits and concerns associated with a new outcome
measurement system. The survey results (Appendix A) illustrated that service providers believed a new
system could increase collaboration with other service providers, improve communication, and expand
understanding of client needs.
CA and HSD followed the survey with onsite interviews with a range of HSD projects to discuss topics
including operations, project design, and administration. CA also conducted interviews with a broad cross
section of City of Santa Monica staff, including staff from HSD; Community & Cultural Services (CCS);
the Information Systems Department (ISD); the Office of Civic Wellbeing; and the City Manager’s Office.
Review of project materials and follow-up interviews with project directors at the VAP, PAL, and CREST
projects provided excellent sources for understanding the work of the many professionals committed to
programmatic excellence.
From these interviews, it became clear that while the City requires projects to report on specific
achievements and client progress towards goals, many targets tend to measure outputs rather than
outcomes, and measures can vary widely across project types and populations. While valuable insight is
provided by the data currently collected, measuring success based on outputs as opposed to outcomes
does not fully determine how effectively a service is resolving the problem.
An output details what your organization does, whereas an outcome defines
changes that have taken place because of your organization’s work.
- The Non-Profit Times
The distinction between ‘outputs’ and ‘outcomes’ can seem purely semantic at first glance, but the
difference is critical. In the provision of human services, outputs are the units of services provided. In a
hypothetical illustration, an output for a project that provides emergency overnight shelter is the number
of bednights provided. A “bednight” is defined as a quantity of nights an individual utilized a bed in the
project. In this example, a project that has a maximum capacity of 20 beds, but only shelters 15 people
each night of the week, provided 105 (15 people in beds X 7 nights) bednights at the end of the week.
Continuing with the hypothetical of the shelter, if the shelter remained at 75% capacity (15/20 occupied
beds) the full year (364 nights), it will have provided 5,460 bednights in that year. Without additional
Privacy and
confidentiality are two
critically important
features of any
successfully outcome
measurement system.
To learn more about
the perspective
recommended here,
jump to Action 9:
Privacy.
Santa Monica Human Services Division Recommendations5
information, the community can perform very little further analysis of this project’s effectiveness. Were the
five beds out of service for the entire year? Is the project in a small town or a large city? Is the community
in a temperate climate or a seasonal one? While all of these details are helpful to understanding the
context of the project and the level of utilization, none of them answers the fundamental, outcome-focused
question asked by policy makers: “How is this project ending homelessness?”
When creating a system to measure project effectiveness, it is important to have clear definitions of
both success and its counterpart, failure. Before creating a centralized database, the City must review
its outcomes and performance measures for clarity and accuracy. Without clearly defined outcomes,
no technology would be sufficient to measure a project’s success, and prematurely implementing an
expensive software system that cannot meet the City’s long-term needs would be a failure. Once
outcomes and performance measures are clearly defined, an appropriate technology component can be
identified. In the interim, CA recommends using simple, inexpensive technology as a temporary solution
to automate the current paper reporting process while a more robust system is designed, piloted, and
implemented.
With this new understanding, CA and HSD decided to focus on laying a strong foundation that would
support a variety of technologies, service providers, and funders rather than immediately selecting
software that replicates current barriers.
Santa Monica Human Services Division Recommendations 6
Partnering with the
local school district
to better use data in
evaluating program
effectiveness may be
one of the priorities
of the new outcome
measurement
system. For example,
it could improve
understanding on
an aggregate level
of how students
who participate in
out-of-school time
(OST) direct services
see improvements in
reading proficiency
and other metrics
as part of their
engagement with
service providers.
The Recommendation
The City of Santa Monica seeks a superior method for collecting and analyzing meaningful data to improve
their projects and support their most vulnerable community members. Based on the interviews and surveys
completed by Canavan Associates, it is clear that HSD needs to define clear outcomes to measure their
projects’ success before they can implement a technology component. CA recommends HSD undertake
a three-phase process to design, test, and implement an outcome measurement system for all services
provided.
The outcome measurement system incorporates the rigor of current models of best practice while preserving
the flexibility and responsiveness required to meet emergent community needs. Time and expertise to
properly define outcomes and the constituent metrics are required. This new programmatic infrastructure
is at the cutting edge of human services delivery and represents a groundbreaking piece of work. The
process will result in the creation of a set of standardized outcomes across projects, an interim technology
solution to get reports off paper and into a digital format, and an eventual integrated technology solution for
centralized data. The comprehensive outcome measurement system consists of two components: one piece
is the conceptual connections between service provided, data collected, and successful accomplishment
of goals and the other is the actual technology or tools that will be utilized by HSD to implement those
interconnections. The initial phase of the project is focused on the development of the conceptual layer while
latter phases target identifying the correct tools or technology to implement it.
The following is a brief overview of the three-phase recommendation with further details specified later in the
report.
In PHASE ONE, the City will choose an implementation consultant and hire a Data Science
Administrator (DSA) to lead the effort to align outcomes, data, and service providers into a rational
system. The City will engage with the community, and Academic and Subject Matter Experts (SME)
to begin to determine how specific services impact the community and to define a framework to
organize the different HSD projects. The City will also construct a plan for a pilot project to test new
outcome measurement techniques by choosing existing projects to model and outcomes to measure.
Meanwhile, the DSA will develop and implement an interim solution to increase the efficiency and
effectiveness of current project reporting until the outcome measurement system is finalized.
In PHASE TWO, the City will continue to engage the community and SME to determine a library
of outcomes, outputs, and other details necessary for the new system. The new DSA will identify the
data the City already possesses and determine the best approach to acquire additional data. An
essential step will be mapping both community values and current law around privacy expectations
for Santa Monica residents. At this point, the DSA will be able to articulate the functional requirements
of any potential technology platforms. Once the outcome measurement system has been designed
and a plan for implementation has been determined, the City will meet with internal City partners and
external community partners to review the proposal.
In PHASE THREE, the City will implement the outcome measurement system using a phased rollout.
The new system will inform decisions for allocating HSD staff resources in order to better serve the
community. In addition, HSD will design a schedule of escalating programmatic support for service
providers to encourage participation and assist projects.
The City of Santa Monica joins communities across the nation as they seek to improve
performance management and implement data-driven systems to better analyze their services.
By collecting more meaningful data, analyzing outcomes rather than outputs, and implementing
a technology component, HSD will better understand their projects. Using a three-phase
recommendation created in conjunction with Canavan Associates, HSD will create an outcome
measurement system with an accompanying technology component that enhances the City’s
capacity to deliver human services to their most vulnerable populations.
Santa Monica Human Services Division Recommendations7
Project Design
The City of Santa Monica is poised for an ambitious and transformative overhaul of the way it uses data to evaluate its human services projects.
The successful realization of this three-phase endeavor will require significant resources, both current and new, along with careful navigation of
the delicate balance between privacy concerns and the benefits of data collection.
One of the City of Santa Monica’s most valuable resource is the years of expertise possessed by staff and service providers. HSD’s staff, in
particular, provides invaluable experience in the administration of projects that serve the City’s most vulnerable populations. Additionally, HSD can
leverage assets like ISD’s technical skill, Wellbeing’s expertise in data analysis, and the Chief Performance Officer’s experience with developing
metrics.
Another existing resource is HSD’s network of service providers, both HSGP grantees and the City’s direct services staff, who have
developed strong long-term relationships within the community. Several projects have senior leaders who were beneficiaries of City
services when they were children or adolescents. These deep connections to Santa Monica’s diverse communities are a unique asset
that will be critical to the success of this process.
HSD needs to add significant technical capacity to their staff by hiring a Data Science Administrator (DSA). The DSA will
develop and maintain a modern system of data collection and analysis and provide ongoing leadership and guidance
to staff and partners. The hiring of a DSA equips HSD with the long-term leadership required to build on the principles
and guidelines established during the initial implementation. In addition, HSD should identify external resources,
including experts from academia or other sectors, that neither receive City funds nor are residents of Santa
Monica. These resources can provide a valuable source of unbiased, objective information.
Privacy concerns understandably arise when collecting sensitive data about residents. Though the
City is already committed to ensuring the privacy of its clients, and current processes guarantee
that sensitive information is not shared, changes in data collection processes require that
City staff be mindful so that privacy is maintained. The crux of any privacy discussion is
consent; the City should identify and address any benefits which are contingent on the
sharing of potentially sensitive, personally identifiable information, such as religious
affiliation or immigration status. The City should also recognize other related
concerns, such as how data is collected, how data will be used, and with
whom it will be shared. Privacy will be discussed in greater detail in
Action 9, but it is necessary to remember the importance of privacy at
every step.
The execution of this recommendation will require the City’s
clear focus, commitment, and careful management
of resources. The following section of the report
breaks the recommendation into key action items,
separated by phase, which the City must
accomplish to develop a new outcome
measurement system.
Santa Monica Human Services Division Recommendations 8
Phase One: Engage
During Phase One, HSD will identify an implementation consultant and prepare City staff for the upcoming
process. They will hire a Data Science Administrator with expertise in data analytics. Engagement with the
community and a network of Subject Matter Experts will provide the public with updates and elicit crucial insight
from the community. A pilot project will test outcome measurement techniques. In addition, an interim solution will
be established to augment the effectiveness of current project reporting. By the end of Phase One, the city will be
prepared to design a robust outcome measurement system.
ACTION 1: Project Launch
Prepare City staff for the design and implementation of the outcome measurement system.
It is important that the City quickly identifies an implementation consultant to ensure successful project kick-
off. This role can be filled by a City staff member or by an independent contractor. The consultant will work
closely with HSD staff and coordinate with the Office of Wellbeing, the Chief Performance Officer, and
external subject matter experts to ensure alignment with strategic initiatives and stakeholder priorities. The
implementation consultant will also need to build and sustain the consensus of senior City leaders, service
providers, and the community throughout the process to deliver a comprehensive outcome measurement
system that remains aligned with City goals.
The ideal consultant will immediately establish a firm foundation to successfully support each phase of the
design and implementation of the outcome measurement system. The consultant will focus on designing
the conceptual framework, convening and facilitating initial groups of experts, and developing the skills of
HSD staff.
The consultant must be well versed in community engagement, outcome measurement, project leadership,
and building a network of experts, and will mentor HSD staff to increase their skills and knowledge in
these key areas. HSD staff should convene regularly with project coordinators and key service providers
to review targets and metrics in each area. Use of an internal team structure within HSD will ensure that
any turnover among staff does not derail progress.
ACTION 2: Hire a Data Science Administrator
Hire a new staff member to support implementation and maintain the new outcome
measurement system.
HSD requires an expert in human service analytics to provide ongoing project direction and maintain
alignment with other City departments and external stakeholder interests. Unlike the implementation
consultant, whose role is finite and will end once the outcome measurement system is designed and
implemented, the Data Science Administrator will serve to expand HSD’s resources by providing ongoing
leadership and guidance to HSD staff. In addition, the DSA will act as the custodian of the system,
ensuring that it continues to grow and operate optimally.
In order to hire the best candidate, HSD must search widely for the right combination of skills and
experience. Interviews should take a practical, skill-based approach and should utilize appropriate skill-
testing instruments that incorporate sample datasets. Expertise in data analytics will bolster the knowledge
1
Santa Monica Human Services Division Recommendations9
of existing staff, and enable HSD to develop and maintain a robust system based on best practices. This
individual should have a strong background in data analytics along with relevant experience in human
services administration. In addition, the ideal candidate will demonstrate the capacity to implement formal
procedures for building and interpreting meaningful datasets.
The Data Science Administrator will work alongside the implementation consultant and will be key to the
ongoing success of the outcome measurement system. The DSA will ensure continued alignment with best
practices and current academic research, and that data resources actually deliver meaningful information
to decision makers. As such, it is essential that the correct candidate be hired as quickly as possible.
ACTION 3: Community Engagement
Increase community engagement to determine how specific services impact the community and
define project success.
While each project has developed strong relationships with the constituents they serve, establishing a
group to review HSD’s portfolio of projects will require additional input from the community. Engagement
should seek to utilize existing resources like Virginia Avenue Park’s parent groups, which structure
community engagement through affinity groups. HSD will also engage the City’s advisory bodies,
including boards and commissions, for essential feedback. Philanthropic organizations are vital partners
to a successful collaboration and should be incorporated in any structured outreach. Initial tasks include
establishing relationships with community leaders, engaging with members of the community impacted by
HSD-funded projects, and convening regular community meetings.
Community members who depend on services provided by HSD are some of the best sources of
information about the effectiveness of those services. While City staff and consultants can draw on lessons
learned from other communities, the residents of Santa Monica themselves know intimately how well a
project is working for them and if it is not, why not. Engaging with Santa Monica residents will allow City
staff to discover what outcomes are most important to the community and how best to measure project
success.
During the first phase, HSD staff will design a strategy for community engagement through regularly held
meetings. Meetings should be scheduled to account for factors and strategies that inhibit or encourage
participation by diverse households. For instance, in order to ensure that meetings are accessible, it would
be helpful to hold the meetings outside of normal business hours. Meetings scheduled during times when
childcare would be needed may limit participation by some households. Making childcare available may
encourage households depending on the services delivered to attend.
Prior to convening any meetings, HSD will need to determine the metrics that define a successful
engagement. Potential indicators include total attendance, audience diversity, documentation of needs,
and discussion of successful interventions. Good initial discussion topics would include identifying the real-
world change HSD-funded projects seek to accomplish, ascertaining how progress towards these goals is
perceived, and generating additional project targets where appropriate. The final outcome measurement
system will be greatly improved by community engagement and feedback.
A successful system provides as much utility to service providers and clients as any other stakeholder.
Designing a system that makes possible reporting to many funders adds to the benefits for service
providers. In addition to government funders, adding philanthropic funders and foundations to early
discussions will support the broadest utility of the system.
Santa Monica Human Services Division Recommendations 10
ACTION 4: Build and Expand Network of Experts
Identify and engage a network of experts to define best practices and explore current outcome
research within different HSD service disciplines.
HSD staff will establish a network of local subject matter experts and refine HSD’s process for reviewing
and integrating academic literature. Developing a knowledgebase of current research related to HSD’s
services will ensure a solid foundation for building projects that are consistent with best practices. Utilizing
expertise to align partners towards common goals will serve to further increase the impact of invested
resources.
HSD provides a wide range of services that require different criteria for defining success. Luckily, the City
has a wealth of relevant expertise available via current residents, local colleges and universities, and
world-class consulting practices. While experts that reside or are employed in the City of Santa Monica
are a valuable resource, they are also members of the community. It is important that City staff have access
to confidential, expert advice that is not influenced by external factors such as future contracts or personal
household considerations.
In order to expand their current supporting network, the City should seek formal relationships with nearby
experts, academics, industry leaders, and potential partner organizations. Building a network of local
subject matter experts who can provide key insight and recommendations will ensure that sufficient rigor is
embedded into the design and administration of services as well as the outcome measurement system.
ACTION 5: Outcome Pilot
Use a sample of existing HSD projects to organize a pilot project that re-engineers outcomes
aligned with Council strategic initiatives, metrics, and action plans.
During the first phase, the City will work with community partners to select a sample of existing HSD
projects to pilot new outcome measurement techniques. The project pilots the union of the newly defined
outcome measures, and existing data collection and reporting capacity. The Data Science Administrator
will manage this project and use this pilot to identify opportunities and gaps that create barriers to full
scale implementation. During the pilot project, a subset of projects willing to collaborate on this new
endeavor will be identified to test the process of implementation. Using a pilot to test procedures minimizes
disruptions and avoidable mistakes. In addition, building momentum through early success will ease the
adoption of new technologies and procedures by encouraging participants and relieving anxieties.
It is important that the projects selected for the pilot project utilize service providers that have previously
demonstrated highly effective staff, excellent administrative capacity, and a willingness to partner with the
City on new projects. This will enable swift identification of problems and minimize other variables that
increase the risk of failure.
Through this facilitated process, the HSD team focused on the pilot will need to reach consensus on a
number of procedural methods, including:
• Review and redefine the intended outcomes of each pilot project;
• Determine what specific data each project will collect to measure achievement of expectations; and
• Decide the reasonable burden (such as number of hours) that service providers should be expected to spend
collecting such data.
During the pilot phase, the HSD team must create methods for evaluating their projects and implement
Santa Monica Human Services Division Recommendations11
needed changes if the pilot is not delivering the required information. Once the pilot establishes enough
proof of concept so that the city can apply a consistent strategy, HSD staff will be well positioned to
define the goals and processes of the outcome measurement system, and discuss them with local partners
and other communities defining human services outcomes.
As discussed, several similar efforts are underway in other communities around the country. Connecting
to cities or counties engaged in these efforts will provide valuable insight into potential pitfalls and
effective solutions. In addition, communication will help City staff establish new connections for continued
collaboration and support. Concrete questions developed at the conclusion of the pilot will ensure all
discussions produce actionable information and the necessary input to guide Santa Monica’s efforts.
ACTION 6: Implement Interim Reporting Solution
The implementation consultant and Data Science Administrator will develop a simple online
process for increasing the efficiency and effectiveness of current project reporting.
Working with appropriate internal and external partners, the DSA will lead the design of an interim
reporting solution that can be deployed immediately and that takes advantage of the City’s existing
software and data. A comprehensive and optimal outcome measurement system—the goal of this
recommendation—will take time to develop, so an intermediate reporting solution will allow HSD to
improve management of data currently being collected. This process side steps privacy related issues
and other concerns by using only aggregate level data, which avoids the inclusion of personal identifying
information and instead generates a cumulative number.
In order to design the interim reporting solution, the implementation consultant and the Data Science
Administrator will utilize a simple process inventorying the data currently collected by providers. This
process will not require new data collection procedures at existing grantee sites. Rather, instead of
submitting a paper report, as is the current practice, the same or similar information would be entered into
an online form or other virtual tool.
Santa Monica Human Services Division Recommendations 12
Phase Two: Design
During Phase Two, the City will design the outcome measurement system and develop a plan for its
implementation. Before the system can be designed, HSD staff will need to assemble a library of outcomes that
fits their projects. Once the library is complete, HSD can proceed with system design and decide on a technology
component that supports service delivery. HSD must pay careful attention to privacy issues when designing the
system to make sure that participant data is properly safeguarded. Once the system is designed, HSD will receive
and respond to feedback from external and internal partners. By the end of Phase Two, the City will have a clear
plan for the implementation of an outcome measurement system.
ACTION 7: Library of Outcomes
HSD staff design a library of outcomes.
At the beginning of Phase Two, the implementation consultant and Data Science Administrator will
collaborate with HSD staff, the Chief Performance Officer, external partners, and City staff to assemble
a comprehensive library of updated and refined outcomes and metrics for all human services projects.
Outcomes will be aligned with City goals and initiatives, and the library will contain all information
necessary to deploy the proposed outcome measurement system across HSD service providers.
The Data Science Administrator will work closely with HSD staff to incorporate key lessons learned from
the pilot project and community engagement. With a completed library of outcomes, HSD will increase its
capacity to document and evaluate project operations and to identify the best-fit outcome measures for
each service provider.
ACTION 8: System Design
Analyze data needs, design the outcome measurement system, and determine a coherent
program for its implementation.
At this stage, the Data Science Administrator is ready to design the outcome measurement system, identify
a technological component, and create an implementation plan. The DSA must determine what data
the City already possesses, what data the system would need to acquire, and the anticipated cost of
developing any additional required data. The DSA must carefully consider which sources of data will best
suit their system design, taking privacy into account. The privacy element is discussed in greater detail in
Action 9.
Once it is clear what data is necessary, the Data Science Administrator will work with the implementation
consultant to decide on the technology required to implement the outcome measurement system. The
purpose of the technology component is to receive, integrate, and organize data from multiple sources
into a central location or database. Once centralized, the data can be used to create dynamic, ad-hoc
reporting to inform more meaningful project analysis. This new technology will improve data collection
methods by taking into account how partners already collect and utilize their data to inform policymaking
and improving upon those methods for increased efficiency. Successful implementation of new technology
will make HSD and their partners more effective and better able to help the people in their community.
Different technologies offer distinct benefits. Matching the right technology to the right project is part of
a process known as requirements gathering. In order to identify which technology to use, the DSA will
2
Santa Monica Human Services Division Recommendations13
lead a requirements gathering process, which will run concurrently with the establishment of the library
of outcomes. At the conclusion of the requirements gathering process, HSD will have documented the
needs of each project and determined the degree to which any technology meets those needs. Specific
tools may easily collect the necessary information without further burdening service providers with data
collection. Soliciting the perspective of information system developers on formal process mapping can
support the identification of efficiencies and how to leverage them effectively. Discussions may include:
• Physical layout of entry and exit points,
• Reasonable data driven assumptions that can be programmed to application specifications, and
• Opportunities to utilize existing data in new ways, for example:
○Use existing data for data quality checks; if staff enter a date of birth that indicates the client is a young
child and they also indicate that the client is a veteran, then the staff would be prompted to resolve the
conflicting data,
○Refine access to ensure the most valuable data for decision making is easily and quickly available, and
○Ensure ad hoc reporting can occur quickly and easily to improve ownership and utility in decision-making.
This process must result in a complete set of functional requirements that the City can use to evaluate any
technology in the event that it decides to release an RFP for a commercial off-the-shelf system.
In order to identify the appropriate technology to pair with the outcome measurement system, the DSA may
consult with internal City experts. ISD would be an ideal partner, as they have extensive experience with
the practical evaluation of technology products. Many potential products struggle with import and export
of data – two functions critical to the success of any system. ISD may be able to share existing internal
or subscribed software that may meet HSD’s needs. Alternately, other appropriate City departments or
external partners may be consulted to achieve a similar result. In any case, all parties must give careful
consideration to the total cost of ownership, system and data control, and goals of any software used.
Total cost of ownership includes software licensing fees, support fees, hardware purchases, and technical
support costs. Some communities have adopted open source software that requires significant investment,
expertise, and data analysis, while other communities have chosen fee based software. Costs vary as a
function of the number of users, the nature of the projects involved, the volume of records entered, and the
complexity of reports required. During the initial year, costs will be higher to cover purchase of hardware,
data integration, and report configuration. An estimate of year one costs would be no more than
$350,000. This does not include potential staffing costs. In years two and three, costs will likely decrease
to less than $150,000 annually.
Using the requirements gathering process to examine technology the City of Santa Monica already owns
may reveal that the right long term solution already exists. While the development of the long-term solution
is under way, significant gains in efficiency and effective reporting can be made by simply moving the
existing reporting process online. As discussed in Phase One, any interim solution should be selected
and implemented with the long-term vision in mind. At the conclusion of this process, the Data Science
Administrator will complete a detailed implementation plan for the outcome measurement system and the
overlaying technology solution.
ACTION 9: Privacy
Data acquisition, issues of privacy, data use, and disclosure must be addressed.
In discussing the best approach for acquiring data, issues of privacy must also be explored. Looking at the
overall outcome library and understanding that some data that is desirable from an analytic perspective
may also be intrusive or inappropriate from a civil society standard is an important step in the process.
It is important to use aggregate, unduplicated, and anonymous data whenever possible as this improves
Santa Monica Human Services Division Recommendations 14
the flexibility and usability for analysts. Statistical and cryptographic procedures should be employed to
ensure that agencies can reliably aggregate individual records across systems without losing the unique
identity of the individual. Maintaining this utility without disclosing the identity of the individual is also
possible. In the design of datasets, this property is known as k-anonymity. Developed by Dr. Latanya
Sweeney, director of the Data Privacy Lab at Harvard University, this concept allows analysts to maintain
the maximum utility of data without compromising the privacy of individuals within the dataset.
Regardless of the technology chosen, both the City and its non-profit partners must be clear and
transparent about their intended use of the data at the time it is collected. To do so, they will need to
build specific use and disclosure commitments into contracts, end-user agreements, and individual intake
forms. Improper use of sensitive data can have a significant negative effect on an individual and must be
prevented. Depending on the nature of the technology involved, the hardware, and the context in which
data is collected (at a project site, through street outreach, in a participant’s home), specific controls and
procedures may be required. Review of City policy and other resources should result in a clear set of
expectations for the City, service providers, end-users, and service participants.
An excellent resource for designing a process that supports transparent decision-making and
accountability for all stakeholders in the system was developed by the Federal Trade Commission (FTC)
and is known as Fair Information Practice Principles, or FIPPs, and could assist the process. The core
principles of privacy addressed by FIPPs are notice/awareness, choice/consent, access/participation,
integrity/security, and enforcement/redress. FIPPs is outlined in greater detail in Appendix E. Having
a well-documented process for registering and resolving privacy complaints, along with an external
appeals process, will also help support the fair collection and use of data. In fact, formal mechanisms
and documentation for sign-off by the City, the community, and service providers will assist in ensuring
transparency to decision-making in the event that future changes become necessary
Clear authority regarding who – within the City – owns the system and is authorized to make changes, as
well as how they will communicate changes, should be included in any documentation. External providers
and end-users should execute commitments to adhere to the established standards. Documents should
transparently identify specific consequences including repayment of HSD funds, termination of grants
and/or employment, and - in the most severe circumstance – criminal or civil prosecution.
Santa Monica Human Services Division Recommendations15
ACTION 10: Continuous Internal and External
Feedback
Acquire continuous feedback from external and internal city partners to remain in alignment.
Throughout Phase Two, HSD will continue to provide progress updates to the community at regularly held
public meetings. At these meetings, residents can evaluate the success of the new system within their
community and provide valuable feedback, which will be responded to through a formal process.
Using the community engagement process, the Data Science Administrator should ensure Santa Monica
residents fully understand the proposed outcome measurement system and implementation plan.
At the same time, HSD should seek buy-in and feedback from service providers, domain specific boards,
commissions, and advisory bodies. Concerns around privacy, stigma, and easy access to life-sustaining
services should be addressed. Without sacrificing required functionality, HSD should incorporate feedback
from external HSD partners into the outcome measurement design.
As important as external stakeholder buy-in is, consensus from internal HSD partners is also essential.
The varying missions of different departments may come into conflict during this process and the Data
Science Administrator will need to meet with internal partners—including interdepartmental partners and
community stakeholders—to collect feedback. It is important to meet with the internal City users of the data
to determine if the new data will introduce new constraints. For example, if a performance management
goal or SamoStat metric is dependent upon access to a HSD data element, then the team will want to
understand the potential impact before making any changes to that particular element.
The Community Engagement Process
Santa Monica Human Services Division Recommendations 16
Phase Three: Implement
During the final phase of the recommendation, the City will begin a phased implementation of the outcome
measurement system. HSD will incorporate the new system into the decision-making process for allocating staff
resources. In addition, HSD will design a structure of escalating support to help service providers improve their
projects. By the end of Phase Three, the City of Santa Monica will have achieved its goal of improving data
collection and project outcome analysis to enhance the lives of project participants.
ACTION 11: Implement the Outcome Measurement
System
Implement the outcome measurement system over an 18-month period.
The optimal method for implementation is a phased approach. Any new procedure requires real-
world testing to identify errors and inefficiencies. Using a subset of pre-selected projects to assist with
this task will minimize disruptions and negative experiences with the new process. Building momentum
through early success will support the adoption of the new technology and procedures. Finding service
providers that have previously demonstrated highly effective staff, excellent administrative capacity, and a
willingness to partner with the City on new projects will enable swift identification of actual problems and
minimize risks that may contribute to project failure.
ACTION 12: Incorporate System into Decision-
Making
Incorporate the outcome measurement system into the decision-making process for allocating
HSD staff resources.
As the phased rollout progresses, HSD must integrate the resulting data into the decision-making process.
Grant agreements should require participation in the new outcome measurement system and technology
solution, and funded staff positions should incorporate timely data entry as a part of annual evaluations.
In addition, HSD’s process for awarding new and renewal grants should incorporate how HSD projects
meet expectations related to the outcome measurement system. Once a majority of service providers use
the outcome measurement system, HSD can begin to understand new baseline requirements and goals
for both the delivery of units of service and the effectiveness of each project. Community partners, service
providers, and other stakeholders should review these requirements and goals to ensure buy-in to the
process and the achievability of new targets.
ACTION 13: Project Support
Design a schedule of escalating programmatic support for service providers.
Over time, these new performance expectations and metrics should be incorporated into the regular
management of service providers and grantees. HSD can encourage participation and assist struggling
3
Santa Monica Human Services Division Recommendations17
projects as it gains insight into the operational capacity of projects. If projects are struggling, HSD can
offer programmatic support for those service providers.
HSD can support projects by helping them carve out distinct achievable goals, understand progress
towards those goals, and encourage commitment to creative new solutions. HSD should have a set
of resources available to assist providers with different challenges that emerge. Returning to resources
developed in Phase One, the engagement of subject matter experts will assist HSD in targeting the correct
resources. Modes of support likely include monitoring for contract compliance, providing opportunities for
project staff professional development, ensuring access to subject matter experts, and supplying technical
assistance.
Additional HSD recommendations may include coaching, administrative support, technical assistance,
or corrective actions. HSD can provide access to the network of experts or professional development
opportunities, improve their assessment instruments or data entry capabilities, or make structural or project
design changes. In this way, HSD can help transform the effectiveness and success of struggling projects.
Many projects will likely welcome access to these opportunities and a structured schedule of escalating
interventions ensures all providers and HSD maintain shared expectations throughout the grant period.
Conclusion
After completing the three-phase recommendation, the City will have implemented a new outcome
measurement system, integrated a technology platform, enhanced staff capacity to measure project
outcomes, and hired a leader in the Data Science Administrator to evolve the system. The City’s
ability to measure project success will be enriched by a greater understanding of outcomes and new
comprehensive techniques for measuring and processing the data collected. The system will allow the City
and HSD to improve their decision-making process, manage human service projects, and assist struggling
projects. These decisions will lead to stronger choices that benefit the entire Santa Monica community as
they receive needed services.
Moreover, the City of Santa Monica will join communities around the nation as a pioneer of data-driven
policymaking. The City will become a model of how to successfully combine data, project effectiveness,
and technology to proactively deliver the best possible human services. The path forward is clear as the
City uses this opportunity to improve human service delivery, enhance their decision-making process, and
transform their approach to system design.
Santa Monica Human Services Division Recommendations 18
City of Santa Monica Service Provider Survey
In order to comprehend the current state of data collection and use, an online survey was sent to the
City of Santa Monica’s HSGP grantees in February 2017. The survey was sent to seventeen providers
and fifteen responded. It looked at agency composition, agency management, data collection, and the
technology that these agencies currently use. It also sought to understand how a community wide client
management system could support each agency’s needs and to ensure availability of the resources
needed for successful operations.
The first section of the survey examined agency composition. Providers were asked about their target
populations, the types of services they provide, their clients, and how they track the total number of services
provided. The survey found that the respondent agencies are diverse in who they serve and can be
dramatically different in the volume of services offered. For instance, while one agency served 25,000
people in a year, another agency served 64 clients.
The second section of the survey concentrated on agency management. Providers were asked how
they update their policies and procedures and at what level they engage in policy decisions. Results
demonstrated that the majority of agencies (66%) reviewed and updated their policies and procedures
annually. Furthermore, agency involvement in City, state, and federal policy decisions varied, with
agencies more engaged in collaborations with local service providers than federal policy decisions.
The third section of the survey centered on data collection and usage. Providers were asked whether they
utilized an electronic data collection tool, what specific software solutions they use, the main benefits they
hope to gain from using a City supported platform, and potential barriers to implementing a community
wide data system at their agency. Respondents indicated a range of data collection tools, from the widely
accessible Microsoft Excel to highly configured systems like the Homeless Management Information
System (HMIS), along with numerous funder specified systems like CliniTrak. The possible benefits they
identified were increased collaboration with other service providers, better communication, and better
understanding of client needs.
The fourth and final section of the survey focused on each agency’s technology status. Providers were
asked who in their agency was responsible for keeping data secure, what type of IT staff they employ,
and the name of their leading IT staff person. The results found that most agencies have at least part-time IT
staff, with two agencies reporting that they have no dedicated IT staff.
The greatest takeaway from the survey was that the needs and resources of the agencies vary significantly.
The surveys were followed by in person interviews with a sample of grantees. Based on the information
gathered, Canavan Associates and HSD were able to proceed with an improved understanding of the
current state of HSD-funded providers and create a recommendation that better suited these agencies.
App
e
n
d
i
x
A
Santa Monica Human Services Division Recommendations19
A
p
p
e
n
d
i
x
B
City of Santa Monica Staff Discussion Guide
Interviewer will conduct interview with survey in hand and ground questions in survey responses.
Describe how you understand the benefits and structure of single platform, participant level data.
Discuss possible drawbacks, open discussion to gauge understanding of underlying concepts
and capacity.
Discuss data usage. What is the purpose of program data? What potential pitfalls and
opportunities do you see with using participant level data for reporting? Daily/weekly/
monthly/annual opportunities to use data? What policies/initiatives (fed/state/local) will
benefit the most from new data driven insights?
How well does your portfolio articulate success and failure of project participants? How do
you define success/failure of the grantee/project? Are these measurable? How so/not? Any
recommendations for what data should drive those determinations?
What do you perceive as the commitment by agencies to demonstrate their program outcomes?
Defined measurable outputs? Articulated goals? Measurable outcomes?
What is your confidence level in the agencies measuring their program performance accurately?
Discuss concerns and how a data system could improve the quality and usefulness of the data.
Discuss challenges of how the agencies measure program capacity (number beds/slots).
What’s going to be the biggest challenge to projects in your portfolio implementing this
system? What support can the City structure/make available to assist in meeting those
challenges? What resources would you like to have for yourself to assist in ensuring a successful
implementation?
Is there anything you wanted to talk about that we did not get a chance to cover? Any key
people in the agencies who you think might be helpful to launching and supporting this project?
Can we reach out to them at some point?
1
2
3
4
5
6
7
Santa Monica Human Services Division Recommendations 20
Santa Monica Service Provider Discussion Guide
Interviewer will conduct interview with survey in hand and ground questions in survey responses.
Describe benefits and structure of single platform, participant level data as an abstract concept.
Discuss possible drawbacks, open discussion to gauge engagement and understanding of
underlying concepts.
Discuss mission of organization. Demonstrate knowledge of operations and/or goals. Do
these tie into larger national goals or overarching schemas? Engage to determine how rigorous
agency’s commitment is to outcomes. Defined outputs? Correct formulation of outcomes
(change accomplished)? How does the agency actually measure that change?
How does the agency understand success and failure of project participants?
Diving deeper into discussion of outputs and outcomes. How long has the agency used the
framework (if using)? What analysis is done on the data and how rigorous is this analysis? Has
the agency updated outcomes or measures based on experience?
How does the agency’s mission and outcomes align/not align with the City’s Cradle to Career
Initiative? Opportunity Youth? Other significant City of Santa Monica initiatives?
Review the population selections that the agency made on the survey. Discuss population
specific goals. Review service provided selections from survey; discuss service outcomes and
measures.
Review scale of services (number beds/slots) discuss how the agency measures projects at
capacity, below capacity, and how service demand informs project scale decisions.
Based on scale, what kinds of special considerations to timing of data collection (scan cards vs.
interviews, sampling plan vs. every client, live data entry vs. delayed data entry).
1
2
3
4
5
6
7
8
App
e
n
d
i
x
C
Santa Monica Human Services Division Recommendations21
A
p
p
e
n
d
i
x
D
Steps to a Standardized Reporting Platform
The City of Santa Monica currently employs an offline process to collect the various data elements
included in the project and fiscal status reports required by City-funded agencies. This data is
aggregated at the project level, duplicated within projects, and is not longitudinal. Even with these
constraints, moving this process to a simple web-enabled database will immediately allow HSD to more
swiftly analyze data in response to grant applications, public inquiries, and service provider requests.
The requisite level of skill to produce such a system currently resides within ISD. The privacy implications
for this data are minimal as no personally identifiable data and very little sensitive data are being
collected. Further, the level of effort to design, implement, and sustain this system should be minimal.
Electronic data collection and reporting tools must articulate the need or business requirements of the
system users. The primary goals are:
City staff receives aggregate project data in an electronic format;
City staff can export the aggregate data to review performance of individual projects or across
multiple projects; and
Agency staff have a simple way to submit aggregate data to HSD.
To achieve these goals, the following describes the functional requirements, testing process, and
implementation steps.
THESE FUNCTIONAL REQUIREMENTS DESCRIBE THE WAY IN WHICH THE
STANDARDIZED REPORTING PLATFORM SHOULD BEHAVE OR PERFORM:
a. General:
i. Low cost hosting either internally by the City of Santa Monica or through an existing
provider that offers hosting solutions to the City;
ii. Web-based interface for end users, compatible with low bandwidth devices;
iii. Easy to use digital forms that are consistently available online and require minimal
training;
iv. Web-based interface for end users;
v. Allows for interface workflow or sequence of data collection questions to be
adjusted;
vi. Customization of questions and response values by City staff;
vii. Easily set-up collection forms with minimal training;
viii. Automatically assign a unique record number for each submission;
ix. Create automated log of when data is submitted and who submitted the data;
x. Display automatic flags for potential data quality issues (e.g. blank fields);
xi. Allow end user to correct data entry issues.
1
1
2
3
GOAL
A uniform electronic
submission process
for Human Services
project status
reports, permitting
staff to receive
agency data in a
format that allows
for easier analysis
and portfolio-wide
reporting.
Santa Monica Human Services Division Recommendations 22
App
e
n
d
i
x
D
DESIGN REFERENCE FOR THE STANDARDIZED REPORTING PLATFORM:
a. The goal is a low-cost form similar to Google Forms or Survey Monkey;
b. Design a simple back end database and front end user interface for data entry and for
running reports that can be exported in common file formats;
c. Set up reporting form with common questions and response categories;
d. Develop a simple data entry and report guide with sample report form.
HSD FORMS DEPLOYED IN THE STANDARDIZED REPORTING PLATFORM:
a. Existing reporting structures can be replicated in the new online forms;
b. Forms will use existing reporting requirements with minimal additions.
TEST THE NEW STANDARDIZED REPORTING PLATFORM:
a. Select two or three projects that will represent the array of agencies currently reporting to
the City of Santa Monica;
b. Train agency/project staff to use the new reporting tool;
c. Train selected City staff to review and export project data from the tool;
d. Have projects make submissions for three consecutive months;
e. Assess solution:
i. Does data entry process work as expected for both City of Santa Monica staff and
end users at the agencies?
ii. Does the data export as anticipated?
iii. Are exports usable and do they meet the goals of the reporting solution?
f. Make necessary adjustments to selected software solution and retest.
IMPLEMENT THE NEW STANDARDIZED REPORTING PLATFORM:
a. Create instructions for collecting and submitting uniform data based on final iteration;
b. Provide training and support to City staff and end users at agencies;
c. On selected “go live” date, all agencies will transition to the new reporting solution and
will be required to submit aggregate data through the new reporting process.
2
3
4
5
Santa Monica Human Services Division Recommendations23
A
p
p
e
n
d
i
x
E
FTC Fair Information Practice Principles (FIPPs)
There are common privacy principles adopted and promoted by the Federal Government that are
used across the country. These basic principles can be used to help ground and inform discussions
on fair data collection and usage activities in Santa Monica. An updated version of these principles
can be found in the 201 1, Fair Information Practice Principle (FIPPS). The principles are a foundation for
understanding privacy and developing policies specific to the data collection and usage. The following
is an excerpt from this update.
a. Access and Amendment. Agencies should provide individuals with appropriate
access to PII and appropriate opportunity to correct or amend Personal Identifiable
Information (PII).
b. Accountability. Agencies should be accountable for complying with these principles
and applicable privacy requirements, and should appropriately monitor, audit, and
document compliance. Agencies should also clearly define the roles and responsibilities
with respect to PII for all employees and contractors, and should provide appropriate
training to all employees and contractors who have access to PII.
c. Authority. Agencies should only create, collect, use, process, store, maintain,
disseminate, or disclose PII if they have authority to do so, and should identify this
authority in the appropriate notice.
d. Minimization. Agencies should only create, collect, use, process, store, maintain,
disseminate, or disclose PII that is directly relevant and necessary to accomplish a
legally authorized purpose, and should only maintain PII for as long as is necessary to
accomplish the purpose.
e. Quality and Integrity. Agencies should create, collect, use, process, store, maintain,
disseminate, or disclose PII with such accuracy, relevance, timeliness, and completeness
as is reasonably necessary to ensure fairness to the individual.
f. Individual Participation. Agencies should involve the individual in the process
of using PII and, to the extent practicable, seek individual consent for the creation,
collection, use, processing, storage, maintenance, dissemination, or disclosure of PII.
Agencies should also establish procedures to receive and address individuals’ privacy-
related complaints and inquiries.
g. Purpose Specification and Use Limitation. Agencies should provide notice
of the specific purpose for which PII is collected and should only use, process, store,
maintain, disseminate, or disclose PII for a purpose that is explained in the notice and is
compatible with the purpose for which the PII was collected, or that is otherwise legally
authorized.
h. Security. Agencies should establish administrative, technical, and physical safeguards
to protect PII commensurate with the risk and magnitude of the harm that would result
from its unauthorized access, use, modification, loss, destruction, dissemination, or
disclosure.
i. Transparency. Agencies should be transparent about information policies and
practices with respect to PII, and should provide clear and accessible notice regarding
creation, collection, use, processing, storage, maintenance, dissemination, and
disclosure of PII.¹
¹ National Strategy for Trusted Identities in Cyberspace, Enhancing Online Choice, Efficiency, Security, and Privacy at Appendix A
(201 1), https://www.hsdl.org/?view&did=7010
Tasks Task Owner Duration
(Days)
Level of Effort
in Hours Y1M1 Y1M2 Y1M3 Y1M4 Y1M5 Y1M6 Y1M7 Y1M8 Y1M9 Y1M10 Y1M1 1 Y1M12
Task ID Project Design Phase 1
1.1
Ac
t
i
o
n
1
-
P
r
o
j
e
c
t
L
a
u
n
c
h
Articulate key attributes and skills required of an implementation consultant along with proposed deliverables Human Services Division
(HSD)30 15
1.2 Implement a procurement process and hire an implementation consultant City of Santa Monica 45 3
1.3 Enter into a contract with an implementation consultant (please note that if it takes longer than anticipated for
the City to execute a contract, the timeframe for other tasks may need to be adjusted)City of Santa Monica 30 3
2.1
Ac
t
i
o
n
2
-
H
i
r
e
a
D
a
t
a
S
c
i
e
n
c
e
A
d
m
i
n
i
s
t
r
a
t
o
r
Announce open position for Data Science Administrator with business expertise in human services and exper-
tise in data analytics HSD 30 5
2.2 Use practical, skill-based interviews to ensure hiring the correct candidate HSD 30 80
2.3 Select and hire the top candidate HSD 14 15
2.4 Orient the new Data Science Administrator to city practices and the outcome measurement system
implementation plan HSD 14 40
3.1
Ac
t
i
o
n
3
-
C
o
m
m
u
n
i
t
y
En
g
a
g
e
m
e
n
t
Identify clear goals for engagement HSD 10 3
3.2 Determine the time and location of community meetings and decide on metrics that define a successful
meeting HSD 10 5
3.3 Hold monthly community meetings on the scheduled dates to share knowledge about project design and
project success HSD Ongoing 67.5
4.1
Ac
t
i
o
n
4
-
B
u
i
l
d
a
n
d
E
x
p
a
n
d
Ne
t
w
o
r
k
o
f
E
x
p
e
r
t
s
Build a network of local subject matter experts (SMEs) to provide insight on the design and administration of
services provided, and on design and measuring services HSD 90 60
4.2 Provide SMEs and Academics with access to data sources or study opportunities that can foster a long-term
relationship HSD Ongoing 92
5.1
Ac
t
i
o
n
5
-
O
u
t
c
o
m
e
P
i
l
o
t
Work with the implementation consultant to form an HSD staffed workgroup and begin facilitating weekly
meetings HSD 7 5
5.2 Come to consensus on which three existing HSD grantee projects to measure and how to execute and
evaluate the pilot project HSD Workgroup 60 25
5.3 Define the intended outcomes for each project, determine specific data to collect, decide the reasonable
burden expected of service providers to collect data, and design a method for evaluating the project HSD Workgroup 30 35
6.1
Ac
t
i
o
n
6
-
I
m
p
l
e
m
e
n
t
I
n
t
e
r
i
m
Re
p
o
r
t
i
n
g
S
o
l
u
t
i
o
n
Utilize a simple process to inventory data currently collected by providers and submitted to the City Data Science
Administrator (DSA)30 20
6.2 Design an interim reporting solution to deploy immediately, taking advantage of the City’s existing software
and data DSA 30 30
6.3 Implement the interim reporting solution DSA 90 30
Project Design Phase 1 - Engage
Tasks Task Owner Duration
(Days)
Level of Effort
in Hours Y2M1 Y2M2 Y2M3 Y2M4 Y2M5 Y2M6 Y2M7 Y2M8 Y2M9 Y2M10 Y2M1 1 Y2M12
Task ID Project Design Phase 2
7.1
Ac
t
i
o
n
7
-
L
i
b
r
a
r
y
of
O
u
t
c
o
m
e
s
Incorporate lessons learned from the pilot project and community engagement to determine a thorough library
of outcomes DSA 60 90
7.2 Document project operations and identify the best-fit outcome measure for each service provider HSD 30 40
8.1
Ac
t
i
o
n
8
-
S
y
s
t
e
m
D
e
s
i
g
n
Analyze the current capacity of City data resources to populate specific outcome metrics HSD 60 30
8.2 Identify any metrics unsupported by current data resources HSD 60 15
8.3 Determine ideal sources for unsupported metrics HSD 60 20
8.4 Identify the anticipated cost of developing missing data HSD 14 5
8.5 If ideal sources include sensitive, personally identifiable information, work with stakeholders to reduce or
eliminate privacy concerns DSA 90 15
8.6 Assemble a complete set of functional and technical requirements that can be implemented and tested against DSA 60 25
8.7 Identify efficiencies to be leveraged (i.e. adjusting workflow, program assumptions into application
specifications, utilizing existing data in new ways)DSA 30 20
8.8 Identify opportunities for automation and technology to relieve service providers of data collection burdens DSA 30 10
8.9 Make a determination about building, acquiring, or leveraging existing technology to deliver the required
functionality DSA and HSD 30 30
8.10 Evaluate prospective products to ensure they will meet functional and technological requirements DSA and HSD 30 60
8 .11 Synthesize information from requirements gathering process into a coherent program that can be systematized
and implemented within 18 months DSA 30 25
9.1
Ac
t
i
o
n
9
-
P
r
i
v
a
c
y
Ensure the proposed solution uses aggregate, unduplicated, and anonymous data as well as statistical and
cryptographic procedures DSA 15 10
9.2 Review City rules, FIPPS, and other resources to set clear use and disclosure expectations, for the City, service
providers, end-users, and service participants DSA 180 20
9.3 Determine what policies and procedures will be developed prior to public engagement DSA 90 15
9.4 Facilitate discussions with the public on privacy and policies that require external stakeholder buy-in DSA 60 30
9.5 Document policies and procedures covering privacy, data use, grievance procedures with stakeholders and
SME’s DSA 60 15
9.6 Determine and document clear authority regarding who within the City owns the system, who is authorized to
make changes, and how changes will be communicated DSA 60 10
9.7 Ensure the proposed solution meets privacy standards DSA 30 15
10.1
Ac
t
i
o
n
1
0
-
C
o
n
t
i
n
u
o
u
s
I
n
t
e
r
n
a
l
a
n
d
E
x
t
e
r
n
a
l
F
e
e
d
b
a
c
k
Over the series of regular community meetings (continued from Phase 1), provide regular progress updates to
the community DSA Ongoing 18
10.2 Provide the community with documents that articulate clear expectations about its role in the development of
the outcome measurement system DSA 5 20
10.3 Meet with external partners to ensure it is understood, assess level of buy-in, and determine if the new data
introduces new constraints DSA 90 40
10.4 Meet with internal City partners to ensure it is understood, assess buy-in, and determine if the new data
introduces new constraints DSA 14 5
10.5 Identify and implement steps to mitigate constraints DSA 90 50
10.6 Resolve any re-use issues (e.g. privacy, stigma, and easy access to life-sustaining services) among external
and internal partners that may be a result of using data in a way not originally intended DSA 180 50
10.7 Set internal policy that ISD and HSD review the system on a regular schedule DSA 30 10
10.8 Set policy to incorporate new projects into the system as they are developed DSA 30 15
10.9 Provide HSD a formal project plan, milestones, phased roll-out of the outcome measurement system and any
procurement process for the an appropriate solution DSA 27 50
10.10 Determine if mobile applications or other modality is necessary for data acquisition DSA and HSD 549 40
Project Design Phase 2 - Design
Tasks Task Owner Duration
(Days)
Level of Effort
in Hours Y3M1 Y3M2 Y3M3 Y3M4 Y3M5 Y3M6 Y3M7 Y3M8 Y3M9 Y3M10 Y3M1 1 Y3M12
Task ID Project Design Phase 3
11.1
Ac
t
i
o
n
1
1
-
I
m
p
l
e
m
e
n
t
t
h
e
O
u
t
c
o
m
e
Me
a
s
u
r
e
m
e
n
t
S
y
s
t
e
m
Develop a phased implementation schedule across the portfolio of projects DSA 30 25
11. 2 Select a sub-set of projects with highly-effective staff, administrative capacity and willingness to partner in
implementing and testing the system DSA 30 10
11. 3 Prepare, train, and implement the new outcome measurement system with the pilot Phase 1 projects DSA 60 120
11. 4 Test for errors and inefficiencies among projects in the first phase of the implementation DSA 60 60
11. 5 Prepare, train, and implement the new outcome measurement system with the next phase of projects DSA 30 70
11. 6 Test for errors and inefficiencies among projects in the second phase of the implementation DSA 60 40
12.1
Ac
t
i
o
n
1
2
-
I
n
c
o
r
p
o
r
a
t
e
S
y
s
t
e
m
i
n
t
o
De
c
i
s
i
o
n
M
a
k
i
n
g
Once projects consistently and successfully participate in the new system, integrate resulting data into HSD
process to embed system use at the institutional level DSA N/A 40
12.2 Ensure grant agreements require agency participation and that the agency implements policy that staff posi-
tions incorporate timely data entry as an element of annual evaluations DSA 546 20
12.3 Ensure the process for awarding new and renewal grants from HSD include meeting expectations related to
the outcome measurement system DSA N/A 15
12.4 Set baseline requirements and goals for both the delivery of units of service and the effectiveness of the project
once there is sufficient coverage of a majority of service providers DSA 79 40
12.5
Facilitate review of baseline requirements and goals for unit and service delivery by community partners, high
functioning service providers, and other stakeholders to ensure buy-in to the process and assess the reason-
ableness of new targets prior to formal implementation
DSA 79 30
13.1
Ac
t
i
o
n
1
3
-
P
r
o
j
e
c
t
S
u
p
p
o
r
t
Implement a set of resources (i.e. technical assistance, funding, capacity building materials) to assist providers
with different challenges that emerge. HSD Ongoing Ongoing
13.2 Engage subject matter experts to assist HSD in targeting the correct resources HSD Ongoing Ongoing
13.3 Monitor for contract compliance, providing opportunities for project staff professional development, ensuring
access to subject matter experts, and supplying technical assistance HSD Ongoing Ongoing
Project Design Phase 3 - Implement
REFERENCE:
Agreement No. 10707
(CCS)