Thank you for Joining Us! 
Portland, Maine
Proceedings
Proceedings from the 12th Annual Summit on Performance Management Excellence are included below.
Wednesday 9/4 Sessions | Thursday 9/5 Sessions | Friday 9/6 Sessions
PEQA Advanced Workshops
Wednesday, September 4
1:00 - 2:15 pm Reviewing Low-cost Methods of Program Evaluation with a Goal of Adoption and Use of Data Terry Donovan, PEQA-TAC, University of Wisconsin-Stout Vocational Rehabilitation Institute; Cayte Anderson, University of Wisconsin-Madison, University of Wisconsin-Stout Vocational Rehabilitation Institute
Part A: Interactive discussion of low-cost, no-cost approaches to program evaluation and data analysis, including exploration of Microsoft products (Excel, Power BI, pivot tables) and others. Bring your own laptop and/or please be willing to share. Part B: Interactive discussion of methods and approaches to presenting evaluation information in ways that lead to adoption and use of the data. Exercises and templates for participants to develop persuasive arguments will be provided.
1:00 - 2:30 pm Evaluating the Effectiveness of Pre-Employment Transition Services Using Reporting and Performance Data Betsy Hopkins, Christine Johnson, Melissa Diehl, and Rachel Anderson; WINTAC, National Disability Institute
Please join WINTAC subject matter experts as they discuss WIOA Performance Accountability System reporting challenges specifically related to evaluating the effectiveness of Pre- Employment Transition Services. The WINTAC team will share tools, creative approaches and strategic solutions related to quality assurance and program improvement from VR programs across the country. Attendees will have the opportunity to share their own best practices and ask questions throughout the presentation.
Thursday, September 5
9:45 - 11:00 am Best Practices in the Development and Implementation of Vocational Rehabilitation Internal Control Processes Rachel Anderson, WINTAC, National Disability Institute; Betsy Hopkins, WINTAC, National Disability Institute; Matthew Markve, Idaho Vocational Rehabilitation; Teresa Pitt, Idaho Vocational Rehabilitation
State VR agencies need to develop Internal Controls as they move forward with developing new Policies and Procedures related to WIOA implementation. VR agency staff can explore a possible framework to use for developing their own internal control documents. Learn about risk assessments, monitoring and auditing issues, case review instruments and quality assurance activities, data validation and integrity, and case
management system controls.
9:45 - 11:00 am - KTER Track Knowledge Brokering in Vocational Rehabilitation Agency Contexts Kathleen M. Murphy, KTER Center; Melissa Scardaville, American Institutes for Research; Amber Brown, Virginia Commonwealth University
This presentation summarizes results from the KTER Center’s intervention study of knowledge brokering in vocational rehabilitation agencies. KTER trained staff who supervised at least 4 VR counselors—and so were in a position to act as knowledge brokers—in either pre-employment
transition services or the employment of adults with autism. KTER staff followed the knowledge brokers for 6 months. They collected survey data from study participants, their staff, and a control group of supervisors and their staff to measure impact on changes in attitudes toward and use of research on-the-job. Follow-up roundtable is on Thursday at 11:15 am - 12:15 pm in Connecticut/Rhode Island.
11:15 - 12:15 pm Measuring Performance in The Future (Not Too Distant) Steve Wooderson, Council of State Administrators of Vocational Rehabilitation; Betsy Hopkins, WINTAC, National Disability Institute; Terry Donovan, PEQA-TAC, University of Wisconsin-Stout Vocational Rehabilitation Institute
A discussion of the data needs and performance metrics as VR programs head into the next five years. What are the underlying data needs for Vision 2020 to measure its impact? How will the field identify and highlight service improvements while under Order of Selection? What is the story VR should be thinking about telling five years from now in effort to support the ongoing growth of the field?
11:15 - 12:15 pm Knowledge Brokering – Where Should the Field Go Next? Kathleen M. Murphy, Joann Starks, KTER Center; Melissa Scardaville, American Institutes for Research; Amber Brown, Virginia Commonwealth University
Reaction/discussion session to previous KTER presentation on knowledge brokering and discussion of ideas about challenges and proposed solutions to engaging in KT in VR agency contexts.
12:30 - 1:45 pm - KTER Track Knowledge Brokering as an Evidence-Based Strategy Maureen Dobbins, School of Nursing, McMaster University
This keynote presentation provides an overview of knowledge brokering as a comprehensive strategy to develop capacity for trainees and to support
change in the work setting. Dr. Dobbins will share the results of her many years of research in this topic and will examine its application in the VR context in the US.
2:00 - 3:00 pm WIOA Performance Management Dashboards: Putting the Power in Power BI Margaret Alewine, South Carolina Vocational Rehabilitation
Follow-up to conference presentation on the development of a data warehouse, WIOA performance reports, and a Key Performance Indicators (KPI) dashboard will be reviewed and demonstrated. Through the use of Power BI software, access to timely performance data with data visualizations and drill down capabilities can be provided to staff. Other elements from the presentation will be included as time permits: development of reports for the WIOA common performance measures, descriptions of data validation efforts, and training issues identified in reporting.
2:00 - 3:00 pm Rethinking IPE Development – A Case Study in Systems Change in a VR Agency Betsy Hopkins, WINTAC, National Disability Institute; Libby Stone-Sterling, Maine Division of Vocational Rehabilitation
This presentation is about Maine’s struggle in developing VR plans within 90 Days. Current and former directors will reflect on this process and will talk about the resulting outcomes for clients, VR counselors, and staying within VR compliance. Strategies and tools that have resulted in a current statewide average of 66 days from eligibility to signed IPE will be shared. Learn how to identify three barriers faced by VR agencies, and how to access the tools used by Maine DVR to create and sustain IPE development.
2:00 - 3:00 pm - KTER Track Working with Schools: Improving Employment Outcomes for Youth Teresa Grossi, Indiana Institute on Disability and Community, Indiana University
Transition from school-to-work, including into the adult service systems can be daunting for the student, families and school personnel. The passage of the Workforce Investment and Opportunity Act 2014 (WIOA) allows for creative opportunities to work with schools on offering pre-employment transition services. This session will share the results of a School-to- Work Collaborative that embedded employment resources within the school from the adult services providers. How collaboration, roles and responsibilities between schools, Vocational Rehabilitation, and community employment providers enhanced agency connections, work experiences and employment outcomes will be shared. Participants will identify strategies to work effectively with schools and ultimately improve employment outcomes for youth within your local community. Understanding the school’s culture, expectations and ways to enhance collaboration will be explored and the challenges and solutions developed by the local collaboratives.
2:00 - 3:00 pm Application of the Statistical Adjusted Model in the VR Program: Measurable Skill Gains Indicator Yann-Yann Shieh, Data Collections and Analysis Unit, Rehabilitation Services Administration
A brief explanation of the Statistical Adjustment Model (SAM) will be shared. Workforce Innovation and Opportunity Act (WIOA) requires the use of a statistical adjustment model for setting targets for the core performance indicators. RSA will have the first performance indicator, MSG, target setting in early 2020. Dr. Shieh will demonstrate the steps that RSA works through on SAM to estimate their Measurable Skills Gains based on 911 data.
3:15 - 4:30 pm Early Employment Outcomes and Engagement of Transition-Age SSI Youth Receiving PROMISE Services Cayte Anderson, University of Wisconsin-Madison, University of Wisconsin-Stout Vocational Rehabilitation Institute; Amanda Schlegelmilch, Stout Vocational Rehabilitation Institute; Emily Brinck, University of Maine-Farmington; Kelli Crane, University of Maryland; Jade Gingerich, Maryland Department of Disabilities Employment Policy
This presentation will provide an overview of Promoting the Readiness of Minors in Supplemental Security Income (PROMISE), a large-scale experimental demonstration project conducted in six sites across the country including VR partners. Presenters will share interventions to move youth toward work-experiences and paid employment, and how they connected youth to VR services, including Pre-ETS. Gain a better understanding of the employment service and support needs of low-income individuals with disabilities in VR and discuss how serving the SSI population aligns with the WIOA and Order of Selection.
3:15 - 4:30 pm Measuring Fidelity of Four State Models for Expanding Work-Based Learning and Transition Opportunities Richard Tulikangas, Vermont Linking Learning to Careers; Libby Stone-Sterling, Maine Division of Vocational Rehabilitation; Susan M. Foley, Institute for Community Inclusion; Ellen Fabian, University of Maryland
Gain information about each state’s unique design elements related to common overall objectives for improving employment and post-secondary career pathway outcomes. Learn about different, innovative approaches to a range of work-based training experiences and related career and college readiness activities for high school students with disabilities. A panel discussion with the four states – Maine, Maryland, Massachusetts, and Vermont – will describe their projects. An overview of specific work-based training and implementation approaches, strategies for monitoring the fidelity of their models, and evaluation results to date at the midpoint of their projects are included.
3:15 - 4:30 pm Innovation and Evaluation in Vocational Rehabilitation and Employment Programs Noelle Denny-Brown, Mathematica; Todd Honeycutt, Mathematica; Purvi Sevak, Mathematica
This session will include three presentations on innovations to help agencies evaluate programs and practices. First is a discussion about an approach to evaluate existing employment programs using comparison groups. Second, an approach for extracting information from case files that could be used for performance management or evaluation will be examined. Third, a grass-roots process for agencies to identify programs that need improvement will be shared. Learn about specific, novel approaches to identify, measure, and evaluate services and programs.
3:15 - 4:30 pm - KTER Track The Vocational Rehabilitation Return on Investment Project: Employment Research and Knowledge Translation Rick Sizemore, Wilson Workforce and Rehabilitation Center; Bob Schmidt, University of Richmond; Joe Ashley, Ashley Consulting, LLC; Kirsten Rowe, Virginia Department for Aging and Rehabilitative Services; Rob Froehlich, The George Washington University
The VR ROI Project’s methodologically rigorous model is used to determine the employment impacts and return on investment of the VR Program. This session describes how a multi-layer knowledge translation plan is an essential component in order to facilitate effective application of research findings to practice. The researcher/practitioner interface, facilitation of stakeholder feedback, and experiences pertaining to all these topics will be discussed. Time will be reserved for a question and answer period.
Friday, September 6
Keynote: Rehabilitation Services Administration Presentation. WIOA at Year 5: Performance Accountability of the VR Program Mark Schultz, Commissioner, RSA; Christopher Pope, Chief, Data Collections and Analysis Unit, RSA; Andrew Kerns, Information Technology Specialist, Data Collections and Analysis Unit
In partnership, RSA, CSAVR, NCSAB, and VR agencies are committed to rethinking the way we look at the performance of the VR program, improving results, and maintaining high expectations of both the VR program and individuals with disabilities. Using data in our decision making, we will identify strategies that RSA and VR agencies may adopt to foster continuous improvement, collaboration, and maximize high-quality employment opportunities for customers served by the VR program.
9:45 - 11:00 am - KTER Track Using Knowledge Translation Models in Disability Employment Research Ann Outlaw, KTER Center
Participants will learn concepts underpinning knowledge translation to move conceptual discoveries from scientific research into practice by improving the relevance and application of research. Following an introduction to the Knowledge-to-Action framework (Graham et al., 2006), participants will practice applying this framework to developing knowledge translation activities appropriate for the vocational rehabilitation context.
11:15 - 12:15 pm - KTER Track Knowledge Translation – Where Should the Field Go Next? Facilitators: Kathleen M. Murphy, Joann Starks, KTER Center; Sarah Boege, Disability Statistics & Demographics RRTC (StatsRRTC), Institute on Disability, University of New Hampshire
Reaction/discussion session to the previous KTER session on Knowledge Translation and its application in VR contexts, with additional information from the Disability Statistics & Demographics RRTC.
11:15 - 12:15 pm Use of Texting for Improved Outcomes Jennifer Gundlach-Klatt and Terry Donovan, PEQA-TAC and University of Wisconsin-Stout Vocational Rehabilitation Institute
This Roundtable session will present statistics on the rise and use of texting as a method ofcommunication, engagement, and motivation. A brief discussion of the principles of behavioral psychology as it relates to use of texting. Examples will be given of how texting has been used in the public sector and specifically how SVRI uses texting to encourage and motivate individuals to finish courses, finish classes, stay motivated, and present timely information.
11:15 - 12:15 pm WIOA Performance Management Dashboards: Putting the Power in Power BI Margaret Alewine, South Carolina Vocational Rehabilitation
Follow-up to conference presentation on the development of a data warehouse, WIOA performance reports, and a Key Performance Indicators (KPI) dashboard will be reviewed and demonstrated. Through the use of Power BI software, access to timely performance data with data visualizations and drill down capabilities can be provided to staff. Other elements from the presentation will be included as time permits: development of reports for the WIOA common performance measures, descriptions of data validation efforts, and training issues identified in reporting.
12:30 - 1:45 pm PEQA-TAC Capstone Projects
Discover how the PEQA Cohort’s activities are impacting quality assurance in vocational rehabilitation and view additional posters related to quality assurance and program evaluation.
Vocational Rehabilitation Case Review Tool
Elaine Robertson, PEQA-TAC Cohort Member
The Impact of Missouri VR’s Summer Work Experience on Employment Outcomes
Chris Clause, PEQA-TAC Cohort Member
Evaluation of the SCVRD Information Technology Training Center
Jacob Chorey, PEQA-TAC Cohort Member
VR-ROI: Estimating Return on Investment in VR
Rob Froehlich, The George Washington University; Bob Schmidt, VR-ROI Principal Investigator; Joe Ashley, VR-ROI Principal Investigator; Kirsten Rowe, VA DARS Grants and Systems Development Specialist
Program Evaluation of Work Support Centers for Individuals with Psychiatric Disabilities in Japan
Jun Yaeda, Associate Professor, University of Tsukuba (Japan)
2:00 - 3:00 pm - KTER Track The Diversity Partners Project at Cornell Wendy Strobel-Gower, K. Lisa Yang and Hock E. Tan Institute on Employment and Disability, ILR School, Cornell University
Job Developers, Employment Specialists, Workforce Development staff and others in both the public and private sector who connect people with disabilities to jobs are the “bridge” between labor supply and demand. In order to serve as an effective bridge, these professionals must form partnerships with employers. A partnership is an arrangement where parties agree to cooperate to advance mutual interests. Ideally, the approach to employers shouldn’t just be about making a single placement, it should also be about understanding and meeting the needs of the employer in order to establish a long-term partnership. Diversity Partners is a combination of both facilitated training and consultation, and online learning modules, that can help professionals find better employment opportunities for people with disabilities through improved business relationships. Diversity Partners: buildingdiversitypartners.org
2:00 - 3:00 pm Evolution of Pre-Employment Transition Services and Internal Control in West Virginia Pisnu Bua-Iam, Douglas Snuffer, “Zeke” Hampton, Basil White, Jr., West Virginia Division of Vocational Rehabilitation Services
Expand your knowledge of successful Pre-ETS practices based on examples from West Virginia. Learn the basics of RSA and GAO internal control principals and guidelines. Identify how to apply the internal control principals to innovative Pre-ETS activities through fiscal and programmatic data (to set goals and objectives). Learn ways to connect available Pre-ETS resources with desired outcomes.
2:00 - 3:00 pm Assessing Future Needs for Students with Disabilities Utilizing State Department of Education Data and Mapping Techniques Barbara Burkett, Virginia Department for Aging and Rehabilitative Services
Explore the use of external data sources to anticipate need for services by the use of the EPI-Info for mapping. Compare current service delivery locations to anticipated service locations. Make better plans for changes in prevalence of disabilities among clients and consider mapping as a tool in future needs assessments. Review regulations and limitations of education data as it pertains to the Family Educational Rights and Privacy Act.
3:15 - 4:30 pm Five- and Ten-Year Return on Investment Estimates for VR Services in Three States Bob Schmidt, University of Richmond; Joe Ashley, Ashley Consulting, LLC; Kirsten Rowe, Virginia Department for Aging and Rehabilitative Services; Steven Stern, State University of New York at Stony Brook
Increase your knowledge of methodologically rigorous and scientifically sound methods for examining the impact of VR on long-term employment outcomes and estimating the ROI of state VR programs. Learn about the value of examining the impacts of up to nine specific types of VR services and four specific disabling conditions. This approach explicitly recognizes VR as an individualized program through which different VR participants receive different kinds of VR services. Learn about results of recent analyses of data from Maryland, Oklahoma and Virginia for individuals who applied for VR services during 2007.
3:15 - 4:30 pm - KTER Track National Employment Team Members: Supporting Vocational Rehabilitation’s Outreach to Businesses Kathleen M. Murphy, KTER Center; Amber Brown, Virginia Commonwealth University
This presentation will summarize results from the KTER Center’s research with members of the Council on State Administrators on Vocational Rehabilitation’s National Employment Team (the “NET”). KTER conducted a training needs assessment and scoping review on studies of efforts vocational rehabilitation has made to reach out to business. Their findings informed a training and follow-up support for NET members on how to function as knowledge brokers in their domains to raise awareness among other staff and employers about how vocational rehabilitation can support the needs of business.
This event is provided with support from the Program Evaluation and Quality Assurance Technical Assistance Center (PEQA-TAC) grant from the Department of Education, Rehabilitation Services Administration (RSA) award number PR#H263B150004. The contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government.