Evaluation FAQs

Vermont YIT Program > Evaluation > Evaluation FAQs

[spoiler title=”What is the evaluation?” open=”0″ style=”1″]The Young Adults in Transition (YIT) evaluation is a required part of Vermont receiving the YIT funding from the federal Substance Abuse and Mental Health Services Administration (SAMHSA). It provides information about your local system of care achievements, young adult and family outcomes, and mental health and substance abuse service needs. The overall evaluation mission is to provide information that is useful to young adults, their families and adult allies, service providers, regional steering committees, and Vermont.[/spoiler] [spoiler title=”What are the differences between the four evaluation studies?” open=”0″ style=”1″]There are four specific studies that make up the YIT evaluation. Regional partners are responsible for collecting data for the Common Study and Services and Costs Study.

1. Common Study (Data collected by regional SOC staff)

This study collects descriptive information on all young adults, ages 16 to 21, served by their region’s funded system of care at the entry (or re-entry) into services. Eligible young adults are those between the ages of 16 and 21 who meet the Act 264 definition of Severe Emotional Disturbance (SED) or who are considered at risk for SED (see Study Inclusion Criteria Checklist). Because there is some identifying information we are required to collect, we need to obtain consent from young adults and if under 18 their parents or guardians as well for them to participate. The consenting and collection of data for the Common Study is designed to take approximately 20 minutes per client. The first Common Study interview must be conducted within seven days of intake into a region’s funded system of care, follow-up interviews are conducted every 6 months while the youth is receiving services, and then a discharge interview is conducted when they leave services.

We also work with the regions to identify eligible study participants and obtain permission for the UVM evaluation team to contact them further to elicit participation in the Vermont and National studies (see below). While introducing the Common Study to the young adult (and their parent/caregiver if young adult is under 18 years of age), we would also like region staff to obtain permission for us to contact them to describe the next two studies (Vermont and National) in more detail. We have a protocol and forms for each region to guide this process.

2. Vermont Study (Data collected by the VCHIP team)

This is a voluntary study for young adults receiving services from their region’s funded system of care and their caregivers (or adult allies). The goal is to review how the system of care promotes positive young adult development and meets the needs of young adults who have experienced mental health, substance use, or other social problems. Interviews will be conducted close to the start of services, at 6 months, and at 12 months. The interviews will take approximately one hour and are conducted by the VCHIP team.

3. National Study (Data collected by the VCHIP team)

This is a voluntary study for young adults with SED and their caregivers (or adult allies) at five points in time: at the start of services, and then at 6 months, 12 months, 18 months, and 24 months. The interviews are more in-depth than the Vermont study (lasting about two-three hours), covering a broader number of life areas and obtaining greater detailed information. These data will also be aggregated and combined with the National sample of System of Care service recipients. The interviews for the National study are also conducted by the VCHIP team.

4. Services & Costs Study (Data collected by regional SOC staff and VCHIP review of MSR data)

This study collects costs information about the services received by all young adults participating in the National Study only. Regions are responsible for working with the VCHIP team to use existing sources of services and costs data to collect this information. If the current services and costs data do not provide the necessary information, then use measures provided by SAMHSA.

Additional data will be collected by the evaluators using existing databases from a variety of Vermont agencies with participants’ permission.[/spoiler] [spoiler title=”Who is in charge of the evaluation?” open=”0″ style=”1″]The Vermont Child Health Improvement Program (VCHIP) at the University of Vermont (UVM) was hired to conduct the evaluation of Vermont’s YIT grant. There are three evaluators (collectively working about 2.0 FTE on this project): Thomas Delaney, Jody Kamon, and Jesse Suter. Tom and Jody work for VCHIP and Jesse works at the Center on Disability and Community Inclusion at UVM. Ellen Talbert is the evaluation coordinator at VCHIP. Stasia Savasuk is a part-time interviewer collecting data in southern Vermont. Interviews with young adults and caregivers (or adult allies) will be conducted by Tom, Jody, Jesse, and Stasia. The evaluators are members of the statewide YIT Operations and Outreach Team and seek guidance from the E-Team, an advisory group of young adults, family members, and adult allies.[/spoiler] [spoiler title=”What questions will the evaluation try to answer?” open=”0″ style=”1″]

Who are the young adults and families served?
What services do young adults and families receive?
What do those services cost?
How much do outcomes improve for young adults and families over time?
How do services get coordinated over time to help young adults and families?
What do young adults and families say about their experience with systems of care?
What factors help to sustain the system of care after federal funding ends?
How can Vermont better support Community Mental Health Centers and other programs and agencies that provide behavioral health interventions for young adults?

[/spoiler] [spoiler title=”What specific information will be collected to answer these questions?” open=”0″ style=”1″]

Common Study: Descriptive information (e.g., age, gender, race / ethnicity), Diagnoses, Enrollment information, Overall functioning, Substance use, Living situation, Perspective about services.
Vermont Study: Strengths, Emotional and behavioral problems (ASEBA), Trauma symptoms, Substance use, Healthcare coverage, utilization, and related behaviors, Caregiver strain.
National Study: Strengths, Emotional and behavioral problems (ASEBA), Trauma symptoms, Substance use, Healthcare coverage, utilization, and related behaviors, Caregiver Strain, Education, Living situation, Delinquent or legal behaviors, Perspective about services including satisfaction.
Services & Cost Study: Only for young adults that are in the National Study: Information on most system of care related services received. For each service provided, need to document dates of services, type of service, location, cost, and the source of payment.

[/spoiler] [spoiler title=”What more can you tell us about the Services and Costs study?” open=”0″ style=”1″]The latest information we have received from SAMHSA has clarified that the Services and Costs study includes collecting information on a wide range of services being received only by young adults participating in the National Study. For many regions this will be approximately 3-8 young adults per year (Burlington has about 20 per year).

What Services to Include:

Your data collection efforts for the Services and Costs Study should include services provided by the following organizations, both public and private:

mental health agencies and providers
child welfare agencies and providers
juvenile justice agencies
education (special education services only)
family and youth organizations
physical health care services including those related to mental health care and psychiatric medications
substance abuse treatment services
general education services

Include any data you may have on informal, natural support, in-kind, or volunteer services. If possible, we are also asking you to include data on services received as early as 1 year prior to the young adult enrolling in your region’s enhanced system of care. Again this is only for young adults participating in the National study who have consented/assented to share these available data.

Your region may be using SAMHSA dollars as flexible funds. Flexible funds represent a pool of discretionary funds your region may have allocated within your plan to support children, youth, and families by purchasing items or services (e.g., rent, utilities, clothes, food, camp, car repair, etc.) that are not covered by other funding sources. If your region is providing services to young adults in the National study using flexible funds, you will need to collect information on services provided through flexible spending funds.

[highlight bg=”#E85C2C” color=”#fff”]There should be only one record for each service event, even though there may be more than one organization involved. For example, if a service was provided by a mental health agency based on a referral from a juvenile justice agency, it should be included as only one service event.[/highlight][/spoiler] [spoiler title=”How can this information be useful for regions?” open=”0″ style=”1″]We are very interested in making the data collected practically useful for young adults, their family and adult allies, regions, and Vermont. Once enough data has been collected in your region, the evaluators plan to provide your region with monthly reports regarding enrollment, demographics, services, costs, and outcomes. We will be working with regional steering committees to determine what information is most useful for you. Here are a few examples for how we think the information can be useful:

Celebrating Success and Identifying Unmet Needs

The regional reports can provide an opportunity to examine how your system of care may be meeting the needs of young adults and for identifying areas where more work needs to be done.

Continuous Quality Improvement

Regions are encouraged to compare their regional plans to the information collected as part of the evaluation. Progress on goals in your regional plans could be measured using information collected through the evaluation.

Sustainability

The federal funds are limited, in both their allocation and how quickly the grant will end. Being able to demonstrate positive effects for young adults from regional plans provides a strong rationale for sustaining the systems, services, and supports after the grant ends.
[/spoiler] [spoiler title=”Who is participating in the evaluation?” open=”0″ style=”1″]

Young Adults

Eligible young adults are:
a. 16 through 21 years of age,
b. Served by their regional system of care funded by the YIT grant, and
c. Meet Act 264 criteria for Severe Emotional Disturbance (SED) OR are considered at risk for SED (see Study Inclusion Criteria Checklist).

Caregivers or Adult Allies

In addition to conducting interviews with young adults, the evaluation includes interviews with their caregivers. Recognizing different young adults may have very different people acting as caregivers (e.g., biological parents, adoptive parents, relatives, foster parents), this is designed to be an inclusive term defined by the young adult and his/her family. In addition, some young adults may not report having caregivers because they are responsible for their own care. In these cases we will ask them to identify an adult ally who could complete the “caregiver” interviews.

Steering Committees

Each steering committee will work with the VCHIP evaluation team to implement the evaluation in their region.

Evaluation Liaisons

Steering committees will need to designate an evaluation liaison who will be the evaluation team’s point of contact and assist in regional data collection.

Evaluation liaison and regional evaluation support

Each region will work with the VCHIP evaluation team to implement the evaluation in their region. Regions will need to designate an evaluation liaison who will be the evaluation team’s primary point of contact assisting with regional data collection. Other members of the regional system of care may be asked to adopt specific roles related to the evaluation depending upon each region’s data collection plan.

Fiscal and/or Information Technology Staff

Collection of event-level data for the Services and Costs Study will most likely involve the participation of fiscal and / or information technology staff in your region to ensure accurate data collection and reporting.
[/spoiler] [spoiler title=”What is the estimated sample size for each region?” open=”0″ style=”1″][/spoiler] [spoiler title=”What is expected of young adults and caregivers (or adult allies)?” open=”0″ style=”1″]

Common Study

The majority of information collected for the Common Study is to determine study eligibility and meet federal requirements of providing descriptive information and National Outcome Measure data. This process can be incorporated into the intake with minimal additional time burden on young adults (15 to 20 minutes at intake). An information sheet will be provided to help inform the young adult and their caregiver as to what information is being collected and how it will be used. All information can be obtained by the young adult and/or caregiver. Written consent (from a parent, legal guardian, or young adult if 18 years or older) and assent (from a young adult younger than 18 years) is needed before administering the Common Study.

Decide if they want to receive a call to learn more about the Vermont YIT SOC evaluation. If the young adult is under 18 years of age, their parent or legal guardian must sign the “May We Call You” form, allowing the UVM evaluation team to call and describe these studies further.
Compensation: None

Vermont Study

Consent to participate.
Participate in three interviews (each approximately one hour) at the start of services, 6 months, and 12 months.
Decide if they want to release records from departments and agencies they have worked with to the evaluators.

Compensation: $10 Per Interview

National Study

Participate in five interviews (each approximately two to three hours) at the start of services, then at 6, 12, 18, and 24 months.
Decide if they want to release records from departments and agencies they have worked with to the evaluators.

Compensation: $20 Per Interview

Services & Cost Study

It is the region’s responsibility to work with their evaluation coordinator to provide services and costs data for all young adults participating in the National Study, so young adults and their caregivers (or adult allies) do not have additional steps to take for this study.
Compensation: None[/spoiler] [spoiler title=”How does our region and our YIT youth in our program benefit from the three levels of research which will take place?” open=”0″ style=”1″]As for direct benefits to young adults and their family members or adult allies from participating in the evaluation, there are no direct benefits to them for being part of this study. At times, participants report a sense of satisfaction at helping to inform the process of developing a system of care, which the evaluation helps to do. Young adults and their family members/adult allies may benefit from the services they receive. They may also learn new things about themselves and the service system. As a result of this project, services for young adults with mental health and/or related needs and their families may get better.

As for benefits to the region, our goal is to be able to provide ongoing reports to regions and the state regarding the implementation of the system of care strategies. So, ideally, we would be able to give your region feedback about services utilized, satisfaction with services, and whether young adults’ functioning improves while participating in the system of care. Also, depending on what your region specifies for indicators, if the data to measure those indicators are something we already collect, we can work with you to look at these indicators. Lastly, our plan is to help regions prepare their financial reports required of the YIT grant using the services and costs data they are expected to report to us for the evaluation.

There may be other benefits that we learn of as we move along but these are the most obvious and immediate.[/spoiler] [spoiler title=”What is expected of regional steering committees?” open=”0″ style=”1″]The overall expectation was stated in the original Invitation to Communities, “the Steering Committee and LIT will also participate in the required Youth in Transition evaluation and must designate an evaluation contact person” (page 7).

[highlight bg=”#E85C2C” color=”#fff”]Perhaps the most important expectation is that your region needs to understand which young adults are served by your SAMHSA funded system of care, because this is the target population for the evaluation.[/highlight] Because there are 12 different regional plans, we do not expect the populations will be the same across all regions. For example, the Designated Mental Health Agency may be the primary “door” into one region’s system of care while a Youth Service Bureau may be another regions primary “door”. Some regions may have many doors (e.g., Education, Mental Health, Vocational Rehabilitation, Corrections).

Specific responsibilities of the regional Steering Committees and their evaluation liaisons.

table-2
[/spoiler] [spoiler title=”What is expected of fiscal and information technology personnel assisting regions with their YIT system of care plans?” open=”0″ style=”1″] The Services and Costs component to the YIT grant requires event-level documentation of all service expenditures for young adults participating in the National Study. This includes individual documentation of services and uses of flexible funds for all young adults. SAMHSA requires specific information to be collected, so if it cannot be gathered from existing data sources in the region we will need to use the forms provided by the National Study.
[/spoiler] [spoiler title=”How is the evaluation young adult driven and family guided?” open=”0″ style=”1″] There are three ways that we are striving for a young adult driven and family guided evaluation:

Collaboration with young adults and family partners on the YIT Operations and Outreach Team.
Forming an evaluation advisory group (E-Team) made up of young adults, family members, and adult allies.

Using a strengths-based measure developed and refined through young adult input in Vermont.
As the evaluation continues, we will seek young adult and family guidance on how best to analyze and share the data. We encourage any questions and suggestions from young adults and families in your region. In addition, if anyone would like to join the E-Team they should contact our evaluation team
[/spoiler] [spoiler title=”How are DMH’s roll out of the ASEBA system and the YIT grant related?” open=”0″ style=”1″]As many or most of you know, the Department of Mental Health’s (DMH) Child, Adolescent, and Family Unit is implementing a statewide roll-out of the ASEBA family of measures. DMH’s expectation is that all providers within the designated and specialized service agencies (DA/SSA) will administer the ASEBA forms with their clients at intake and every 6 months thereafter for as long as the client and their family are enrolled in services. One priority area for this rollout is young adults, ages 16 to 21, as this population overlaps with the YIT grant.

The YIT grant, in order to sustain its funding over the next 5 years, has to meet several requirements across a number of areas, with evaluation being one of the key areas. One of the evaluation requirements is to obtain ASEBA forms from young adults and their family members/adult allies at their intake or re-entry into services and every 6 months thereafter for up to 2 years. Participation in this evaluation is voluntary on the part of young adult clients and their families/adult allies.

To reduce the burden on clients and family members/adult allies who choose to participate in the evaluation, we plan to have them sign a release form, allowing the University of Vermont’s VCHIP evaluation team to obtain the ASEBA data from the DMH ASEBA database. Thus, it is critical providers within the DA/SSA’s who are working with clients ages 16 to 21 complete the ASEBA forms as indicated by DMH.

The VCHIP evaluation team is working with DMH to provide technical assistance and support in the implementation of the ASEBA system within your community’s DA/SSA. In addition, with the ASEBA data we will be able to provide data back to your region which will help to provide a sense of how young adults, ages 16 to 21, are benefitting from the services they receive. Such information can help to generate community and statewide, including legislative support for such services.[/spoiler] [spoiler title=”I’m filling out the Common Study and I’m not sure about…” open=”0″ style=”1″]

Do I need to complete the Study Inclusion Criteria Checklist (SICC) for all young adults or only those eligible for the Common Study?

You should complete a SICC for each young adult that you plan to provide YIT funded services or strategies in your region. This may mean that some of the young adults you complete a SICC for are not eligible for the Common Study because they are: (a) out of the age range (16-21), (b) not eligible for or receiving cross-agency case management and individualized services planning, or (c) not eligible for or receiving YIT funded services or strategies. For these young adults please complete the SICC and send to VCHIP. For young adults who are eligible for the Common Study, you should seek their consent to participate.

There are some situations in which you do not need to complete a SICC for a young adult. For example, several regions are providing street outreach and may only have a single (and anonymous) contact with a young adult. In this situation you would not need to complete the SICC. However if you began to work with the young adult further (e.g., conducting an intake, developing a plan) you need to complete the SICC. If you are unsure if you need to complete a SICC, please contact the evaluator working with your region.

How soon do I need to have the Common Study Intake complete?

Starting with a young adult’s intake into your region’s system of care, you have 7 calendar days (sorry not business days!) to complete the SICC and the Common Study Intake. Because this is such a short window of time there are several important things to consider:

Make sure you’re clear what date is considered the intake. In some regions, service providers meet first to determine the young adult’s eligibility and complete the SICC. If you’re not meeting with the young adult at that time that is probably not the intake date. The intake date is the first time you meet with the young adult to talk about their goals and plan (see explanation of different dates below).
You may want to complete the Common Study Intake form during your first meeting with the young adult, so you don’t have to schedule another meeting with them within 7 days.

Can you explain what the difference is between all the dates requested on the Common Study forms?

table-3

What to do if I don’t know or have access to all the answers on the Common Study?

Please complete the Common Study forms to the best of your knowledge following instructions on what information must be asked of the young adults directly and what information can come from other sources. Because there is a short time frame for completing the Common Study Intake, you may need to send it to us without answering all questions. For example, many regions have told us they know the young adult has a diagnosis, but they have not been able to get a definitive answer on it within the 7-day window. In situations like this, please send us the Common Study forms and let us know that you can send us the missing information later. It is definitely better to send us an incomplete response that can be filled in later than to miss the 7-day window or to guess.

On what forms do I need to get permission from a legal guardian?

n order to administer any of the Common Study forms, you must first obtain permission from the young adult (if 18 years or older) or their legal guardian (if they are younger than 18 years). In addition, you need their permission to submit the May we call you? forms for the evaluators to contact them about the Vermont and National Studies.

There are different ways of obtaining permission depending on the age of the young adult:

table-4
You do not need permission to complete and send the Study Inclusion Criteria Checklist.

For the first question in Section 2 of the Common Study Intake, do I need to circle all the problems the young adult is struggling with?

No. This question is asking for you to circle those problems that directly led to their referral to your region’s YIT funded services and strategies.

How should we send the Common Study forms to VCHIP?

Please send us the SICC, appropriate consent form, Common Study Intake, and May we call you? form as soon as you complete them within the 7-day window. The forms should be sent together by fax or mail (contact info below), but please note that we need an original copy of the consent form on file. Please do not separate the forms for an individual youth. If you must send them separately, contact us to let us know as you are sending them.

802-656-8368 (attn. Ellen Talbert)
Ellen Talbert
UVM VCHIP
UHC St. Joseph 7
1 South Prospect St.
Burlington, VT. 05401
[/spoiler] [spoiler title=”Who can I contact if I want to know more or have questions?” open=”0″ style=”1″]The VCHIP evaluation team (Tom Delaney, Jody Kamon, & Jesse Suter) is happy to speak with your region to review the responsibilities, benefits, and logistics of the evaluation. Each region has a member of the evaluation team as their primary contact (see below). We want to give you as much information as soon as possible to help in your planning. Please let us know how we can work together with you to make the evaluation effort a successful and smooth one for your region.