CAPE Development Background
In January of 2025, the Pennsylvania Department of Aging (the Department) rolled out the Comprehensive Aging Performance Evaluation, or CAPE, an innovative tool designed to boost transparency and accountability of Pennsylvania’s Area Agency on Aging (AAA) network, which provides a host of services to older adults. In April 2025, the Department began publicly posting performance results on its website for the AAAs monitored under CAPE. Publicly posting results to the website was a significant milestone for the Department and an unprecedented action toward public transparency and accountability. Performance results are now regularly posted to the Department’s website. The public is now able to see how well their local AAA is performing in major program areas, such as investigative activities related to suspected elder abuse and helping older adults at home with tasks of daily living.
The Department’s past monitoring approach was in need of change. As Secretary of Aging Jason Kavulich stated, “...one of the Shapiro Administration’s top priorities has been to modernize how we oversee AAA performance at the local level and make those results easily available to the public. The completion of CAPE – the first major overhaul and upgrade of the Department’s monitoring system in decades – is a huge achievement and monumental leap forward for this agency. We are transforming our ability to evaluate and improve services for older Pennsylvanians.” CAPE transforms the Department’s prior monitoring approach into a more holistic philosophy, where each of the 52 AAAs is evaluated using a set of performance metrics across multiple program areas during a singular monitoring review. CAPE’s review includes Older Adult Protective Services, the OPTIONS program, the Caregiver Support Program, and has a fiscal component.
Prior to CAPE
Prior to CAPE, the Department used a variety of monitoring tools in its evaluation of Older Adult Protective Services. These historical tools, and the prior monitoring approach in general, is commonly referred to as the “legacy” system. The legacy monitoring system’s Protective Services component received input and feedback on needed areas for improvement from stakeholder groups as well as independent auditing entities and was in need of a major overhaul.
Internal Control Assessment
The Department conducted an internal control assessment to identify inconsistencies and outdated information pertaining to the legacy Protective Services monitoring tool and associated processes. This internal control assessment identified key areas of improvement for quality assurance monitoring.
For example, some of the legacy system’s documented internal procedures were in draft form and incomplete during the legacy period. Additionally, legacy procedures were not consistently used and were not always referenced as part of the onboarding and training process for new PDA staff. The legacy tools did not stay current with citations and legal references, which contributed to inconsistent application and interpretation of performance measures. This variance in the internal compliance with procedures by Protective Services staff who were conducting AAA Protective Services monitoring led to inconsistent findings and corrective action plans with each of the 52 AAAs.
In the development of CAPE, the Department addressed the legacy issues by documenting procedures, as well as documenting detailed reviewer instructions and all regulatory and policy citations. All staff now adhere to CAPE procedures and the monitoring process now requires peer review, which promotes interrater reliability when evaluating AAAs.
Key Improvement Highlights
There were multiple areas for further efficiency that were identified and further developed through the creation and roll out of the CAPE.
The monitoring tool that is used to evaluate the AAA’s performance is locked to prevent edits and promote version control. There is one master file located within a documented and secured location, and all staff must use the most recent version of the tool during monitoring. This version control process is validated and verified by the supervisor and multiple staff who are completing the review. The master tool is password protected and end users are not able to make changes to the underlying tool and related formulas. These controls did not exist with the legacy tools.
Any changes that are needed to the tool (updated legal citations, reviewer instructions, spelling, etc.) are documented in a change control log. This change log documents when a change was made, who made the change, and the reason for the change. All changes are documented with supervisory signoff. This ensures appropriate version control and historically identifies changes that were made. For example, if a policy is updated and the citation changes, this would need to be updated and noted in the reviewer instructions. The change would be reviewed, approved, noted in the change log, and the tool would be updated. Once the tool is updated, the older versions are archived and the newest version is made available for upcoming monitoring reviews.
Performance Measures, legal citations, and reviewer instructions are embedded within the tool for version control. This means that the reviewers always have the most recent version of the tool, legal citations, and reviewer instructions. In the legacy process, the reviewer instructions were in draft form, incomplete, and stored in a separate document. The legacy process caused staff to inconsistently adhere to reviewer instructions, use outdated materials, and ultimately provide varying results.
Interrater reliability is a cornerstone of CAPE. Interrater reliability is achieved through two methods: objective measures based on legal citation/policy, and peer-reviewed results. CAPE evolved some of the legacy system’s measures into objective, grounded performance measures. For example, the legacy protective services tool evaluated whether an older adult was “left at risk.” This legacy measure allowed a reviewer’s thoughts, beliefs, and/or feelings to influence the result. CAPE transformed this measure into “Documentation supports all identified risk were eliminated, mitigated, or reduced.” This improved CAPE measure relies on reviewers locating the supportive information within the record that address risks to the older adult, rather than rely on the reviewer’s subjective interpretation of potential outstanding risks. Additionally, through CAPE, teams of individuals are assigned records to review and consensus meetings are held to ensure that staff are achieving consistent results. If there is a variation in whether a record is meeting the criteria to be considered compliant, then several steps are taken which include:
- staff discuss their findings, and if a mutual conclusion cannot be reached;
- then the supervisor is involved, and if a mutual decision still cannot be made;
- then the appropriate bureau (for example the Bureau of Protective Services) would be engaged for additional review. Bureau staff typically have the final say in whether a record is compliant, because they will ultimately be the individuals responsible for providing technical assistance to the AAA and providing training to correct non-compliance.
In addition, prior to any findings being formally issued to a AAA, internal Department meetings are held between the program office and the monitoring staff to ensure consensus.
This entire process has been put in place to ensure that records are consistently scored and results are objective and rooted in policy and regulations. When a record is determined to be non-compliant, it has been thoroughly vetted and agreed upon by Department staff.
Category score vs overall score. Previously in the legacy tool one, overall score was assigned to an agency indicating whether they passed or failed their monitoring. This overall score combined all metrics evaluated for the AAA. During development of the CAPE, categories were developed, which grouped certain areas of performance measures together. This was designed so the general public would have a better idea of how their local AAA is performing in specific areas. It also helps the monitored AAA understand which area(s) needs improvement and assists the Department in seeing trends and identifying specific trainings to help AAAs improve on their work.
Some of these categories cross over to different program areas (for example “Documentation Requirements,” “Data Management,” and “Administrative Oversight”). If an AAA is struggling in one of their program area categories, but not in the other similar program area category, the category scores could identify best practices within the agency. Or, if the AAA is struggling in both program areas, the category scores may be identifying systemic issues. In either instance, the Department is ready to provide Technical Assistance and training to the AAAs for any areas of non-compliance. Additionally, if any one performance measure is under 85% compliance, then a Performance Improvement Plan (PIP) is assigned to the AAA.
Consistent standards and thresholds for expectations is a key focus of CAPE. CAPE has clearly identified thresholds for compliance at 75% for category scores and 85% for each individual performance measure – which has created a standard for program evaluation.
CAPE is administered consistently for every AAA, which means that trends are able to be identified. This level of consistency and the ability to analyze data is a key advancement in our monitoring system capabilities.
The CAPE monitoring schedule is publicly posted and adhered to for each cycle of CAPE. A cycle is completed once all 52 area agencies on aging (AAAs) are monitored. Historically the monitoring cycle was not shared and was not publicly available. Additionally, the legacy system did not consistently ensure each AAA was evaluated during review periods. In some instances, some AAAs were monitored many times over a two or more year period, while others may have only been monitored once.
Because CAPE takes a consistent streamlined approach to monitoring across all 52 AAAs, trends are able to be identified, and training is able to be tailored to the AAA network.
Performance Improvement Plan (PIP) thresholds are established and adhered to consistently. If any one Performance Measure is found to be non-compliant (less than 85%), then the AAA must complete a PIP for that measure.
A set number of non-compliant measures requires technical assistance, and a higher threshold requires leadership meetings. For instance, any Protective Services Program that has 10 or more non-compliant Performance Measures is required to receive mandatory Technical Assistance from the Department. And similarly, for the OPTIONS/CSP program, if any agency has 14 or more non-compliant Performance Measures, that AAA is required to receive mandatory technical assistance from the Department. Mandatory Technical Assistance is put in place to ensure that the AAA is receiving direction and support from the Department to fully understand the areas of non-compliance and to confirm that the Performance Improvement Plans are targeting the root cause of non-compliance. The Department’s technical assistance is designed to improve AAA performance during subsequent CAPE monitoring cycle(s). Additionally, if 25 performance measures are non-compliant between both the Protective Services program and the OPTIONS/CPS program, the Department sends a formal letter to the AAA letting them know that they are required to have a leadership discussion with the Department regarding the AAA’s overall performance. This is designed as a higher level of accountability, which will promote consistent practice and encourage change and remedial action at the AAA level.
For the first time – the Department’s block grant agreement with the 52 AAAs has accountability language that is tied back to their performance in the CAPE monitoring. This formal documentation adds a level of additional accountability and oversight. It also adds consistency to the evaluation of each AAA’s performance.
IIn development of CAPE, sample size was another key improvement area. In the legacy process, sample size logic existed; however, it was often up to the reviewer to determine which cases to include, the number of cases for the sample size, and whether additional cases should be reviewed. CAPE identifies specific sample size logic that is adhered to for all AAAs. A minimum number of records to be reviewed for every record type is identified. This improvement promotes consistency and standardization to monitoring practices.
During the legacy process, record keeping was not consistent. In CAPE, not only are final results of category and performance measure records stored in specified locations, they are also made publicly available. This creates greater accountability and consistency for the AAAs and for the Department.
Ultimately, CAPE has introduced a new level of accountability and transparency. The Department is holding itself accountable via public data regarding monitoring cycles, standards, and performance measures. The AAA network is held accountable through public data, standardized performance measures, and enhanced performance expectations in their individual grant agreements. Transparency is evident through continued dialogue and information and performance outcomes being continuously added to the Department’s website. The goal is to make improvements to the AAA network and thereby making improvements to the lives, safety and well-being of older adults across Pennsylvania.
The CAPE was designed to evaluate multiple program areas within an AAA. Historically, these multiple program areas were evaluated independently, on different schedules and with inconsistent methodology. Now, under CAPE the Department has made the review more efficient by streamlining the approach and minimizing disruption to the AAA’s provision of services. The CAPE was also designed to be modular in nature, meaning additional programmatic areas of review can be (and are anticipated to be) appended to the CAPE making an even more robust and meaningful evaluation.