# PiP Pilot Measurement and Referral Outcome Framework

**Research cut-off:** 4 August 2026
**Status:** Public-source evaluation framework
**Applies to:** village, HCSS/community-provider and media-to-digital pilots

## Measurement hierarchy

1. **Primary:** reach, task completion, attendance, retention, referral closure and workload.
2. **Secondary:** resident/client experience, implementation feasibility and digital inclusion.
3. **Exploratory:** loneliness or social-connection change.

An eight-week pilot can test feasibility and produce directional outcome signals. It cannot establish that PiP reduced loneliness, prevented residential care or improved health.

## Evidence boundary

The national HCSS survey is a standardised patient-experience survey. PiP feedback, SNZ reviews, satisfaction ratings and testimonials are not PREMs or PROMs merely because consumers supplied them. The 2025 HCSS survey added EQ-5D-5L as a formal PROM; that does not convert its experience questions or PiP measures into PROMs.

New Zealand's Digital Inclusion Outcomes Framework requires motivation, access, skills and trust. Account activation alone is not evidence of digital inclusion.

Feasibility pilots should focus on recruitment, retention, delivery, measurement and implementation—not formal effectiveness testing.

## Frozen metric dictionary

Definitions and denominators should be signed off before baseline collection.

### Participation funnel

- **Eligible:** meets the agreed pilot criteria.
- **Contactable:** has an approved contact channel.
- **Invited:** actually offered participation.
- **Consented:** gave informed pilot/evaluation consent.
- **Activated:** completed a meaningful first task, not merely account creation.
- **Used:** completed a discovery, booking, attendance or referral action.
- **Follow-up complete:** completed the endpoint measure.
- **Paired:** has linkable baseline and endpoint responses.

Report recruitment, activation, reach, active use, retention and paired-outcome availability with both numerator and denominator.

### Events and activities

- unique attendance reach;
- booking completion;
- capacity use;
- repeat participation;
- cancellation and no-show rates with reasons;
- channel used: app, phone, reception, paper or staff-assisted.

Repeat participation is evidence of engagement, not proof of improved social connection.

### Community referrals

Track every stage:

`need identified → offer made → accepted → sent → acknowledged → contacted → scheduled/accepted → attended/completed → outcome notified`

Report offer acceptance, partner acknowledgement, completion, terminal failure, overdue-open cases and median time to acknowledgement, contact and completion.

Do not combine overdue-open and terminal failures. A referral is closed only when completed or explicitly resolved with a reason and the person/referrer has been told the outcome.

Minimum reason codes:

- could not contact;
- person declined or withdrew;
- ineligible or wrong service;
- insufficient information;
- provider capacity or waitlist;
- transport, accessibility, language or digital barrier;
- appointment missed;
- duplicate or invalid referral;
- consent/privacy prevented transfer;
- other, reviewed.

### Digital inclusion and usability

Use all invited people as the denominator, not app users alone. Capture device/connectivity access, accessibility needs, independent and assisted task completion, failure/abandonment, time and errors, help requests, reason for non-use, channel and brief confidence/trust questions.

Preserve non-digital routes throughout the pilot.

### Workload

Record separately for staff, resident committees, volunteers and community partners:

- setup and training hours;
- weekly administration minutes;
- support contacts per 100 active users;
- minutes per booking/referral;
- duplicate entries and manual follow-ups;
- unresolved tasks, corrections, complaints and privacy incidents;
- outage workaround time.

Recurring minutes per completed booking/referral is useful, but one-off setup cost must remain visible.

## Instrument choices

Select one loneliness instrument only if the evaluation genuinely expects that construct to move.

| Instrument                          | Suitable use                                     | Boundary                                                                                              |
| ----------------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| De Jong Gierveld six-item scale     | Separate social and emotional loneliness signals | NZ use does not prove cultural validation, eight-week responsiveness or a meaningful-change threshold |
| Three-Item Loneliness Scale         | Lower respondent burden                          | Does not distinguish social and emotional loneliness                                                  |
| LSNS-6                              | Network size/support or isolation screening      | Measures network structure rather than felt loneliness                                                |
| Single Ease Question                | Immediate task-ease rating after observed tasks  | Must be paired with completion, assistance, time and errors                                           |
| Feasibility of Intervention Measure | Staff/partner feasibility at week 2 and endpoint | Not a default resident-experience measure                                                             |

Do not administer multiple loneliness scales in a small pilot. Do not use EQ-5D-5L without registration/licensing, an evaluation rationale and appropriate expertise.

## Route-specific design

### Selwyn Village

- Four-week baseline and eight-week live pilot.
- Independent-living activities only in phase one.
- App and non-digital channels run in parallel.
- Use identical activity, attendance and workload definitions before and after.
- Measure discovery-to-booking, booking-to-attendance, unique/repeat attendance, first-task success, event-change communication, committee/reception workload and non-digital completion.
- Involve the Independent Residents' Committee in definitions, burden and interpretation.

### HCSS/community-provider pilot

- Keep zero-integration and non-clinical.
- Measure the full referral funnel, reasons, time, ability to attend and partner workload.
- Use exact national HCSS questions only where methodology and reuse permission allow; otherwise describe questions as aligned with HCSS experience themes.
- Use short staff/partner feasibility measures and qualitative interviews to explain failures and burden.
- Do not claim results are formal national HCSS reporting without written confirmation.

### Active Seniors

Use separate QR codes and short URLs for advert, editorial, scorecard and later awards promotion.

Track:

`distributed copies → unique visits → starts → completions → result viewed → consented contact → qualified opportunity`

Report responses per 1,000 verified delivered copies. Distribution is not exposure, and scans are not reader-reach conversion. Use task completion plus a short ease question for the scorecard journey; do not use loneliness or health-outcome measures.

## Analysis and claim rules

- For fewer than 30 observations, show count, denominator and percentage.
- Use median, interquartile range and range for skewed time/workload data.
- Show paired trajectories or median paired change with paired sample size.
- Do not use p-values or “statistically significant” language in a tiny feasibility pilot.
- Missing follow-up is attrition, not improvement.
- Record self, proxy and assisted completion separately.
- Avoid small outward-facing demographic cells that risk re-identification.
- Qualitative comments explain mechanisms; they do not replace denominators.
- Use “observed,” “associated with” and “directional signal.”
- A later causal evaluation needs a comparator, wait-list, matched site or staggered rollout with adequate duration and power.

## Illustrative progression gates

Thresholds must be co-designed. Consider progression only if:

- core funnel completeness is at least 80%;
- no serious privacy or safety incident occurs;
- non-digital access remains available and measured;
- core tasks succeed independently or through an acceptable assisted path;
- referral failures and overdue cases have named owners and reasons;
- recurring workload is acceptable to operators and partners;
- residents/clients and staff judge the model feasible enough for a larger evaluation.

## Source register

- Ministry of Health — National Framework for HCSS: https://www.health.govt.nz/system/files/2020-06/national_framework_for_home_and_community_support_services_29sept2020.pdf
- HQSC — Patient experience survey methodology: https://www.hqsc.govt.nz/our-data/patient-reported-measures/patient-experience/survey-information-and-methodology/
- HQSC — HCSS survey methodology: https://www.hqsc.govt.nz/resources/resource-library/hcss-experience-survey-methodology-and-procedures/
- HQSC — HCSS national results 2025: https://www.hqsc.govt.nz/assets/Our-data/Publications-resources/Surveys/HCSS-Survey-Report-2026.pdf
- HQSC — Patient-reported measures: https://www.hqsc.govt.nz/our-data/patient-reported-measures/
- Digital.govt.nz — Digital Inclusion Outcomes Framework: https://www.digital.govt.nz/dmsdocument/167~digital-inclusion-outcomes-framework/html
- Stats NZ — Loneliness in New Zealand: https://www.stats.govt.nz/assets/Uploads/Retirement-of-archive-website-project-files/Reports/Loneliness-in-New-Zealand-Findings-from-the-2010-NZ-General-Social-Survey/loneliness-in-nz-2010-23apr2013.pdf
- CONSORT extension for pilot and feasibility trials: https://www.bmj.com/content/355/bmj.i5239
- MRC complex-intervention framework: https://www.bmj.com/content/374/bmj.n2061
- De Jong Gierveld six-item scale validation: https://doi.org/10.1177/0164027506289723
- Three-Item Loneliness Scale: https://pmc.ncbi.nlm.nih.gov/articles/PMC2394670/
- LSNS-6 validation: https://doi.org/10.1093/geront/46.4.503
- Feasibility of Intervention Measure validation: https://pmc.ncbi.nlm.nih.gov/articles/PMC5576104/

## Research classification

- **Validated instrument:** supported by published methodology, within its stated use boundary.
- **Pilot operational measure:** pre-defined event, funnel, workload or usability indicator.
- **Strategic gate:** proposed progression threshold requiring partner agreement.
- **Exploratory outcome:** signal unsuitable for causal or health-impact claims in a small pilot.