Set up human spot checks
Spot Check Sampling
Spot check sampling lets you route a configurable percentage of submissions for manual review before they are exported. This is useful during tenant onboarding when extraction quality needs to be validated, and can be gradually reduced as confidence grows.
Configuration
Spot checking is configured per tenant in the organization settings. The configuration is a JSON object added to the tenant's integrations array.
Example
{
"type": "spot-check",
"enabled": true,
"sampleSize": 50,
"spotCheckerEmails": [
"[email protected]",
"[email protected]"
],
"templateOverrides": [
{ "templateId": 3, "sampleSize": 50 }
]
}How it works
When a submission is fully processed and ready for export, the system rolls a random number against the configured sample rate. If selected:
The submission is not exported automatically
It stays in "Ready for Spot Check" status in the dashboard
All configured spot checkers receive a notification email
A reviewer opens the submission, assigns it to themselves, and reviews the extracted data
The reviewer clicks Mark as completed — this exports the submission, sets the status to "Completed", and sends the end user a confirmation email
The end user receives a confirmation email that their submission was processed
Submissions that are not selected continue through the normal automatic export flow with no change.
Settings reference
enabled
boolean
Yes
Enables or disables spot checking for the tenant. Set to false to disable without removing the configuration.
sampleSize
integer (0-100)
Yes
Percentage of submissions routed to spot check. 0 = no submissions selected (effectively disabled). 100 = every submission is selected.
spotCheckerEmails
array of strings
Yes
Email addresses that receive a notification when a submission is selected. Supports multiple addresses. Can include team inboxes or Linear triage email addresses.
templateOverrides
array of objects
No
Optional per-template overrides for the sample rate.
templateOverrides[].templateId
integer
Yes (within override)
The ID of the template to override.
templateOverrides[].sampleSize
integer (0-100)
Yes (within override)
The sample rate for this specific template. Takes precedence over the tenant-level sampleSize.
Template overrides
You can set different sample rates for different templates. For example, a tenant might need 100% review on a new template while an established one only needs 10%:
In this example, template 7 has every submission reviewed, while all other templates are sampled at 10%.
Enabling spot checking for a tenant
To enable spot checking for your tenant, contact the Nora team with:
The email addresses that should receive spot check notifications
The desired sample size
Any per-template overrides if needed
Tip: Start with
sampleSize: 100during initial onboarding so every submission is reviewed. Reduce gradually as extraction quality is validated (e.g. 100 → 50 → 10 → 0).
Disabling spot checking
Either set enabled to false or set sampleSize to 0. Both stop new submissions from being selected, but existing spot check candidates in the queue are not affected.
Reviewer workflow
1. Receive notification
When a submission is selected for spot check, each email address in spotCheckerEmails receives an email with the submission name and number.
2. Open the submission
In the FormFlow dashboard, spot-checked submissions are displayed with an orange "Ready for Spot Check" status badge, making them easy to identify in the submissions list.
3. Assign and review
Open the submission and assign it to yourself. The status changes to "In Spot Check". Review the extracted data against the source documents to verify accuracy.
4. Complete the review
When satisfied with the data, click Mark as completed. This exports the submission, delivers a confirmation email to the end user, and sets the status to "Completed".
End user experience
The end user (the person who submitted the original documents) is not aware that their submission was selected for spot check. From their perspective:
They submit documents as usual
They receive the same confirmation email they would normally receive
The only difference is a potential delay while the reviewer checks the data
Recommended ramp-down schedule
Initial onboarding
100
First 1-2 weeks — review everything
Building confidence
50
Weeks 2-4 — spot check half
Monitoring
10-20
Ongoing — periodic quality checks
Fully automated
0
When confident in extraction quality
Adjust timelines based on submission volume and error rates observed during review.
Last updated
Was this helpful?