Ctrlk

Set up human spot checks

Spot Check Sampling

Spot check sampling lets you route a configurable percentage of submissions for manual review before they are exported. This is useful during tenant onboarding when extraction quality needs to be validated, and can be gradually reduced as confidence grows.

Configuration

Spot checking is configured per tenant in the organization settings. The configuration is a JSON object added to the tenant's integrations array.

Example

{
  "type": "spot-check",
  "enabled": true,
  "sampleSize": 50,
  "spotCheckerEmails": [
    "[email protected]",
    "[email protected]"
  ],
  "templateOverrides": [
    { "templateId": 3, "sampleSize": 50 }
  ]
}

How it works

When a submission is fully processed and ready for export, the system rolls a random number against the configured sample rate. If selected:

  1. The submission is not exported automatically

  2. It stays in "Ready for Spot Check" status in the dashboard

  3. All configured spot checkers receive a notification email

  4. A reviewer opens the submission, assigns it to themselves, and reviews the extracted data

  5. The reviewer clicks Mark as completed — this exports the submission, sets the status to "Completed", and sends the end user a confirmation email

  6. The end user receives a confirmation email that their submission was processed

Submissions that are not selected continue through the normal automatic export flow with no change.

Settings reference

Setting
Type
Required
Description

enabled

boolean

Yes

Enables or disables spot checking for the tenant. Set to false to disable without removing the configuration.

sampleSize

integer (0-100)

Yes

Percentage of submissions routed to spot check. 0 = no submissions selected (effectively disabled). 100 = every submission is selected.

spotCheckerEmails

array of strings

Yes

Email addresses that receive a notification when a submission is selected. Supports multiple addresses. Can include team inboxes or Linear triage email addresses.

templateOverrides

array of objects

No

Optional per-template overrides for the sample rate.

templateOverrides[].templateId

integer

Yes (within override)

The ID of the template to override.

templateOverrides[].sampleSize

integer (0-100)

Yes (within override)

The sample rate for this specific template. Takes precedence over the tenant-level sampleSize.

Template overrides

You can set different sample rates for different templates. For example, a tenant might need 100% review on a new template while an established one only needs 10%:

In this example, template 7 has every submission reviewed, while all other templates are sampled at 10%.

Enabling spot checking for a tenant

To enable spot checking for your tenant, contact the Nora team with:

  • The email addresses that should receive spot check notifications

  • The desired sample size

  • Any per-template overrides if needed

Tip: Start with sampleSize: 100 during initial onboarding so every submission is reviewed. Reduce gradually as extraction quality is validated (e.g. 100 → 50 → 10 → 0).

Disabling spot checking

Either set enabled to false or set sampleSize to 0. Both stop new submissions from being selected, but existing spot check candidates in the queue are not affected.

Reviewer workflow

1. Receive notification

When a submission is selected for spot check, each email address in spotCheckerEmails receives an email with the submission name and number.

2. Open the submission

In the FormFlow dashboard, spot-checked submissions are displayed with an orange "Ready for Spot Check" status badge, making them easy to identify in the submissions list.

3. Assign and review

Open the submission and assign it to yourself. The status changes to "In Spot Check". Review the extracted data against the source documents to verify accuracy.

4. Complete the review

When satisfied with the data, click Mark as completed. This exports the submission, delivers a confirmation email to the end user, and sets the status to "Completed".

End user experience

The end user (the person who submitted the original documents) is not aware that their submission was selected for spot check. From their perspective:

  • They submit documents as usual

  • They receive the same confirmation email they would normally receive

  • The only difference is a potential delay while the reviewer checks the data

Recommended ramp-down schedule

Phase
sampleSize
When

Initial onboarding

100

First 1-2 weeks — review everything

Building confidence

50

Weeks 2-4 — spot check half

Monitoring

10-20

Ongoing — periodic quality checks

Fully automated

0

When confident in extraction quality

Adjust timelines based on submission volume and error rates observed during review.

Last updated

Was this helpful?