A.6. Safeguards
A.6.1 Multi-Layered Output Verification
Run a lightweight validation pipeline to ensure outputs are as desired. Some examples of what can be included in this:
- Check that dates conform to
ISO-8601 - Verify that entity types match your controlled vocabulary (e.g., if the model outputs an unexpected label, flag for manual review)
- Deploy a "sanity-check" function call to a smaller, cheaper model to verify the consistency of outputs (for example, “Does this statement parse correctly as a Fact? Yes/No.”)
A.6.2. Audit Logging & Monitoring
- Implement structured logging with configurable verbosity levels (e.g., debug, info, warn, error)
- Store input pre-processing steps, intermediate outputs, and final results with full tracing, such as that offered via OpenAI's tracing
- Track token throughput, latency, and error rates
- Monitor data quality metrics where possible, such as document or statement coverage, temporal resolution rates, and more
- Measure business-related metrics such as user numbers, average message volume, and user satisfaction