Using ChatGPT to Generate Contextual Test Data

Why realistic test data is difficult

A useful QA dataset must satisfy more than column types. It needs valid relationships, believable distributions, lifecycle history, and deliberate exceptions. A customer record may require an active account, an eligible product, a sequence of transactions, and a support case whose timestamps occur in the right order. Random strings rarely exercise those rules, while copies of production data create privacy and security risk.

Large language models can help draft synthetic records because they can interpret written business rules and produce varied examples. They should not be treated as a database constraint engine or given confidential production records. The safest pattern is to let the model propose candidates, then use deterministic code to validate, repair, and load them.

A repeatable generation workflow

1. Minimize the input. Provide table and field names, allowed values, relationship rules, and fictional examples. Remove credentials, personal data, customer text, and proprietary values before any prompt is sent.
2. Define a machine-readable contract. Ask for JSON that conforms to a JSON Schema, or rows that match an explicit CSV header. State nullability, formats, ranges, uniqueness requirements, and foreign-key dependencies.
3. Specify distributions and scenarios. Describe the mix you need: common journeys, rare states, geographic or product segments, and invalid records intended to test rejection paths. Personas are useful only when translated into observable field values.
4. Generate in dependency order. Create parent entities first, assign stable synthetic identifiers, and pass only those identifiers into prompts for child records.
5. Validate outside the model. Parse the response, reject unknown fields, check constraints, run referential-integrity tests, scan for accidental personal information, and load through the same API or migration path used by the application.

Example prompt contract

Generate 25 fictional orders as JSON.
Required fields: order_id, customer_id, status, created_at, total.
customer_id must come from: C1001, C1002, C1003.
status must be pending, paid, shipped, or cancelled.
created_at must be ISO 8601 and shipped orders must be at least one hour old.
total must equal the sum of the supplied line items.
Return JSON only; do not invent additional fields.

This contract still needs automated assertions. For example, a test can recompute every total, confirm IDs are unique, and verify that a cancelled order has no shipment. Save the prompt, model identifier, generation settings, and validator version so a failed fixture can be reproduced.

Design edge cases deliberately

Do not ask for “some edge cases” and hope the model finds the important ones. Build a coverage matrix from the product's rules: minimum and maximum lengths, time-zone boundaries, leap days, duplicate submissions, Unicode names, missing optional fields, authorization boundaries, and conflicting lifecycle events. Tag each generated record with its intended scenario in a separate test manifest so the suite can prove that every case was exercised.

Privacy, security, and reliability caveats

• Use synthetic inputs and an approved model environment. Never paste production rows into a consumer chat tool.
• Treat generated content as untrusted input. Escape it, scan it, and prevent generated SQL from being executed directly.
• Keep deterministic factories for small unit tests. LLM generation is better suited to broader integration, exploratory, and performance datasets.
• Measure usefulness by defects found, rule coverage, reproducibility, and maintenance effort—not by how human the names sound.

Further reading

The NIST AI Risk Management Framework offers a practical structure for governing AI risks, and the OWASP Top 10 for LLM Applications covers threats such as prompt injection and insecure output handling. Apply those controls to the generation pipeline just as you would to any other system that produces executable or stored data.