We first noticed the question over the last Christmas break, when our agents performed better even though no new model had been released. If quality can shift while the model version stays the same, changing service conditions may help explain why. Time-of-use pricing, like DeepSeek’s, is one way to manage demand; quality of service is another.

We are a 24×7 company, but most of our benchmarking happens on weekends while feature development happens on working days. We wanted to know whether weekend runs gave us a skewed picture of the performance customers experience during the workweek. We measured in Pacific Time, whose business hours overlap with peak hours in Eastern Time, so the time windows may reflect demand across both coasts. To compare like with like, we used the same aidnn test suite as in our earlier post, “Upgrading the Brain Behind aidnn.”

Across eight runs on six dates, we recorded 12,609 LLM API calls: 11,438 for generation and workflow steps, and 1,171 for judging.

We expected higher demand to put pressure on API capacity and quality of service. Our workload is not big enough to produce a scaling issue for these providers, but they have strategies to deal with the worldwide workload as it ebbs and flows like a tide. Trying to observe that through our evals is a little like measuring the tide from inside a swimming pool. Comparing time windows equally shows whether performance shifts across the day, regardless of how many requests arrive in each window. Weighting results by actual request volume answers a different question: how often customers encounter those conditions. We need both views to tell whether a time-of-day difference exists and how much it matters in practice.

Slower in the morning, faster in the evening

Every morning coding run had a higher mean time than every evening run. Morning means ranged from 12.15 to 14.50 seconds; evening means ranged from 10.67 to 11.98 seconds. We had four morning runs, three evening runs and one late-night run. That 23:20 Pacific Time run fell in late morning in India, so it may have overlapped with demand there; we do not have regional provider-load data to test that possibility.

The run means show a clear morning-evening difference, but they do not establish that latency varied more within morning runs. Quality also shifted: evening runs passed more tests, though the change was much smaller than the change in latency.

All eight runs used the same application commit. All 100 cases kept their model, reasoning and requested service-tier settings, and their saved system and user prompts matched wherever artifacts were available.

Next we checked whether extra agent work explained the time. The coding timer stops before code execution and judging, and the same morning-evening gap appears inside the API request timings themselves. Across 799 timed requests, work outside the request averaged less than 12 milliseconds per case in every run. Tool execution, judging and our local rate limiter do not explain the pattern.

On both dates with a morning and an evening run, the evening mean API request time was lower: by 16.9% on October 20 and by 4.7% on October 21. The gap also holds when we drop the earlier paired runs and the run with one missing timing. That leaves two morning and two evening runs, and the direction survives.

Our prompts, settings and code stayed the same, so the extra morning time was spent inside the request: on the network, in response handling and in the provider's own work. This timer cannot isolate the server, and output and reasoning length can vary even with identical prompts. The broader workflow tests and the quality scores did not show the same clean split.

The measurement notes records each check and its limits. The same-day comparisons and the checks excluding earlier protocols give us a reason to keep tracking this pattern, without treating it as a universal rule about mornings.

Latency was unrelated to token volume

In every run with complete cost data, generating the same 100 code cases cost between $2.34 and $2.42, a 3.7% spread. Mean code-generation time spread much wider: 10.67 to 14.50 seconds, or 35.9%.

The latency difference was unrelated to input tokens, output tokens, or context size. Cache-hit rates did change between morning and night, however, so cache behavior was not held constant. These runs cannot tell us whether that shift contributed to the differences in latency or quality.

Customer business-rule answers behaved the same way. The same 50 questions cost $2.48 to $2.51, while mean latency ranged from 13.08 to 15.59 seconds. A budget built on the bill would have held. A response-time promise built on the fastest run would not.

These ranges describe the runs we observed; they do not bound the next one. In this sample, similar cost and token volumes coexisted with longer, more variable waits.

The multi-step analysis workflows moved on both fronts: generation cost varied by 34.1% and mean latency by 22.6%. Those workflows gave us a reason to check our own execution path before drawing a conclusion about model response times.

Was the extra time coming from our agents?

Before attributing the wait to the model request, we had to check what our agents were doing. The 50 analysis workflows include dataset exploration, calculations and chart creation. Their timings mix model requests with tool execution and orchestration, so we first broke them down by the kind of work.

The seven chart workflows had mean latencies from 79.95 to 130.10 seconds across runs. The 18 dataset-exploration tasks ranged from 16.67 to 22.00 seconds. That made the overall workflow average a poor starting point for isolating a timing change.

Serving choices can affect output quality as well as speed. KV-cache quantization is one possible factor; Cloudflare describes techniques for making models smaller and faster. We did not measure quantization in these runs.

Coding gave us one: each completed case made a single generation request, with its saved prompts and settings held fixed. We could measure that request before executing or judging the generated code, then compare the runs by when they started.

Test the model at the hours your customers use it

Every morning coding run was slower on average than every evening run we measured, while evening runs passed slightly more tests. Our feature development and much of our customers’ work happen on weekdays, so weekend benchmarks may not reflect the conditions they experience.

The extra time appeared inside the API request, before code execution or judging, with the application and saved prompts unchanged. Cache-hit rates varied between morning and night, and serving choices such as KV-cache quantization may affect quality. These runs do not isolate which service-side factors contributed. Providers could make this easier to investigate by reporting queue and processing times, resolved model versions, and cache behavior.

We should keep running the same representative suite at the hours customers use the product, including both weekends and working days. Compare time windows equally to spot shifts, then weight them by request volume to estimate how often customers encounter each condition. Track quality, latency, and cost together, with results visible for each window.