Splash! A Test of Real Work

I had been getting promising speeds from Splash in tool tests. I wanted to see whether that carried over to a longer job: reading an existing analysis, checking a live site, following up on problems and coming back with useful recommendations. How long would I actually wait for the work?

So I gave two Qwen 3.8 27B builds the same SEO assignment for a wine e-commerce site. Splash delivered its findings in about 4 minutes 16 seconds. Youssofal’s time was about 9 minutes 17 seconds after subtracting an extra audit-file step, as explained below. Then I read the sessions, and the comparison became more interesting: partway through, they had chosen different jobs for themselves.

Both checked the known issues. After that, Splash went looking through collections and structured data. Youssofal explored old pages, stock availability and internal links. Each found things the other never looked at. I had given them the same brief, but I hadn’t obtained two copies of the same audit.

Fortunately, the logs let me line up the work they shared. Splash was faster there too, although by a smaller margin than the full completion times suggest.

The assignment

I used the pi agent harness, on the same machine and in the same project directory, with a fresh session for each build. The model identifiers were incoai/Qwen3.8-27B-Splash and youssofal-qwen38-27b-optimized-quality. Both had reasoning set to medium. These were two builds in the same 27B family, accessed through different provider configurations; this comparison includes the runtime and serving setup, rather than isolating a change in model architecture.

The prompt asked them to read the latest SEO analysis, check the site and tell me what improvements were still needed. I explicitly told them not to refresh DataForSEO. They could use the existing research and fetch live pages, but the task did not include another keyword-data pull.

Both started from the same review, dated 15 September. They listed the audit directory, read the review, fetched a batch of core pages and extracted titles, descriptions, canonical URLs, indexing directives and H1 headings. Both also checked product structured data before delivering prioritized findings.

This was a multi-step audit lasting several minutes, with the model deciding what to inspect next. It was closer to my everyday work than a short tool test, though it says nothing yet about an agent working reliably for hours.

They agreed, then went exploring

On the items both checked, the recorded conclusions agreed. They confirmed unresolved pagination and competing-page issues, found two H1 headings on the homepage, and checked that product structured data and several earlier fixes were in place.

The split came during discovery. Splash found two orphan event collections that were still indexable and included in the sitemap. It also reported missing structured data on restaurant, event and club pages. Youssofal sampled the pages sitemap and reported roughly 110 legacy or test pages still live and listed there. It inspected the product feed and found 65 of 111 products out of stock, then followed internal links through competing pages and an older subscription route.

These are findings reported by the sessions, not a separately scored measure of SEO quality. A missing schema type, for example, still needs to be judged against what the page actually offers.

It was funny to watch the same prompt turn into two different investigations. It also makes an open-ended comparison difficult. Each choice changes what the agent sees next and how much work it takes on. Repeating the prompt does not hold the workload constant.

After excluding the audit-file step, Youssofal generated about 30% more tokens than Splash. Counting findings or comparing the finish times alone would mix speed with the amount and kind of work each chose to do.

Comparing the work they shared

The session logs record call timestamps and token usage. That allowed the analysis to separate orientation, the core page checks, exploration and reporting.

The closest match was the core fetch and extraction: both processed the same core pages and pulled out the same kinds of metadata. The opening orientation was also similar, although Youssofal read one extra document.

These are accumulated model-call durations for those stages, including the wait for generation to begin. They are not elapsed times including every intervening tool execution.

Splash used about 36% less model-call time across that shared opening. On the more closely matched core-page stage alone, it used about 22% less. Those are the comparisons I would use when deciding whether the speed I saw in short tests survived contact with this job.

Token throughput helps explain the result, but it gives a different ratio:

Splash generated about 1.7 times as many tokens per second, while producing slightly more tokens for this part of the audit. That is why a 1.7× throughput advantage does not translate directly into a 1.7× reduction in time to complete the checks.

What the whole session cost in time

For this comparison, I excluded the extra audit file Youssofal chose to write before answering in chat.

The clock starts with the original SEO request and stops when each model delivers its first completed findings in chat. Later requests to measure speed, save statistics, consolidate findings or do further research are excluded. I also subtracted the call that generated the audit file and the write operation: 107.529 seconds and 2,035 generated tokens, including 46 reasoning tokens. The final chat answer remains included. This produces an adjusted time, not a measured rerun without the file; the final answer still had the file contents in its context.

The adjusted time ratio is about 2.2×. Youssofal still investigated different surfaces, so even after removing the file-writing step this is not a comparison of identical workloads.

Both devoted roughly 30% of generated tokens to reasoning, so the runs did not show a large difference in the share spent thinking. Both also used prompt caching heavily. The logs do not separate the initial wait from token generation, so these rates describe the calls as experienced by the harness, not pure decoding speed. They cannot tell me how much of the gap came from the model build, caching, or other serving behavior.

There was one run per build, with no repeats or estimate of normal variation. The timings favor Splash in this trial. The different investigations leave the overall quality comparison open.

The next test

I wanted to test sustained work because a fast tool demonstration only gets me so far. Splash’s speed held up on the shared checks, and it returned useful findings sooner in this run. Youssofal’s longer investigation also found problems I would want in the audit. I would keep those findings regardless of which model finished first.

For the next comparison, I would give both agents a fixed first pass: the same core URLs, collections sitemap, pages sitemap, product feed, internal-link checks and structured-data checks. I would require the same deliverable, then give each a separate allowance for exploration. That would let me compare coverage and completion time on a defined job while still seeing what each discovers when left to choose.

For now, Splash has earned a longer trial in my workflow. The result I can carry forward is the shared opening: 74.0 seconds of model-call time against 115.4, with agreement on the checks both performed. I still want to see that repeated across more jobs before changing the recommendation in my local-model guide.

Tested 19 September 2026. Figures come from the A/B session analysis for this SEO audit. Generated-token counts include visible output and reasoning; call durations include the wait before generation starts. The underlying sessions contain client project material and are not reproduced here.