We talk to a lot of clients about AI, everything from website chatbots to bespoke solutions integrating with their data to drive analytics and personalization. In all these cases, there is one question we have to absolutely make sure we never lose sight of: Is this actually a good use of AI for the value it will provide, or are there other alternatives for what we’re doing?

So when the product team behind the Sitefinity CMS told us they'd been building AI-powered upgrade + testing, and regression tools, our first reaction was to ask that same question. As we learned more about what they were investing their time in and why, we grew just as excited as they were and jumped right into this project with them.

You see, Sitefinity CMS has been integrating AI across the platform for a long time now. AI-assisted content rewrites, SEO suggestions, AI-based personalization, purpose-built AI Agents, and so much more.

But what really caught our attention here is that they also looked at the unglamorous, expensive, high-friction work that partners and clients deal with behind the scenes: testing, regression validation, upgrading confidence - and decided that was worth solving too. That's a vendor investing in the total platform experience, not just the features list.

To help them test their process, we offered to validate their approach on some of our own projects. And because we wanted to be honest about the results, we ran the same upgrade twice: once with the AI tooling, once the traditional way, with different team members on each path.

Yes. We did the upgrade twice. We wanted definitive proof that the AI testing process did what it was supposed to do and did it right the first time.

The Work Nobody Wants to Pay For

If you run a Sitefinity site hosted on-prem with any real complexity, customizations, or integrations, you might dread the upgrade conversation, just like with any other CMS. The more complex your site, the more tedious the work can be, with a real risk of breaking something that works fine today.

But the upgrade itself is only part of the whole process. There's the content freeze, the deployment coordination, the QA cycles, and the stakeholder sign-offs. And underneath all of it is a testing problem: most CMS projects don't have automated test suites. They've never been worth the investment when things either change too quickly to keep up, or the overhead of an automatic test suite is too great to justify. This leaves testing a mostly manual effort where you prioritize changes and features. In fact, that’s technically true of any time you develop anything. You prioritize based on risk, time, and effort.

That’s the gap between the new tool available for Sitefinity developers' targets. Not the upgrade specifically, but the testing and validation work that makes upgrades (and every other deployment) risky.

What We Tested and How

Our largest pass was on a site with a serious codebase: 40+ custom widgets, multiple search indexes and integrations, and deep customizations over 5 separate major Sitefinity upgrades of this project during the years.

Working with Sitefinity's team before public release, we ran their AI agents through the full workflow:

Test generation. The agents analyzed the live site and produced 120 Playwright-based tests covering visual regressions, functional workflows, navigation paths, and integration points. We gave it the context for the tests we wanted, and the AI created and ran them.

Stabilizing those tests took about two hours total. Building an equivalent suite by hand would take days of skilled engineering time, which is exactly why it almost never gets done.

Upgrade execution. The agents ran the version upgrade and worked through the build and runtime errors that followed — dependency changes, API shifts, assembly mismatches. Predictable, tedious work that's well-suited to automation. Sitefinity already has a command-line tool that helps a lot- but now agents are going one step further and fixing common build problems and breaking changes. They start the project and fix runtime problems as well.

Validation. The same test suite ran against the upgraded site. Failures were flagged with screenshots, video, and analysis. Engineers reviewed everything and made the final calls on what was and wasn’t a bug.

What the Side-by-Side Showed Us

Both paths produced a working upgrade. The AI-assisted path was faster, but speed wasn't the point of running it twice.

Post-upgrade, 118 of 120 tests passed. One small, quick fix, and one false positive.

The traditional upgrade path also found the same issue, but involved more human intervention.

NOTE: To be fair, we also spent time performing a manual smoke test on the AI-assisted approach. While AI is a fantastic tool, we treat it like just like we would a comprehensive automated test suite. These are all assisting testing when and where it makes sense. Humans never go away; they just go do other things while AI automates what makes sense.

Why the Test Suite Matters More Than the Upgrade

Here’s the best part of the entire process. The upgrade is done and live, but the test suite is going to continue providing value.

Those 120 tests run against any change to the site going forward. Feature deployments. Content restructuring. Integration updates. Configuration changes. For a client that had zero automated regression testing before this project, which describes most CMS clients we work with, that changes the risk profile of every future deployment.

Upgrades are the most visible use case because they touch everything at once. But the ongoing value is quieter: catching regressions early, before they reach production, on every release. That's the real return on Sitefinity's investment in this tooling, and it's what makes this more than an upgrade story.

What This Says About Sitefinity CMS

We've been a Sitefinity partner for a long time. What made this project worth writing about isn't just the tooling and the use of AI; it’s what the tooling represents behind the scenes.

CMS vendors have plenty of incentive to invest in flashy features that win new business. Very few take the time to invest in the operational pain their existing partners and clients live with every day. The Sitefinity team looked at the testing and upgrade workflow, the thing nobody puts on a features page, and decided it was worth solving.

That's the kind of platform investment that makes a real difference in how it feels to build on and maintain a Sitefinity site over time. The ability to reach out and work with real human beings before a feature is launched, and know that day 1, this is going to be something special.

Get Started

We worked closely with the Sitefinity product team on this before the public launch. These tools are now available to the broader community. Find them here.

If you've been deferring a Sitefinity upgrade, or you want to understand what automated regression testing could look like for your site, we'd be happy to walk through what we learned. Contact us here.