RELEASE 01

One theme, two lenses. Tom Barber on the support matrix nobody agreed to. Amelia Prasad on what users say they want versus what they use.

{{current_date_full}} · 25 to 35 min episode · Read online

Welcome to the Concept to Cloud newsletter. Here is the deal with it: one theme an edition, two takes on it, mine and Amelia's, and an episode where we talk it through properly. If it would not earn a place in my own inbox, it does not go in yours.

In this edition I get into the support matrix nobody agreed to, what it actually costs to ship software you cannot see running, and why “we fixed it” stopped being a sentence I can honestly say. Then Amelia takes the other lens: the gap between what users tell you they want and what they actually use, and how to tell a tack-on from a ground-up rebuild. Grab a coffee. Let's get into it.

By Tom Barber

“Can you just send us a patch?” is the most expensive sentence a customer can say to you, and it always sounds like the cheap option.

How do I know? Because back in the day I used to bend over backwards making patches, fixes and enhancements for Saiku users. Most of it went out for very little and a good chunk of it went out for nothing at all. Cheap for them. The bill came to me instead, in installments, and it kept arriving for years afterward.

I don’t know how many copies of Saiku, our data analysis tool, are running right now. I know roughly how many downloads, over a decade and more. I know precisely how many installs I can see, which is the small number. Everything else in this article comes from exactly that gap.

I build products, and I have spent years helping other companies ship theirs, mostly by telling them what I got wrong first.

So this is about what it costs you to ship it, not what it costs your customers to run it. Worth getting that straight early, because the two get confused constantly and the whole argument falls apart if you are standing on the wrong side of it. Build versus buy is your customer’s decision. This is yours.

Side patching, or cutting a release every time you want to keep somebody happy, is never free. You just do not get the invoice at the time.

As a software vendor you have three options. You run it, as a SaaS platform. They run it, on their own kit. Or both. This piece is about the third one, because the complexities of doing both are not small and almost nobody prices them before they commit.

That is really the point. Shipping is an event you cause. Upgrading is an event you ask for. Once even one customer is running your software somewhere you cannot see, your release stops being a deploy and becomes a request, and everything else in this article follows from that.

In practice you are running the latest release for the SaaS customers, migrated all at once as part of the platform in a managed way.

Then you have the on-prem customers, the ones who self-host. They still expect support. Some are stuck on an older version and still want features and fixes; some pay you a great deal of money and flatly refuse to move off the 2019 release, which barely works in a modern browser.

Here is what that gap looks like from my side. 97 new instances. Another 180 older ones still out there, running unpatched and untouched, some of them for the better part of a decade. And those are only the ones that phone home.

Nearly two thirds of everything I can see is a version I stopped thinking about years ago.

That puts your support team in a bind, and it is a measurable one. On-prem is about a third of our ticket volume, and those are the slow tickets: triage against an aging codebase, no telemetry, and a fix that then has to travel.

A Saiku release has to factor in how customers move between versions. Features have to survive upgrades; schemas have to work forward and backward. That sounds like a technical constraint and it is not. It is a product constraint: it decides what you are allowed to build and when you can ship it.

The clearest one I have is the dashboard work. We had a big extension to the old dashboard support in Saiku, built and ready to go. One paying customer needed other things in that same release and could not take the dashboard change alongside them. So we worked with them rather than around them, and the release went out six months late. Not the dashboard piece. The whole release.

If you are reading that and thinking a feature flag would have solved it, you are right, and that is the point. Nobody in the room framed it as a structural problem. There was no meeting where we agreed to trade half a year for one customer’s upgrade path. It looked like one customer with one reasonable objection, and it happened one reasonable conversation at a time. It always does.

Somewhere along the way a backport policy becomes a roadmap. Your release dates belong to other people’s change windows, and to decisions I made fifteen years ago that never looked like they would matter.

We used to think self-hosted data platforms were a legacy issue. Something you kept supporting for the customers you already had, while everyone new went to SaaS. That is not what my inbox says. Three quarters of the inquiries we have taken recently asked to self-host.

I could point at a market report here and I am not going to, because market reports measure interest and my inbox measures intent. Three in four is not a trend line. It is next year’s roadmap.

People care about their data, its security and its provenance, and in regulated industries it matters more than almost anything else. It does not mean anyone is buying up rack space again. It means they want a platform where they stay in control of what happens to their data.

That demand never arrives as a strategy document. It arrives as a request, and it drags a smaller product organization from SaaS only to SaaS and on-prem almost by accident, without a product decision ever being made.

When you are building for both SaaS and on-prem there are a number of product design decisions to make, and I do not mean in the UI.

Nobody wants to maintain a fork for the rest of time, so can you ship one build and change its behavior with config? That is the load-bearing decision and the one most worth defending later, because every exception you allow becomes a permanent one.

Then environment parity, where the biggest problems come from authentication. Everyone wants your software wired into their own ecosystem their own way, and you have to decide how much of that you will absorb before it stops being one product. Upgrades sit in the same bucket. Shipping a release is rarely just a package. It is migrating backing datastores and making sure configs survive, on a machine you cannot log into.

And telemetry, which is the thing that quietly makes SaaS easy. Anything you host you can instrument. On an on-prem deployment you see none of it.

Which leads directly to the problem nobody warns you about.

All of this feeds straight into pricing, so here is ours. On-prem is a single instance license, per annum, not per user.

That is deliberate. Counting seats on a machine you cannot see means either trusting the customer or building surveillance into software somebody installed specifically to avoid it. So you price the instance, and you charge annually, because the renewal is the only reliable moment you get to talk to them about moving.

The harder question is feedback. How do you find out what they are running, how they are using it and what they want next, when the software lives somewhere you have never seen? Easy in SaaS mode. Not here.

And when they do ask for something, knowing it needs a release they may not take for eighteen months, how do you say no without losing their trust?

Mostly you do not say no. You say yes, and you attach it to the upgrade. The request becomes the reason to move, which is the only currency you have with a customer who is perfectly happy where they are. It works more often than refusing does, and when it does not work you have learned something useful, which is that the request was never that important to them.

That is less a technique than an admission: on-prem, every feature conversation is an upgrade conversation in disguise.

So how have I got through fifteen years of this, in Saiku and elsewhere? There are a handful of rules I try to keep with any product.

Keep it as boring as possible. In my early days I loved adding features, plugins, extensions, anything that sounded interesting. What that actually does is add a large support burden. People have to install them, configure them, work out how to connect them, and every one of those steps eventually lands back with you.

Keep migrations forward only. Do not backport features, functionality or data structures.

Which sits awkwardly next to those customers stuck on the 2019 release, and I know it. So here is what actually happens when the 2019 customer rings.

I try to talk them into upgrading. Most of the time that works, because the annual renewal gives me a reason to raise it and, frankly, they would rather not be where they are either. When it does not work, the version eventually becomes genuinely unsupportable, and at that point the honest answer is that migrating is a piece of work somebody has to pay for. Not a patch. An engagement, scoped and quoted like any other.

That looks harsh written down. In practice it is the kinder option, because the alternative is quietly maintaining a 2019 branch forever while pretending it is free. It is not free. It comes out of the release everybody else is waiting for.

Have a way to get diagnostics out of the product on the client site. Preferably a CLI or an application command, not a list of files you ask them to go and find. Build something that zips the lot into one bundle so they only have to send you a single file. It keeps their side light and it usually gives you everything you need.

If yours is the only copy, “we fixed it” is a statement of fact. You change the code, you push it, and it is true for everyone at once.

Somewhere out there are people running a version of Saiku I last touched in 2016. I could fix their bug this afternoon and it would still be sitting in their install next year. The fix exists. It is just not where they are, and getting it to them is not a patch. It is a project, with a budget and a change window and a date that nobody wants to own.

So “we fixed it” stops being a fact and becomes an offer. I have been making that offer for fifteen years, and I have never once been able to say it and mean that their problem was over.

TOM IS SPEAKING

DC Systems, September 8th. Killing the Cluster: Replacing Spark with Polars in a Pipeline That Ships to Places You Can't SSH Into.

Federated learning inverts the normal deployment model: data stays put, and the pipeline travels, into on-prem data centers, restricted-egress networks, and architectures you don't control, past each institution's security review. On a financial-crime detection network deployed across major financial institutions, we learned that Spark's JVM-and-cluster footprint was the wrong kind of distributed for that world. This is the story of migrating the pipeline to Polars: proving engine equivalence, handling larger-than-memory work without a shuffle, deleting operational surface, and discovering that the real requirement was never “distributed”, it was “portable.”

By Amelia Prasad

We’re in a season of AI where much of the product work is like that of a child at a wishing well: a piece is chosen, given its best polish, and thrown with the best intentions and the acute hope of success. But AI isn’t magic and as ever, the mechanism for success lies behind the honing of a craft, and in this case, that craft is knowing how to listen. The goal is to hear the user, to understand the difference between what they say they want, what they are actually using, and what they truly need all in service of providing a product that moves them forward.

So how to do it? A bit of metacognition, a bit of process. We have those three signals, but we need to think about the nature of their presentation, understand any inherent biases in our listening to make sure we see the full picture. It’s a bit UX 101, but the principles hold.

We start with the signals themselves. The first, want, is often the loudest and least deeply true because it sits within the bias of the bounds of the existing environment. Much of the customer feedback is only in terms of what we make space for - feature requests, bugs, frustrations, but all are in bounded dome of the product that exists. It’s important to separate the feedback that is about customer need and those that are our errors in execution.

The second signal is what they use, and it’s the evidence that should guide our focus: the jobs they get done, the flow their real life requires, the tools they reach for. But even use is only part of the picture, because what we see them use is still bounded by the product’s defined offerings. That is what makes the third signal, need, the hardest: it is reached largely by inference. Are we really serving everything they are trying to achieve? Their experience lives both within the product and outside it, and the need is the part that lives in the gap.

Want is stated, use is observed, need is inferred.

Without understanding and addressing these biases, the product cycle will lean towards only adding new features. No one can request, in the vocabulary of the product they already have, the thing that would replace it. This unasked-for replacement is the request of a customer we cannot yet see, one the current product has no words for, and the work we must do is imagine them anyway.

There are plenty of ways to listen past those bounds, and oftentimes it’s solid user interviews that do the trick. However, when you’re deep in the development of a product, listening is a bigger ask, because it begs the question: do you tack on another feature, or is it time to rebuild? Rebuilding can sound terrifying, or exciting. It reads as time, resources, new tech, starting over, and wondering why it wasn’t quite right the first time around. But usually, it isn’t starting over. Technologies evolve, and so do workflows. The capacity often already exists in your engineering team and your data structures, and the ask is only to present it differently, or to integrate something new alongside it. Consider this, that as a society, we’ve moved from dial-up to gigabit, without anyone losing services for swaths of time. Products can change underneath people the same way: the ground shifts, and the experience on the surface slowly improves, and is never greatly disrupted.

So why does the listening carry all the weight? Because the cost sits on the other side, in getting the need wrong. A rebuild frightens stakeholders, but the cost of not doing it is greater: when we have the chance to give a customer what they truly need and instead keep them inside the bounds of what we already built, we’ve laid a pitfall into the product. You can hotfix a bug, if you can reach the user. A product built for the wrong job, you can’t: the miss is baked in, and the customer is the one living inside it. You cannot fix for a customer you cannot reach, and you cannot answer a need you never heard. Get it right, and the work that follows is smoother than it sounds: re-surfacing the information you already hold, unified across engineering and product, to meet the workflow the user actually needs.

Every venture that ever moved an industry forward depended on a single thing: listening past the words. So by all means, look forward, but don’t forget to look around.

25 to 35 minutes

Not a reading of the article. The two of them on the theme.

Concept To Cloud · product engineering for regulated and PE-backed companies · concepttocloud.com

You are getting this because you subscribed. Unsubscribe · Preferences