Simply Business talk observability at FutureStack 2022

Back in February 2022, New Relic invited Simply Business to join their Technical Advisory Board. "As an important, strategic customer to New Relic, [they were] honoured to include [us] as part of this exclusive group of engineering leaders who are shaping the future of observability."

Their words not ours! The Technical Advisory Board is a group of technologists hand-picked by New Relic to help them develop new features. It’s a good sign for our DevOps team to be invited to join this group.

The first meeting was to be at New Relic's annual FutureStack conference in Las Vegas in May 2022. We're not ones to turn down an expenses paid trip to a five star hotel with three swimming pools, so we also agreed to present at the conference.

Observations on observability

The DevOps team at Simply Business had recently moved all of our telemetry data from Datadog and Kibana to New Relic, so we had a lot of lessons to share. In our breakout session called Three Heresies about Observability: Things I Used to Do but You Definitely Shouldn't, we shared what we'd learned about:

• What motivated us to put everything in New Relic
• What benefits we got as a result
• Potholes we hit along the way
• Why, if you have a side hustle selling on Amazon Marketplace, you should buy cover from Simply Business

Hear what we had to say in the video recording of our conference talk.

Three Heresies About Observability: New Relic FutureStack 2022

Duration: 40 minutes (approx)

Our timing was good. The benefits of consolidating tools was a repeating theme in this year's conference. Many people attending were in a similar situation as we'd found ourselves in, so they appreciated our tips (as well as the many cat memes).

The wisdom of crowds

But we didn't attend FutureStack 2022 just to lecture people. We also wanted to learn from what other people were doing. It was encouraging to see that the main themes of the conference were things that we are already doing at Simply Business:

• Adopting OpenTelemetry
• Making connections between metrics, logs and traces to get new insights
• 'Shifting left' – putting more power in the hands of developers who are closer to the code
• Building a platform-as-a-service for developers, to make it easier for them to deploy their apps

As a developer-focused conference, there was a lot of attention on developer experience at FutureStack 2022. Developer experience isn't just about giving developers a warm fuzzy feeling. A better developer experience makes it easier to find the cause of problems, quicker to fix problems, and easier to learn the tools. Spending time on improving 'DevEx' returns practical and tangible benefits, so it’s something that we in the DevOps team are putting a lot of time and effort into.

Dear Santa ...

At the conference, New Relic announced a Christmas wishlist's worth of new features that they'll be adding to their platform. Here's a handful that stood out to us.

Shiny new UI

Now that the DevOps team at Simply Business have moved more and more systems and tooling into New Relic, the UI can get busy. It can be tricky for developers to find what they want. Different developers use different features, but everyone gets the same UI.

In the new UI for New Relic, you'll be able to customise the interface to show you only the things you care about. So if you're in a team that doesn't use browser monitoring, you can hide that from the sidebar to make more room for everything else.

We've signed up for the beta, so expect to see this in a browser tab near you soon!

More secure PII protection

Simply Business takes data protection seriously, so we're careful about not logging personal identifiable information (PII). But sometimes, some get through the net.

We already do some log redaction using our Logstash log collector, but we can also now do log redaction inside New Relic. Previously, we would have to raise a Logstash pull request with drop rule changes and wait for it to deploy. But now, any developer can redact incoming data inside New Relic. This lets us patch PII leakages more quickly.

Even more exciting is masking. This puts sensitive data through a one-way, deterministic hashing algorithm. Big words, but this means you get scrambled data that no one can reverse to get the original, sensitive value. What makes it different from redaction (overwriting the data with meaningless stuff) is that if you knew the original value beforehand, you can match it with the scrambled value. The scrambled value is a sort of fingerprint for the original value.

The masking feature means that people who have a legitimate need to see sensitive information like payroll data or personal email addresses can still answer questions like "How many requests did the person who lives at this address create?" They can do this without revealing the actual address to people who didn't already know it before. This opens up new opportunities for safely using the data in places in which it had previously had to be redacted or deleted.

In-app discussions

At Simply Business, we rely on New Relic for debugging issues and resolving incidents. Until now, that discussion happened in a mix of Slack channels, impromptu Zoom meetings, and incident reviews in Google Docs. It can be difficult to remember which discussion happened where. Important conversations aren’t always recorded. When dealing with an incident, jumping between meetings and apps breaks the developer's flow, at a crucial and stressful time. Developers want to stay focused on solving problems, not writing explanations.

Now we can make comments on issues directly in New Relic. This treats our telemetry data a lot more like our application data or code. By having comments close to the data, people who come across the same data later on can read up on what happened.

In this way, we:

• Reduce the chance of 'documentation rot' (information that was accurate at the time, but the data it was referring to has now changed)
• Make information more discoverable by showing it when it's most needed
• Don't interrupt developers' flow state by making them context switch to write docs

It keeps getting better

The DevOps team at Simply Business keeps working to make observing our applications a better experience for our developers. Our aim is to create an experience that developers enjoy, rather than deal with because they have to; that makes their jobs easier, rather than gets in the way. With the migration to New Relic completed, the DevOps team can now build things that we were not able to do before.