AWS re:Invent – technology talks

Simply Business joined 65,000 cloud computing enthusiasts at AWS re:Invent, Las Vegas, 2019, to talk all things infrastructure, security and technology. Lead DevOps Engineer at Simply Business – Paul McAdam, talks to a cross-section of Simply Business technologists to hear what they learned from the conference. Here are the key takeaways.

Tech talks in Vegas

This was Simply Business’s second year at AWS re:Invent. Representatives from SB Tech infrastructure, DevOps, BizOps, security and engineering set out with one unified mission – to discover how Simply Business’s technology stack stands up in the AWS journey.

In this interview, Lead DevOps Engineer – Paul McAdam talks to BizOps Leads (Vijay Prasad, Nathan Emmens), Senior Performance Engineer (Drew Curd), Security Engineer (Vincenzo Zambianchi), Senior Application Security Engineer (Dan Murphy), Engineering Manager (Quynh Nguyen), Engineering Lead (Angel Carballo), Lead DevOps Engineer (Jayne Fox) and DevOps Engineer (Andy McKay) about their technology insights at AWS:reInvent.

Bots meet BizOps in an unchained melody

[Paul:] What struck a chord for BizOps at AWS:reInvent?

[Vijay:] Vegas can be a little overwhelming to say the least. With over 4000 talks, workshops and events, and five days of Tech conference, it had me feeling similar to my last attendance at a music festival! This came in handy, as some notable highlights were the in-depth workshops on Generative AI in AWS DeepComposer – a musical take on getting developers started with machine learning.

Generative AI

To explore Generative AI, you use the AWS DeepComposer keyboard to key in a melody or use a pre-set one (think “Mary had a little lamb“), and then pick one of the built-in custom AI templates to create a full musical arrangement for your melody, in the form of pop, rock or jazz.

You can play with the AI template to see how it works and make variations. The AI was fed tonnes of music from the past 100 years and can distinguish between music genres. At the end, your melody is backed with complete arrangement, all developed by AI. A bonus was getting to walk away with the AWS midi keyboard for free!

Mixin’ with Saas architects

I enjoyed meeting with our software and SaaS providers from Datadog and Looker, as well as AWS lead architects who offered some great advice on how to optimise or save money for the business (and encourage us to come back next year!). I also had a moment of reflection and gratitude at the AWS closing party. Being in Tech is a great place to be! It’s fun, it’s full on and there’s forever more to learn.

Infrastructure ain’t heavy

[Paul]: Sounds like you really enjoyed the mix of content and exposure to areas not strictly related to your day-to-day. What was your take on it for infrastructure Nathan?

[Nathan:] As an infrastructure engineer still in the early days of using AWS, re:Invent exceeded my expectations for the variety of sessions, catering for anyone who touches on AWS, from leadership to builders, from workshops to talk chalks and hands-on labs.

My main takeaways were around simplifying infrastructure access management, network monitoring and machine learning:

• S3 Access Points
• IAM access analyser
• Peer to Peer transit Gateways
• Transit gateway network manager
• AWS Outposts
• Kendra

I focused on networking to deepen my understanding and learn about new products. The ML theme underlined re:Invent this year and I’m keen to dig deeper into SageMaker Studio to explore how it can be applied at Simply Business.

One last takeaway, from Andy Jassy – CEO of AWS:

Don’t put off until tomorrow what you can do today. Don’t put off big changes.

For big transformations:

• Align senior teams
• Have an aggressive top-down goal
• Train your builders
• Don’t let paralysis stop you before you start

Something we can all relate to I think.

Tools not rules

[Paul:] That’s great to hear. Now over to Drew, our fresh ‘Team-Player of the Year’ award-winner. Drew, congrats on the award! You must have learned about it while at re:Invent – I hope that didn’t go without a few toasts! As you were on national soil, you must have felt at home among all the cowboy hats?

[Drew:] A soon as I saw the cowboy hats, I realised why I felt at home all week! Well, that’s not quite true, what seemed like the unofficial focus of AWS re:Invent was reliability and observability. While these topics are usually in the same space as monitoring discussions, what I loved about re:Invent was the next level of many of the whiteboard sessions.

Autonomy versus anarchy

One of my favourite aspects about working at Simply Business is the elevation and autonomy of the software engineer. However, this often leads to discussion on how to balance the freedom of the engineer against the prevention of anarchy. One of the teams I’m part of – Application Tooling – is helping to address this contention at Simply Business by implementing technology to address the concept of ‘tools not rules‘.

In one of the talks, an AWS architect mentioned that, “… relying on good intentions never works”. Most people going to work don’t want to break their systems, so why not create systems that implement best practices automatically?

Infrastructure boundaries

One of the talks that really resonated with me was about the reliability of AWS’s own pipelines. As standards are constantly evolving and security is ever-changing, AWS has written internal rules for pipelines as Lambda functions. These Lambda functions ensure that every pipeline within AWS’s internal infrastructure is kept within proper boundaries. If a pipeline is outside of compliance, it is immediately quarantined, with teams determining the level of exposure and timelines for remediation. One can take this a step further and start thinking of the multiple ways this idea could be implemented – ensuring teams have proper monitoring, are following best practices, are using certain internal tools – the list could go on!

Handling outage

But what happens when things move in a negative direction? The cryptocurrency exchange Coinbase had a terrific talk on this subject on how to deal effectively with outage situations.

Coinbase has written a Slackbot that allows for a one-line command to begin this process. It enables the creation of war rooms and escalations during outage-level events. When the crisis is over, this bot streamlines the post mortem process to allow for effective analysis with minimal effort, by auto-generating templates and compiling available data at the time.

Just another example of ‘tools not rules‘ benefiting the customer and easing daily life for many an engineer. And for the record, I’m hoping that Coinbase make their Slackbot available as open source!

Information security

[Paul:] You are making me envious! Now let’s hear from InfoSec. What did re:Invent touch on for information security? Any chalk talks you want to make people aware of?

[Vincenzo:] This was my first time ever in Vegas, and I found it simply awe-inspiring!

Glitz and glamour inside the Bellagio

Fraud detection

A personal highlight for me was the chalk talk (smaller interactive sessions that do not get recorded) on fraud detection. AWS’s fraud detection feature in SageMaker is capable of choosing the best-fit model from several ones, and can be trained on a given fraud-labelled or unlabelled dataset.

This actually has broader implications, as it’s not necessarily scoped to fraud detection alone. The main idea I take back from this talk is to research whether there are ways of using auto encoder-based algorithms to help us parallelise unsupervised learning model training; this would be helpful if we were to rely on a rapidly-evolving bot detection model.

Related to this, I believe we could take a fresh look at how we might integrate a Lambda function to return bot-detection model results into a Cloudflare / AWS Web Application Firewall traffic authorisation flow. This is something that was covered in SEC332-R1 – [REPEAT 1] Bot mitigation at the edge (keep an eye out in case it becomes available on YouTube).

Traffic mirroring and logs ingestion into Elasticsearch

A couple more things worth mentioning are VPC traffic mirroring and Elasticsearch logs ingestion patterns.

Our information security team could leverage VPC traffic mirroring to get better insight into our intra VPC communications, as this is a major enhancement over VPC flow logs, allowing for more detailed inspections.

At re:Invent, you also learn about the trials and tribulations of fellow attendees when you talk to them standing in queues, so you are continuously reminded of how important it is to introduce queues in any workload pipelines.

When you do so, reliability is brought to new heights; that is going to be beneficial for our Elasticsearch logs ingestion pipeline. It goes without saying that teams need to work together to introduce this new change, aiming for higher sturdiness of the system as a whole.

On a final note, this has been the best business trip I’ve ever been on (even if it left me wondering whether I was still on the Simply Business company trip in Paris? Maybe).

Paris – Vegas

[Paul:] Let’s continue with some more insights from InfoSec. Dan, was the conference an opportunity to bond with other ‘SBers’ as well as a level-up opportunity?

[Dan:] I found the conference to be a great opportunity to get to know some of my colleagues that I hadn’t previously had any social interaction with. What really stood out for me was that we’d often end up discussing sessions we’d been to the day before, which sparked some really great conversations about how we could apply what we’d learned to our work at Simply Business.

For example, things like the possibility of creating AWS accounts on-demand for experimentation that clean themselves up, or how to secure inter-API communications, or make changes to our base images to increase self-service.

Security and durability

One of my favourite sessions was STG-331: Beyond eleven nines, lessons from the Amazon S3 culture of Durability (video), which talked about how the S3 team at AWS use threat modelling to identify risks to the durability of data in S3, and SEC-401: AWS encryption SDK multi-master key encryption, a workshop to protect data using multiple master encryption keys to solve problems such as decrypting objects in S3 that have been replicated across regions, without the latency involved in making a cross-region KMS call.

Security challenges

I also had the chance to attend the AWS Security Jam, which was really fun! In teams of four, we were given an AWS Capture the Flag environment, with different types of security challenges, mostly defensive. Some example challenges included applying configuration compliance rules to a kubernetes cluster in AWS to ensure it’s configured securely, performing analysis on a compromised AWS account to determine the actions taken by an attacker, and using an ELK stack and AWS WAF to identify the source of malicious traffic and a suitable signature to block only the malicious traffic without affecting benign traffic.

An engineering take on AWS re:Invent

[Paul:] Let’s now move on to product engineering. Quynh, tell us about your re:Invent experience from a development perspective.

[Quynh:] re:Invent had high expectations to live up to, having received rave reviews from the Simply Business team who attended in 2018. Simply Business sent a large delegation to Vegas in 2019, and I must say it did not disappoint. I came away more inspired, having learned so much in the space of one week than I have in a long time.

One of the highlights for many people attending the conference, myself included, was seeing AWS’s CEO Andy Jassy on stage – an electrifying presence, announcing new features of AWS and setting the tone for the industry for the year ahead.

Networking

On a more practical level, I happened to attend a heroes welcome dinner – a party AWS threw in honour of the AWS community heroes, which gave everyone a chance to socialise with them. I drew much inspiration from speaking to some of the heroes there. They were developers working in various industries who used AWS services in many different ways. What set them apart was their deep technical expertise and their willingness to share it with the community.

Technical tracks

The technical tracks at re:Invent were really effective too. Each event at the conference was ranked with a number – 100, 200, 300 or 400, indicating the level of technical background required by attendees in the subject, with 100 being suitable for beginners, and 400 aimed at experts. I spent a good amount of time in Vegas attending some 300/400 sessions in subjects that have been on my list of things to learn for some time. An intensive few days gave me a good working knowledge of machine learning, API Gateway, Event-Driven Architecture, Patterns and DynamoDB. The hands-on knowledge learned in Vegas could come in handy as our engineering teams at Simply Business look at ways of re-architecting our platform. Read more about how we architect our systems at Simply Business for scale, versioning, our domain and machine learning.

[Paul:] Angel, what other aspects did you take away for engineering?

[Angel:] I had no doubt that re:Invent was going to be interesting (it’s hard to be bored at re:Invent if you work in Tech), but I was curious to see how relevant the content would be for someone working on application code rather than in infrastructure or security. Fortunately, there was plenty of content for back-end engineers.

Event-driven architecture

It was particularly useful to hear from engineers working on Amazon products. For example, in Moving to event-driven architectures (video), Tim Bray gave an insight into how, and maybe more importantly, why amazon.com is built with an event-driven architecture. It was quite powerful to hear him describe the challenges and possible solutions while thinking about the products we build every day here at Simply Business.

Another highlight was the Introduction of the Builder’s Library (video), covering common patterns that Amazon uses to build and operate its own software. This included topics such as shuffle sharding, retries, backoff and jitter. I couldn’t help but think of the products we build at Simply Business, and how to architect them so that issues affecting a particular partner, product or customer can be isolated from the rest. The Amazon Builders’ Library is a work in progress but it’s definitely worth checking out.

Serverless

Serverless was a big topic, with plenty of content for all levels of experience, from diving into CI/CD for serverless applications (video), to getting to know how AWS Lambda works under the hood (video). Great for people moving towards making serverless a first class citizen in our architecture.

There was plenty of database content too, with deep dives into Neptune (AWS graph database) (video), and DocumentDB for those of us dealing with document databases. Oh, I particularly liked PartiQL’s (video) presentation, an SQL-based language designed to support relational and document DBs and even S3 buckets on the same query. Really cool stuff.

Re:Invent had so much more, tons of ML / AI content, integration patterns for microservices, and even sumo fighting! I can’t recommend it enough.

DevOps

[Paul:] Finally, let’s move to the experiences of my own team – DevOps. Jayne, I am sure you’ll be banging my head all year round with some crazy good stuff.

[Jayne:] The biggest thing I took away from re:Invent was that we’re doing a lot of the right things already at Simply Business. I was able to recognise our approach in many of the talks.

There’s also a lot more we could be doing, either because we’ve outgrown our initial approach or because AWS have released newer tools that would potentially be useful for us. I’d like us to take advantage of VPC sharing and centralise our networking control across all our VPCs, introduce more event-driven systems and automate some error remediation.

We already have a move to Transit Gateway in the pipeline, and we’ve been talking about working towards an account-per-dev setup for a while; it’s good to see these approaches being promoted by AWS and other companies.

There was a lot of focus on resilience and ‘correction of error’ meetings, which is what AWS calls post mortems, and how to improve processes and practices to empower developers to really understand how their code gets deployed and how to fix deployment problems. We can definitely do more of that.

I was impressed with the depth of the technical talks. I often find talks at conferences to be frustratingly light on technical detail and heavy on marketing, but almost every talk I went to was informative and technical. I was pleasantly surprised.

[Paul:] I see all this coming back into play sooner than later. Last but not least, Andy – the stage is yours to close with a bang.

[Andy:] As a DevOps engineer, I like to think my experience and knowledge of AWS is pretty good. But re:Invent really doesn’t shy away from diving deep into any service you might be interested in.

A couple of deep dive sessions stood out in particular, around Transit Gateway for multiple VPC architectures, AppMesh, enforcing compliance using Control Tower and EFS/EBS performance optimisation strategies. All of which went far beyond the knowledge you can gather either by reading AWS documentation or by taking online AWS courses.

Resilience engineering

A major take away for me though came from a talk and hands-on sessions that I attended on resilience engineering, which focused on continual resilience and performing resilience experiments on apps and architecture. The idea of chaos engineering is nothing new, but outside of tools such as Netflix’s Simian Army, the idea of performing resilience testing seemed somewhat obscure and was not particularly easy to get started with in an automated and controlled manner.

The sessions introduced key concepts around resilience engineering, probing applications and infrastructure to understand how they deal with specific failure, as well as introducing tools such as Chaos Toolkit to perform experiments in an automated, controlled and repeatable fashion. I see these tools becoming really useful to help us make Simply Business’s applications and infrastructure much more resilient to failure going forward, helping us to understand how they fail and how failure manifests.

There were also some really good discussions with members from different Simply Business teams and some really great takeaways over potential cross-team collaborations that I’m hoping can take place over the coming months; these are things that might never surface in everyday working or outside an event like re:Invent.

[Paul:] One last thing that’s been puzzling me. Vijay, rumour has it you’re the first person ever to return from Vegas with more money in their pocket than when they left. How did you manage it?

[Vijay:] Indeed! It’s a wonderful feeling. Getting back more money than you’ve lost in Vegas on the flight back. I’ll leave you to ponder how.

[Paul:] Intriguing. That concludes our AWS re:Invent journey. I hope you enjoyed it as much as the Simply Business team did in Vegas. See you at the next AWS re:Invent folks!