What are the biggest challenges of application architecture today? What are the most efficient and innovative solutions to those challenges?
Every week, the Big Ideas in App Architecture podcast seeks to extract answers to questions like those from the brightest minds in software development.
CTOs, CIOs, VPEs, and Architects from companies like Twilio, Greenhouse, Starburst, and others will share their career path, the current landscape of application architecture challenges today, and strategies they’ve employed to build modern, resilient, data-intensive apps and services.
The first episode features Ken Pickering, SVP of Engineering at Starburst Data. In this episode Ken discusses the marriage of Trino and Presto and the problem Starburst is building to solve. Here are three big ideas that came up in his conversation with our host, Tim Veil.
Separation of Compute and Storage
The idea of separating storage from compute has been part of the infrastructure dialogue for years because of the fundamental limitations of having those two things bound together. As someone building a platform to help businesses glean insights from all their data, Ken Pickering brings a unique Tino/Presto perspective.
The separation of compute and storage makes the concept of a data lake more functional. Data Lakes are an exceptionally cheap way to store data but the question is how to leverage all that data. Because a lot of companies eventually route their data to a data lake but it’s not the source of truth. Which makes it more difficult to find ways to improve performance and find business value from a lot of that data. Watch Ken explain two solutions to this challenge in this short excerpt from the podcast:
Hybrid/Multi-Cloud approach
“At the end of the day, whether you’re running in multiple clouds for things like resiliency and failover or playing the vendors off of each other for cost reasons - multi-cloud is a huge use case.” -Ken Pickering
Ken’s perspective is that hybrid/multi-cloud architecture is an “uncomfortable reality” that people have to accept despite the economics of it being challenging. He concedes that it isn’t absolutely necessary to build a cloud agnostic product, but if you want to solve problems at scale then you don’t have much choice.
One multi-cloud mistake he suggests people avoid is running a siloed product across all three clouds. It can be daunting to try and work across all three clouds to create one coherent system, but there are tools that can help.
“But I also think 10 years from now, if you’re thinking long term, people are going to want to treat CSPs like a commodity because no companies like being locked in, no company likes being 100% dependent on another business…”.
For context, Starburst initially launched their product in all three public clouds. And an eventual goal of theirs is solving multi-cloud hybrid cloud analytics.
Make “two-way door” architectural decisions
Ken had some sage advice for architects at growth stage companies:
“If you’re not actively making architectural mistakes almost every other day, I don’t know what you’re doing. Because, in all honesty, you’re proceeding in an unclear space and you’re not a hundred percent sure where your business is going to go. And so you make decisions. And I think a lot of the ways that we make decisions technically is we try to make sure we’re going through two-way doors instead of one-way doors. So we can revisit decisions later.”
Applications grow out of technologies and vendors. It’s usually a good problem to have. It means you’re successful. Ken shared an example from his time building the travel platform, Hopper:
“We were super reliant on Kafka, which over time really got progressively, progressively more complicated. We forked the open source, and we wrote our own client, we did a lot of stuff to make it keep working. We knew we had to unwind that decision at some point…at the beginning it was absolutely the right decision, but at the end of the day, eventually you’re like, “Okay, we are way beyond where we intended to be here and we’ve made it work, but what’s the go forward strategy?”
This kind of approach lends itself well to Ken’s philosophy about proving concept before trying to make absolutely perfect technical decisions:
“A lot of what I coach my teams to do is to prove the concept first. Let’s prove an idea, let’s prove this business idea is worth investment. Don’t lose sight of the eventual goal and let’s not make actions that are counter to where we want to be. But you don’t have to design a bulletproof system if we don’t even know it’s going to work.”
Coming up on the Big Ideas in App Architecture podcast
In the most recent podcast episode we talked with Mike Boufford, the chief technology officer at Greenhouse. Mike was actually the first engineering hire at Greenhouse and pushed their very first line of code. Since then he’s watched the engineering team grow to ~150 people and has helped the company evolve and flourish.
Following that episode we’re excited to connect with Gregg Mojica, the CTO and co-founder of Alloy Automation. Gregg’s insight into the proper use cases for leveraging serverless tools and serverless architecture to increase efficiency is truly cutting edge.
You can click here to subscribe to the podcast. And yes, we’d appreciate a rating and a review!
