Optimizing costs and maximizing the value you get from using modern cloud-based data platforms is an urgent question for many enterprises. Across industries, organizations are moving their applications and processes to cloud-based infrastructures, saving money that would have gone to data centers, hardware and personnel. However, as you migrate workloads to the cloud and add new applications with more users, you may find your costs accelerate up. This can sometimes be a good thing especially if your business is growing rapidly, but without the proper visibility, you won’t know why your costs are rising and what you can do to optimize them. Now, more than ever, business leaders want more visibility into the drivers of unexpected costs, but they also want actionable direction on how to fix the root causes of those inefficiencies.
Cloud optimization delivers big savings. As more data workloads move to the cloud that data will become a larger portion of a company’s spend and draw more scrutiny from finance. According to Google, even minimal cloud optimization efforts can net a business as much as 10% savings per service within two weeks. Based on Bluesky’s customer experience, customers can get 20% cost reduction within the initial weeks of deployment.
Workload and cost optimization are even more challenging when it comes to data cloud platforms such as Snowflake - primarily because of its consumption-based pricing model. Pricing is mainly driven by credit consumption, which is mainly driven by compute and storage, which is mainly driven by the amount of tables in your environment, the SQL queries being run on those tables and the sizes. When optimizing costs, it’s important to focus on compute rather than storage. For most Snowflake customers, compute is by far the dominant factor, often an order of magnitude greater than storage and cloud services.
When IT, Finance and Data teams lack visibility into their data cloud infrastructure and processes, optimal performance and cost containment are hard to achieve. One fundamental cultural factor is that cost is not a priority for Data Eng teams. Today, most of your Data Eng team’s time is spent building data pipelines and writing queries instead of optimizing workloads. Engineering prioritizes performance for their end users and cost doesn’t naturally play into the picture. In addition to poor data cloud visibility, inefficient queries and inefficient warehouse management can cause enterprises to rack up unnecessary costs.
Lack of visibility - It’s not easy for users to find out the root cause of cost growth. The cost of the data cloud is associated with warehouses and storage, but users interact with queries and there is a non-trivial mapping from the warehouse to query costs.
Inefficient queries - The main responsibility of Data engineers and scientists is to leverage data to support the business. While some are good at writing efficient queries, many see that as a burden to their productivity. Additionally, there are so many queries accumulated over time that optimizing the query efficiency manually is a never-ending task.
Inefficient warehouse management - Warehouse management is often more art than science - there are many factors to be considered from latency and scalability to cost efficiency. In general, many users follow simplistic guidelines on warehouse settings which may leave room for big improvements due to the lack of tooling.
When query complexity changes, Data Eng must manually increase or decrease the size of the cluster running the Snowflake data warehouse. Snowflake lacks a resource manager to assign resources to queries based on their importance and overall system workload. Data Eng cannot focus on cost management, innovation and scaling if they need to constantly tune the data cloud platform. In order to reduce waste and achieve true data cloud optimization, capabilities like continuous monitoring, automation and intelligent recommendations are essential to keep infrastructure running optimally while minimizing costs and meeting SLA standards.
Data cloud cost management and optimization is a relatively new discipline and we are still figuring out the best way to do it. Many organizations are either not aware of optimization best practices or lack a structured approach for executing them to reduce costs and increase operational efficiency. Many organizations today take the approach of doing point-in-time optimizations - monthly, quarterly or when something spikes unexpectedly. As the infrastructure complexity of data cloud platforms such as Snowflake continues to rise with the amount of data, increased utilization of services, number of users, etc, Data and Finance professionals will need to shift their focus from visibility to optimization and consistency in data cloud spend.
It’s important that optimization is not just a set of isolated processes or tools. Optimization should be seen as a way of working and controlling your workloads over the long term to achieve the best value from your spend. Doing cost optimization after a workload is in production is more expensive than ensuring cost control is built in. Data leaders should recast workload optimization and cost management as a core business process by designing and executing a series of tasks to be performed throughout a workload’s life cycle. These tasks should include steps for continuously monitoring, identifying and resolving performance related issues as early as possible - shift optimization left.
Another best practice to become ever more efficient is to reduce the time actual humans have to spend discovering and fixing workload performance issues. Eventually we will move towards models of data cloud cost management and workload optimization that require minimal human intervention, shifting instead to more hands-off functions with human guardrails. Maybe one day, we will even have AI powerful enough to take over the whole process! That’s when we will truly have continuous efficiency and every data team will be able to balance costs without slowing down innovation.
Ultimately proper optimization of your Snowflake environment is the only viable route to preserving resources as you scale. Bluesky develops cloud-based software that enables any company to use its Data Cloud resources more efficiently and in turn, save costs while improving workload performance. Bluesky continuously monitors your Snowflake environment and streamlines the process to find, validate and implement workload optimizations across the query, warehouse and storage layers. Plus, Bluesky has an intuitive UI that doesn’t require a lot of technical expertise to use.
Bluesky recommendations are truly actionable - not just suggestions. For example, we identified optimization opportunities that reduced Certik’s overall Snowflake spend by ~20%. Bluesky identified a QA workload that would have cost more than $300K annually and shut it down; it found certain expressive SQL queries without the right filtering conditions and generated alternative query formulations to reduce the costs by more than 300X and improve query speed by 500X; it also computed optimal warehouse settings that led to further cost efficiency improvement.
Bluesky empowers Data Engineers to spend more time innovating with their Data Cloud and less time manually monitoring and tuning it. With Bluesky deployed, Certik’s Data Eng team is able to monitor their evolving Snowflake workloads, identify expensive and inefficient query patterns and react in a timely manner to avoid queries similar to the $300k one from happening again. Additionally, the team can focus on strategic initiatives that further maximize the ROI of Snowflake.
We built our solution with the goal of providing actionable insights so that Data Engineers, Analysts, Snowflake admins aren’t spending time deciphering the data. Rather, they’re acting on it. We are maniacally focused on helping our customers succeed, with most achieving performance improvements of 90% and cost savings of ~ 30% within a matter of weeks of onboarding. For more insight into the outcomes Bluesky helped these companies achieve, visit www.getbluesky.io/resources.
Ready to continuously discover opportunities to optimize your Data Cloud workloads? Contact Us