In 2013, Amazon Net Companies revolutionized the info warehousing trade by launching Amazon Redshift, the primary fully-managed, petabyte-scale, enterprise-grade cloud knowledge warehouse. Amazon Redshift made it easy and cost-effective to effectively analyze massive volumes of knowledge utilizing present enterprise intelligence instruments. This cloud service was a big leap from the normal knowledge warehousing options, which had been costly, not elastic, and required important experience to tune and function. Since then, buyer calls for for higher scale, larger throughput, and agility in dealing with all kinds of adjusting, however more and more enterprise crucial analytics and machine studying use circumstances has exploded, and we now have been conserving tempo. Right this moment, tens of 1000’s of shoppers use Amazon Redshift in AWS world infrastructure to collectively course of exabytes of knowledge each day and employs Amazon Redshift as a key element of their knowledge structure to drive use circumstances from typical dashboarding to self-service analytics, real-time analytics, machine studying, knowledge sharing and monetization, and extra
The developments to Amazon Redshift introduced at AWS re:Invent 2023 additional accelerates modernization of your cloud analytics environments, conserving our core tenet that will help you obtain the very best price-performance at any scale. These bulletins drive ahead the AWS Zero-ETL imaginative and prescient to unify all of your knowledge, enabling you to raised maximize the worth of your knowledge with complete analytics and ML capabilities, and innovate sooner with safe knowledge collaboration inside and throughout organizations. From price-performance enhancements to zero-ETL, to generative AI capabilities, we now have one thing for everybody. Let’s dive into the highlights.
Modernizing analytics for scale, efficiency, and reliability
“Our migration from legacy on-premises platform to Amazon Redshift permits us to ingest knowledge 88% sooner, question knowledge 3x sooner, and cargo each day knowledge to the cloud 6x sooner. Amazon Redshift enabled us to optimize efficiency, availability, and reliability—considerably easing operational complexity, whereas rising the rate of our end-users’ decision-making expertise on the Fab flooring.”
– Sunil Narayan, Sr Dir, Analytics at GlobalFoundries
Diligently driving the very best price-performance at scale with new enhancements
Since day 1, Amazon Redshift has been constructing progressive capabilities that will help you get to optimum efficiency, whereas conserving prices decrease. Amazon Redshift continues to steer on the price-performance entrance with as much as 6x higher price-performance than different cloud knowledge warehouse and for sprint boarding purposes with excessive concurrency and low latency. We carefully analyze question patterns within the fleet and search for alternatives to drive customer-focused innovation. For instance, earlier within the yr, we introduced pace ups for string-based knowledge processing as much as 63x in comparison with different compression encodings akin to LZO (Lempel-Ziv-Oberhumer) or ZStandard. At AWS re:Invent 2023, we launched extra efficiency enhancements in question planning and execution akin to enhanced bloom filters , question rewrites, and assist for write operations in auto scaling . For extra details about efficiency enchancment capabilities, seek advice from the record of bulletins beneath.
Amazon Redshift Serverless is extra clever than ever with new AI-driven scaling and optimizations
Talking of price-performance, new subsequent era AI-driven scaling and optimizations capabilities in Amazon Redshift Serverless can ship as much as 10x higher price-performance for variable workloads (based mostly on inside testing), with out handbook intervention. Amazon Redshift Serverless, usually accessible since 2021, lets you run and scale analytics with out having to provision and handle the info warehouse. Since GA, Redshift Serverless executed over a billion queries to energy knowledge insights for 1000’s of shoppers. With these new AI optimizations, Amazon Redshift Serverless scales proactively and mechanically with workload adjustments throughout all key dimensions —akin to knowledge quantity, concurrent customers, and question complexity. You simply specify your required price-performance targets to both optimize for value or optimize for efficiency or balanced and serverless does the remainder. Be taught extra about further enhancements in Redshift Serverless, below the record of bulletins beneath.
Multi-data warehouse writes by knowledge sharing
Information sharing is a extensively adopted function in Amazon Redshift with clients operating tens of tens of millions of queries on shared knowledge each day. Clients share reside transactionally constant knowledge inside and throughout organizations and areas for learn functions with out knowledge copies or knowledge motion. Clients are utilizing knowledge sharing to modernize their analytics architectures from monolithic architectures to multi-cluster, knowledge mesh deployments that allow seamless and safe entry throughout organizations to drive knowledge collaboration and highly effective insights. At AWS re:Invent 2023, we prolonged knowledge sharing capabilities to launch multi-data warehouse writes in preview. Now you can begin writing to Redshift databases from different Redshift knowledge warehouses in only a few clicks, additional enabling knowledge collaboration, versatile scaling of compute for ETL/knowledge processing workloads by including warehouses of various varieties and sizes based mostly on price-performance wants. Expertise larger transparency of compute utilization as every warehouse is billed for its personal compute and consequently maintain your prices below management.
Multidimensional knowledge layouts
Amazon Redshift affords trade main predictive optimizations that repeatedly monitor your workloads and seamlessly speed up efficiency and maximize concurrency by adjusting knowledge format and compute administration as you employ the info warehouse extra. Along with the highly effective optimizations Redshift already affords, akin to Automated Desk Type, Automated kind and distribution keys, we’re introducing Multidimensional Information Layouts, a brand new highly effective desk sorting mechanism that improves efficiency of repetitive queries by mechanically sorting knowledge based mostly on the incoming question filters (for instance: Gross sales in a selected area). This technique considerably accelerates the efficiency of desk scans in comparison with conventional strategies.
Unifying all of your knowledge with zero-ETL approaches
“Utilizing the Aurora MySQL zero-ETL integration, we expertise close to real-time knowledge synchronization between Aurora MySQL databases and Amazon Redshift, making it doable to construct an evaluation atmosphere in simply three hours as a substitute of the month of developer time it used to take earlier than”
– MoneyForward
JOYME makes use of Amazon Redshift’s streaming ingestion and different Amazon companies for threat management over customers’ monetary exercise akin to recharge, refund, and rewards.
“With Redshift, we’re in a position to view threat counterparts and knowledge in close to actual time—
as a substitute of on an hourly foundation. Redshift considerably improved our enterprise ROI effectivity.”– PengBo Yang, CTO, JOYME
Information pipelines will be difficult and expensive to construct and handle and might create hours-long delays to acquire transactional knowledge for analytics. These delays can result in missed enterprise alternatives, particularly when the insights derived from analytics on transactional knowledge are related for less than a restricted period of time. Amazon Redshift employs AWS’s zero-ETL strategy that permits interoperability and integration between the info warehouse and operational databases and even your streaming knowledge companies, in order that the info is definitely and mechanically ingested into the warehouse for you, or you’ll be able to entry the info in place, the place it lives.
Zero-ETL integrations with operational databases
We delivered zero-ETL integration between Amazon Aurora MySQL Amazon Redshift (normal availability) this yr, to allow close to real-time analytics and machine studying (ML) utilizing Amazon Redshift on petabytes of transactional knowledge from Amazon Aurora. Inside seconds of transactional knowledge being written into Aurora, the info is out there in Amazon Redshift, so that you don’t must construct and keep advanced knowledge pipelines to carry out extract, remodel, and cargo (ETL) operations. At AWS re:Invent, we prolonged zero-ETL integration to further sources particularly Aurora PostgreSQL, Dynamo DB, and Amazon RDS MySQL. Zero-ETL integration additionally lets you load and analyze knowledge from a number of operational database clusters in a brand new or present Amazon Redshift occasion to derive holistic insights throughout many purposes.
Information lake querying with assist for Apache Iceberg tables
Amazon Redshift permits clients to run a variety of workloads on knowledge warehouse and knowledge lakes utilizing its assist for varied open file and desk codecs. At AWS re:Invent, we introduced the overall availability of assist for Apache Iceberg tables, so you’ll be able to simply entry your Apache Iceberg tables in your knowledge lake from Amazon Redshift and be part of it with the info in your knowledge warehouse when wanted. Use one click on to entry your knowledge lake tables utilizing auto-mounted AWS Glue knowledge catalogs on Amazon Redshift for a simplified expertise. Now we have improved knowledge lake question efficiency by integrating with AWS Glue statistics and introduce preview of incremental refresh for materialized views on knowledge lake knowledge to speed up repeated queries.
Be taught extra in regards to the zero-ETL integrations, knowledge lake efficiency enhancements, and different bulletins beneath.
Maximize worth with complete analytics and ML capabilities
“Amazon Redshift is among the most essential instruments we had in rising Jobcase as an organization.”
– Ajay Joshi, Distinguished Engineer, Jobcase
With all of your knowledge built-in and accessible, you’ll be able to simply construct and run close to real-time analytics to AI/ML/Generative AI purposes. Right here’s a few highlights from this week and for the complete record, see beneath.
Amazon Q Generative SQL functionality
Question Editor, an out-of-the-box web-based SQL expertise in Amazon Redshift is a well-liked device for knowledge exploration, visible evaluation, and knowledge collaboration. At AWS re:Invent, we launched Amazon Q Generative SQL capabilities in Amazon Redshift Question Editor (preview), to simplify question authoring and enhance your productiveness by permitting you to specific queries in pure language and obtain SQL code suggestions. Generative SQL makes use of AI to investigate consumer intent, question patterns, and schema metadata to determine frequent SQL question patterns straight permitting you to get insights sooner in a conversational format with out intensive data of your group’s advanced database metadata.
Amazon Redshift ML massive language mannequin (LLM) integration
Amazon Redshift ML permits clients to create, prepare, and deploy machine studying fashions utilizing acquainted SQL instructions. Clients use Redshift ML to run a median of over 10 billion predictions a day inside their knowledge warehouses. At AWS re:Invent, we introduced assist for LLMs as preview. Now, you should use pre-trained open supply LLMs in Amazon SageMaker JumpStart as a part of Redshift ML, permitting you to convey the ability of LLMs to analytics. For instance, you may make inferences in your product suggestions knowledge in Amazon Redshift, use LLMs to summarize suggestions, carry out entity extraction, sentiment evaluation and product suggestions classification.
Innovate sooner with safe knowledge collaboration inside and throughout the organizations
“Hundreds of thousands of firms use Stripe’s software program and APIs to just accept funds, ship payouts, and handle their companies on-line. Entry to their Stripe knowledge through main knowledge warehouses like Amazon Redshift has been a prime request from our clients. Our clients wanted safe, quick, and built-in analytics at scale with out constructing advanced knowledge pipelines or shifting and copying knowledge round. With Stripe Information Pipeline for Amazon Redshift, we’re serving to our clients arrange a direct and dependable knowledge pipeline in a couple of clicks. Stripe Information Pipeline permits our clients to mechanically share their full, up-to-date Stripe knowledge with their Amazon Redshift knowledge warehouse, and take their enterprise analytics and reporting to the subsequent stage.”
– Tony Petrossian, Head of Engineering, Income & Monetary Administration at Stripe
With Amazon Redshift, you’ll be able to simply and securely share knowledge and collaborate irrespective of the place your groups or knowledge is situated. And have the arrogance that your knowledge is safe irrespective of the place you use or how extremely regulated your industries are. Now we have enabled nice grained permissions, a simple authentication expertise with single sign-on in your organizational id—all offered at no further value to you.
Unified id with IAM id middle integration
We introduced Amazon Redshift integration with AWS IAM Id Heart to allow organizations to assist trusted id propagation between Amazon QuickSight,, Amazon Redshift Question Editor, and Amazon Redshift, . Clients can use their group identities to entry Amazon Redshift in a single sign-on expertise utilizing third get together id suppliers (IdP), akin to Microsoft Entra ID, Okta, Ping, OneLogin, and so on. from Amazon QuickSight and Amazon Redshift Question Editor. Directors can use third-party id supplier customers and teams to handle nice grained entry to knowledge throughout companies and audit consumer stage entry in AWS CloudTrail. With trusted id propagation, a consumer’s id is handed seamlessly between Amazon QuickSight, Amazon Redshift decreasing time to insights and enabling a friction free analytics expertise.
For the complete set of bulletins, see the next:
-
Modernizing analytics for scale, efficiency, and reliability
-
Unifying all of your knowledge with zero-ETL approaches
-
Maximize worth with complete analytics and ML capabilities
-
Innovate sooner with safe knowledge collaboration inside and throughout the organizations
Be taught extra: https://aws.amazon.com/redshift
In regards to the authors
Neeraja Rentachintala is a Principal Product Supervisor with Amazon Redshift. Neeraja is a seasoned Product Administration and GTM chief, bringing over 20 years of expertise in product imaginative and prescient, technique and management roles in knowledge merchandise and platforms. Neeraja delivered merchandise in analytics, databases, knowledge Integration, software integration, AI/Machine Studying, massive scale distributed programs throughout On-Premise and Cloud, serving Fortune 500 firms as a part of ventures together with MapR (acquired by HPE), Microsoft SQL Server, Oracle, Informatica and Expedia.com.
Sunaina AbdulSalah leads product advertising for Amazon Redshift. She focuses on educating clients in regards to the affect of knowledge warehousing and analytics and sharing AWS buyer tales. She has a deep background in advertising and GTM features within the B2B know-how and cloud computing domains. Exterior of labor, she spends time together with her household and pals and enjoys touring.
