Tuesday, April 13, 2021
  • Setup menu at Appearance » Menus and assign menu to Top Bar Navigation
Advertisement
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
No Result
View All Result
Home Big Data

Azure Synapse Analytics combines data warehouse, lake and pipelines

November 5, 2019
in Big Data
Azure Synapse Analytics combines data warehouse, lake and pipelines
585
SHARES
3.3k
VIEWS
Share on FacebookShare on Twitter

Microsoft Ignite 2019: Hybrid 2.0, Azure, Chromium Edge
ZDNet’s Larry Dignan recaps the highlights from Microsoft’s Ignite 2019 with Mary Jo Foley.

The first generation of Azure SQL Data Warehouse (SQL DW) was announced in 2015, and SQL DW “Gen 2” reached general availability in 2018. Today, at its Ignite confab on Orlando, Microsoft is announcing Synapse Analytics, essentially the third generation of SQL DW, along with new capabilities in preview. In general, Synapse Analytics seeks to unify an array of analytics workloads, including data warehouse, data lake, machine learning and the data pipelines that act as the mortar between those bricks.

You might also like

Weaviate is an open-source search engine powered by ML, vectors, graphs, and GraphQL

MinIO simplifies onramps to do-it-yourself hybrid cloud object storage

Trifacta goes all in on the cloud

Also read: Microsoft BUILDs its cloud Big Data story
Also read: Azure SQL Data Warehouse “Gen 2”: Microsoft’s shot across Amazon’s bow

Break it down for me

In a briefing with ZDNet, Daniel Yu, Microsoft’s Director Products – Azure Data and Artificial Intelligence and Charles Feddersen, Principal Group Program Manager – Azure SQL Data Warehouse, went through the details of Microsoft’s bold new unified analytics offering. Based on that briefing, my understanding of the transition from SQL DW to Synapse boils down to three pillars:

  1. The core data warehouse engine has been revved, with new features to compete with other cloud data warehouse platforms, including the ability to accommodate workloads through explicitly provisioned or on-demand (serverless) infrastructure, each with its associated pricing model
  2. The integration of Apache Spark (the open source flavor, and not Azure Databricks) and Azure Data Lake Storage (ADLS) to accommodate data lake workloads
  3. A unified Web user interface, called Azure Synapse studio, the provides control over both the data warehouse and data lake sides of Synapse, along with Azure Data Factory, to accommodate data prep and data management

Also read: Databricks comes to Microsoft Azure
Also read: Azure Data Factory v2: Hands-on overview

Spark integration, and more

The integration of Apache Spark seems to be more than just a “bundling” of the open source big data analytics framework. For example, when a Synapse cluster is provisioned, ADLS capacity — which can store Spark SQL tables — is requisitioned along with it (as is Azure Data Factory). Spark SQL tables are immediately query-able from the SQL-Server based T-SQL language, without first requiring explicit commands like CREATE EXTERNAL TABLE. The engine these queries leverage apparently integrates natively with data files stored in Apache Parquet format.

Such a feature will serve as a close competitor to Amazon Web Services’ Athena service, which provides SQL query over data in S3. Beyond that capability, however, Azure Synapse studio integrates a notebook experience, ostensibly accommodating the development and execution of Python, Scala and native Spark SQL code blocks. Spark integration also means that Synapse can handle machine learning workloads, by virtue of Spark MLlib.

Beyond Spark ML, Microsoft is also discussing integration with Azure Machine Learning, Power BI, Azure Data Share and applications/services that support the Open Data Initiative (based on Microsoft’s Common Data Model), though with fewer specifics. Those integrations will likely gel over time, and while the Synapse brand launches today, the new features that accompany it are being rolled out only in preview form.

Also read: Microsoft, Adobe and SAP are out to prove the Open Data Initiative is ‘open’

A fork in the SQL Server-Spark road?

Interestingly, the on-premises SQL Server product, from whose engine and Transact SQL language Synapse Analytics can trace its heritage, is also launching a new version today (SQL Server 2019 — which I cover in a separate post) that, with a feature called Big Data Clusters (BDC) also integrates Apache Spark, and data lake workloads. And despite SQL Server’s on-premises identity, BDC is completely based on Kubernetes container orchestration, which is implemented particularly well by Azure Kubernetes Service (AKS).

Also read: The big data odyssey of SQL Server 2019, and more data and AI news from Microsoft Ignite

Effectively, this means Microsoft is, on the same day and at the same event, launching two new options for combining SQL Server technology with Apache Spark, and both can run on Azure. Meanwhile, the two are implemented differently. And while Synapse has its Azure Synapse studio, SQL Server 2019 offers a notebook-capable, cross-platform (Windows/macOS/Linux) desktop user interface for database and data lake workloads, called Azure Data Studio.

This bifurcated path for Spark integration and tooling is bound to cause customer confusion, unfortunately. And the offering of yet another Apache Spark implementation on Azure, separate from Azure Databricks, may pose difficulties of its own, especially since Microsoft lists Databricks as one of its partners for Synapse.

There are important differences between all these services, though. SQL Server is geared primarily towards OLTP (Online Transactional Processing) requirements; Databricks shines in the realms of data engineering and machine learning; Synapse is the service you’ll want if MPP (massively parallel processing) data warehouse analytics are front-and-center for your needs. The fact that Spark and data lakes cut across all three of these just shows how important that technology and analytics model, respectively, have become.

Brust is a Microsoft Data Platform MVP and has done work for the Microsoft Advanced Analytics team.

Credit: Zdnet

Previous Post

2 Reasons Why Investor Returns Will Diminish Over the Next Decade

Next Post

Your WordPress site is at risk: These precautions and plugins can keep it secure

Related Posts

Weaviate is an open-source search engine powered by ML, vectors, graphs, and GraphQL
Big Data

Weaviate is an open-source search engine powered by ML, vectors, graphs, and GraphQL

April 8, 2021
MinIO simplifies onramps to do-it-yourself hybrid cloud object storage
Big Data

MinIO simplifies onramps to do-it-yourself hybrid cloud object storage

April 7, 2021
Trifacta goes all in on the cloud
Big Data

Trifacta goes all in on the cloud

April 6, 2021
Cloudera Data Platform hits Google Cloud
Big Data

Cloudera Data Platform hits Google Cloud

March 31, 2021
Cloudera fills gap in streaming platform with SQL
Big Data

Cloudera fills gap in streaming platform with SQL

March 31, 2021
Next Post
Your WordPress site is at risk: These precautions and plugins can keep it secure

Your WordPress site is at risk: These precautions and plugins can keep it secure

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recommended

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

January 6, 2019
Microsoft, Google Use Artificial Intelligence to Fight Hackers

Microsoft, Google Use Artificial Intelligence to Fight Hackers

January 6, 2019

Categories

  • Artificial Intelligence
  • Big Data
  • Blockchain
  • Crypto News
  • Data Science
  • Digital Marketing
  • Internet Privacy
  • Internet Security
  • Learn to Code
  • Machine Learning
  • Marketing Technology
  • Neural Networks
  • Technology Companies

Don't miss it

Brave browser disables Google’s FLoC tracking system
Internet Security

Brave browser disables Google’s FLoC tracking system

April 13, 2021
New NAME:WRECK Vulnerabilities Impact Nearly 100 Million IoT Devices
Internet Privacy

New NAME:WRECK Vulnerabilities Impact Nearly 100 Million IoT Devices

April 13, 2021
Machine Learning Approach In Fantasy Sports: Cricket
Machine Learning

Machine Learning Approach In Fantasy Sports: Cricket

April 13, 2021
These new vulnerabilities put millions of IoT devices at risk, so patch now
Internet Security

These new vulnerabilities put millions of IoT devices at risk, so patch now

April 13, 2021
BRATA Malware Poses as Android Security Scanners on Google Play Store
Internet Privacy

BRATA Malware Poses as Android Security Scanners on Google Play Store

April 13, 2021
6 Limitations of Desktop System That QuickBooks Hosting Helps Overcome
Data Science

6 Limitations of Desktop System That QuickBooks Hosting Helps Overcome

April 13, 2021
NikolaNews

NikolaNews.com is an online News Portal which aims to share news about blockchain, AI, Big Data, and Data Privacy and more!

What’s New Here?

  • Brave browser disables Google’s FLoC tracking system April 13, 2021
  • New NAME:WRECK Vulnerabilities Impact Nearly 100 Million IoT Devices April 13, 2021
  • Machine Learning Approach In Fantasy Sports: Cricket April 13, 2021
  • These new vulnerabilities put millions of IoT devices at risk, so patch now April 13, 2021

Subscribe to get more!

© 2019 NikolaNews.com - Global Tech Updates

No Result
View All Result
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News

© 2019 NikolaNews.com - Global Tech Updates