Friday, January 15, 2021
  • Setup menu at Appearance » Menus and assign menu to Top Bar Navigation
Advertisement
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
No Result
View All Result
Home Big Data

Google announces Kubernetes Operator for Apache Spark

January 30, 2019
in Big Data
Google announces Kubernetes Operator for Apache Spark
586
SHARES
3.3k
VIEWS
Share on FacebookShare on Twitter

Credit: ZDnet

Apache Spark is a hugely popular execution framework for running data engineering and machine learning workloads. It powers the Databricks platform and is available in both on-premises and cloud-based Hadoop services, like Azure HDInsight, Amazon EMR and Google Cloud Dataproc. It can run on Mesos clusters too.

You might also like

Oracle Database 21c spotlights in-memory processing and ML, adds new low-code APEX cloud service

Grafana Cloud adds free tier

Cockroach Labs closes $160M Series E funding round

But what of you just want to run your Spark workloads on a Kubernetres (k8s) cluster sans Mesos, and without the Hadoop YARN strings attached? While Spark first added Kubernetes-specific features in its 2.3 release, and improved them in 2.4, getting Spark to run natively on k8s, in a fully integrated fashion, can still be a challenge.

Kube operator

Today, Google, which created Kubernetes in the first place, is announcing the beta release of the Kubernetes Operator for Apache Spark — “Spark Operator” for short. Spark Operator allows Spark to run natively on k8s clusters and thus allows Spark applications — be they purposed for analytics, data engineering or machine learning — to deploy to these clusters as they would any Spark instance.

Spark Operator’s Architecture


Credit: Google

According to Google, Spark Operator is a Kubernetes custom controller that uses custom resources for declarative specification of Spark applications; it also supports automatic restart and cron-based, scheduled applications. Further, developers, data engineers and data scientists can create declarative specifications that describe their Spark applications and use native Kubernetes tooling (e.g. kubectl) to manage their applications.

Get yours today

Spark Operator is available on the Google Cloud Platform (GCP) Marketplace for Kubernetes, in the form of Google Click to Deploy containers, for easy deployment to Google Kubernetes Engine (GKE). But Spark Operator is an open source project and can be deployed to any Kubernetes environment, and the project’s GitHub site provides Helm chart-based command line installation instructions.

It will be interesting to see if the likes of Amazon and Microsoft will endorse and offer simple deployment of the Spark Operator for their own Kubernetes services (Elastic Container Service/EKS and Azure Kubernetes Service/AKS, respectively). Doing so would be a great service to their customers who do not wish to have the overhead of an EMR, HDInsight or Databricks workspace and cluster.

Et tu, Hadoop?

Since many non-Databricks Spark clusters have in fact run on Hadoop, the release of Spark Operator begs the question of whether Hadoop’s influence is waning. But the Hadoop team isn’t resting either. For example, the Open Hybrid Architecture Initiative is focused on the containerization of Hadoop. Furthermore, Hadoop 3.2 was released just last week and, among other features, includes native support for Tensorflow, new connectivity to Azure Data Lake Storage Gen2 and enhanced connectivity to Amazon S3 storage.

As usual, such robust competition benefits customers.

Credit: ZDnet

Previous Post

Intel Bets Big On Kubernetes For Nauta Deep Learning Platform

Next Post

Facebook Paid Teens $20 to Install 'Research' App That Collects Private Data

Related Posts

Oracle Database 21c spotlights in-memory processing and ML, adds new low-code APEX cloud service
Big Data

Oracle Database 21c spotlights in-memory processing and ML, adds new low-code APEX cloud service

January 14, 2021
Grafana Cloud adds free tier
Big Data

Grafana Cloud adds free tier

January 13, 2021
Ryuk gang estimated to have made more than $150 million from ransomware attacks
Big Data

Cockroach Labs closes $160M Series E funding round

January 12, 2021
2021 Technology trend review, part 1: Blockchain, Cloud, Open Source
Big Data

2021 Technology trend review, part 1: Blockchain, Cloud, Open Source

January 12, 2021
Informatica’s journey to the cloud
Big Data

Informatica’s journey to the cloud

January 7, 2021
Next Post
Facebook Paid Teens $20 to Install ‘Research’ App That Collects Private Data

Facebook Paid Teens $20 to Install 'Research' App That Collects Private Data

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recommended

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

January 6, 2019
Microsoft, Google Use Artificial Intelligence to Fight Hackers

Microsoft, Google Use Artificial Intelligence to Fight Hackers

January 6, 2019

Categories

  • Artificial Intelligence
  • Big Data
  • Blockchain
  • Crypto News
  • Data Science
  • Digital Marketing
  • Internet Privacy
  • Internet Security
  • Learn to Code
  • Machine Learning
  • Marketing Technology
  • Neural Networks
  • Technology Companies

Don't miss it

Toyota slapped with $180 million fine for violating Clean Air Act
Internet Security

Toyota slapped with $180 million fine for violating Clean Air Act

January 15, 2021
AI Research at Amazon: Brand Voice, Entanglement Frontier, Humor Recognition  
Artificial Intelligence

AI Research at Amazon: Brand Voice, Entanglement Frontier, Humor Recognition  

January 15, 2021
Machine Learning in Manufacturing Market 2020 Report Forecast By Global Industry Trends, Future Growth, Regional Overview
Machine Learning

Machine Learning in Manufacturing Market 2020 Report Forecast By Global Industry Trends, Future Growth, Regional Overview

January 15, 2021
More than 10mil users installed Android apps that showed out-of-context ads
Internet Security

More than 10mil users installed Android apps that showed out-of-context ads

January 15, 2021
Chief Data Scientist Seen as Bridging CTO to Business Managers 
Artificial Intelligence

Chief Data Scientist Seen as Bridging CTO to Business Managers 

January 15, 2021
Data Science and Machine Learning Service Market size and Key Trends in terms of volume and value 2020-2025
Machine Learning

Data Science and Machine Learning Service Market size and Key Trends in terms of volume and value 2020-2025

January 15, 2021
NikolaNews

NikolaNews.com is an online News Portal which aims to share news about blockchain, AI, Big Data, and Data Privacy and more!

What’s New Here?

  • Toyota slapped with $180 million fine for violating Clean Air Act January 15, 2021
  • AI Research at Amazon: Brand Voice, Entanglement Frontier, Humor Recognition   January 15, 2021
  • Machine Learning in Manufacturing Market 2020 Report Forecast By Global Industry Trends, Future Growth, Regional Overview January 15, 2021
  • More than 10mil users installed Android apps that showed out-of-context ads January 15, 2021

Subscribe to get more!

© 2019 NikolaNews.com - Global Tech Updates

No Result
View All Result
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News

© 2019 NikolaNews.com - Global Tech Updates