IBM Commits to Advance Apache Spark

Apache Spark Framework is Hadoop was the most dominant engine for big data processing Enterprises using Hadoop and heavily relying on it for their analytical infrastructures.

IBM Commits to Advance Apache Spark

IBM Commits to Advance Apache Spark

IBM is now gearing innovation for Spark ecosystem as part of its commitment to driving new technologies in Big Data. It reveals company’s commitment to the open source projects as well. As expressed by the company, Spark as the Big Data processing engine has been most ground breaking open source project of recent times.
General Manager of the IBM analytics platform commented recently that “Spark is undoubtedly a force to be reckoned with in the Big Data ecosystem”. For developing next-generation applications collaboration with Databricks is the next inevitable step, as expressed by him.

Planning to incorporate Spark in core development platform

IBM is now having plans to bring Spark into core data platforms and utilize resources and engage developers for driving innovation around it. From now on Spark will be incorporated into the analytics and commerce platform of IBM.  By combining IBM data analytics with Apache Spark something bigger can be expected in data applications of future. It is more than just about accessing data. It is about developing data algorithms for actionable analytics. Below we provide some of the steps taken by the company as part of its commitment to Apache Spark.

  • To incorporate the advanced attributes of IBM SystemML machine learning into Apache Spark core project and to materialize this objective collaborating with Databricks.
  • Incorporate IBM Analytics for Apache Spark™ on IBM Bluemix
  • A new Spark Technology Center in San Francisco has been planned by the company for data science and developer community
  • Train and educate data scientists and data engineers extensively on Apache Spark. Presently, 1 million data scientists and data engineers are going to be educated on Apache Spark. The company for this purpose have partnered with major Apache Spark contributors like AMPLab, DataCamp, Galvanize, MetiStream and Big Data University MOOC.

Offering its own machine learning SystemML to open source community

Recently as part of its commitment to open source technologies, IBM decided to announce IBM SystemML, its machine learning technology as open source project. By announcing this as open source IBM actually helped Databricks to advance machine learning abilities of Spark.

SystemML which has been known as Apache SystemML offers developers ability to embed this in applications without requiring any prior expertise. This allows utilizing machine learning capability in a wide variety of scenarios on diverse computing platforms including the mainframe computers to smartphones.

Accelerating open source innovation for Apache Spark ecosystem

For accelerating innovation in open source environment for the Spark ecosystem, the company is going to take the below mentioned actions.

  • Spark is going to be incorporated into the core of IBM’s analytics as well as commerce platforms.
  • Spark will be used in IBM's Watson Health Cloud guaranteeing faster time to value in medical sectors and research facilities.  Medical service providers and research facilities now can access new fast paced analytics around health data of the population.
  • IBM committed SystemML machine learning technology to open source with the objective of collaborating with Databricks to make the machine learning ability of Spark more advanced.  
  • IBM Bluemix by incorporating Spark as a Cloud service will help app developers to load data, model it, and derive analytical output faster.
  • More than 3,500 new researchers and developers will be engaged in projects related to Spark by IBM. Besides opening more than dozen labs worldwide for research on Spark, IBM is going to open a Spark Ccnter in San Francisco to facilitate Data scientists and developers drive Spark based innovation in applications.
  • Through wide ranging partnerships with Spark companies and contributors such as AMPLab, DataCamp, MetiStream, Galvanize and Big Data University MOOC, IBM will educate a million developers on Spark.

How Spark is transforming business for IBM clients?

Spark has rapidly rose to popularity among developers and data scientists as a crucial BIG Data analytics platform. It came as a big help for organizations to integrate Big Data into their applications more easily. Naturally, to boost decision making process with latest analytics utilizing Spark powered applications is invincible now. Let us see here below some examples as to what extent and how Spark is transforming business for IBM clients.

  • Spark coupled up with IBM, is providing a greatly scalable platform for some real time transportation planning applications like Optibus.
  • IBM Analytics coupled with Spark is also helping business customers to utilize the potential of Big data to their advantage. Small and medium businesses now can tap the hidden analytical power of Big Data through the combined force of IBM and Spark.  Businesses that have found MapReduce as less efficient for meeting their analytics demand, are now finding this combination faster and more result driven.
  • By incorporating advanced machine learning capability of IBM SystemML  into Apache Spark the company provided a big impetus to research and development on Spark.