Are Hadoop and Spark Important for Data Science?

Data science is very new concept. Many institutes around the world found data science as source of their income. Apart from managing data, data science deals with new research ideas from which data can be converted into actionable format.

Although technical training for data science deals with database, data mining, machine learning techniques, data warehousing, SAP and much more. However, a new idea for development gives life to data science project.

Data science project includes

  • Automatic car
  • Internet of Things
  • Google Glass
  • Gesture recognition based brain games
  • Virtual reality
  • self running cycle
and much more.

There are ample of local problems of society which can be solved using data science subject.

Understanding of existing projects and research areas gives wide scope of development and improvement in this field. 

It is not good idea to go for Hadoop and Spark type projects initially. One can always start with existing machine learning techniques and existing algorithms to solve any real-time problem.

Data science projects are quite different from those of developing web applications and window applications. It is some beyond copy paste projects. It is all about imagination. The era of science and information technology has proved that you can create anything that you can imagine.

Belief is the only key of success in this area.

Many people frame beautiful and really challenging problems but unable to solve thinking that probably they are unaware of technology. However, maximum products deal with efforts and willingness to solve.

Large amount of data or big data shall need Hadoop and Spark type skill set. But again, nothing is impossible to a willing heart.

Dedicated to the regular reader 'Neha' 

Thanks for your query!

Comments

Post a Comment

Popular posts from this blog

Analysis and Research trends using Word Co-occurrence Network

Image
Co-occurrence networks   are basically used to provide and give a graphic visualization of potential relationships between people, community, organizations, concepts or other entities represented within the form of written material. The generation and visualization of   co-occurrence   networks has become practical with the advent of electronically stored and save the text amenable to   text mining . By way of definition, co-occurrence networks are the collective interconnection of terms based on their paired presence within a specified unit of text. Networks are generated by connecting pairs of terms using a set of criteria defining co-occurrence. For example, terms A and B may be said to “co-occur” if they both appear in a particular article. Another article may contain terms B and C. Linking A to B and B to C creates a co-occurrence network of these three terms. Rules to define co-occurrence within a   text corpus   can be set according to desired criteria. For example, more str
Read more

Significance of Woman in Data Science

Image
Research scientists, regardless of gender, are doing great in this field. This has been evidenced by a great conference " Global Women in Data Science " hosted by Stanford University (livestream) for 75+ locations all over the world on 3rd February 2017. Many talented academic researchers and practitioners have successful connected to make their dreams and goals burst into reality. It has been sponsored by Microsoft, Google, WalmartLabs, SAP and many more A Grade firms.  Fig 1: Global Women in Data Science Another great initiative, Women in Machine Learning   has been hosted on Facebook. Many female potential researchers join hands for research in machine learning. Many of them have achieved great milestones and many are still on the way.  The world is full of talented, successful and fantastic female speakers Women in Machine Learning and Data science are often un-noticable by traditional recruiters and we can turn it down. Another initiative at Meetup.c
Read more

Different types of clustering for textual documents

Clustering Model 1: Nearest Neighbour Algorithms: KD-Trees and Locality Sensitive Hashing Core Machine Learning Concepts: Distance Metrics and Approximation Algorithms Problem Domain: Finding similar documents Nearest neighbour: Find distance between all other documents and query document. Retrieve the document which is closest Critical Component Document Representation Computing Distances Scaling to large dataset: KD Trees - Not appropriate for very high dimensional documents Thus, Locality sensitive hashing is used for approximate nearest neighbour search. Clustering Model 2: Capturing uncertainty in clustering: Mixture Models Any document can be related to t i topic by x%. Can learn users topic preferences Making soft assignments to clusters -Expectation Maximization Clustering Model 3: Latent dirichlet allocation Mixed membership Probability of words in vocabulary Unsupervised learning task
Read more