Docker container-based big data processing system in multiple clouds for everyone

Nitin Naik

Research output: Chapter in Book/Report/Conference proceedingConference contribution

751 Downloads (Pure)


Big data processing is progressively becoming essential for everyone to extract the meaningful information from their large volume of data irrespective of types of users and their application areas. Big data processing is a broad term and includes several operations such as the storage, cleaning, organization, modelling, analysis and presentation of data at a scale and efficiency. For ordinary users, the significant challenges are the requirement of the powerful data processing system and its provisioning, installation of complex big data analytics and difficulty in their usage. Docker is a container-based virtualization technology and it has recently introduced Docker Swarm for the development of various types of multi-cloud distributed systems, which can be helpful in solving all above problems for ordinary users. However, Docker is predominantly used in the software development industry, and less focus is given to the data processing aspect of this container-based technology. Therefore, this paper proposes the Docker container-based big data processing system in multiple clouds for everyone, which explores another potential dimension of Docker for big data analysis. This Docker container-based system is an inexpensive and user-friendly framework for everyone who has the knowledge of basic IT skills. Additionally, it can be easily developed on a single machine, multiple machines or multiple clouds. This paper demonstrates the architectural design and simulated development of the proposed Docker container-based big data processing system in multiple clouds. Subsequently, it illustrates the automated provisioning of big data clusters using two popular big data analytics, Hadoop and Pachyderm (without Hadoop) including the Web-based GUI interface Hue for easy data processing in Hadoop.
Original languageEnglish
Title of host publication2017 IEEE International Systems Engineering Symposium (ISSE)
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages7
ISBN (Electronic)978-1538634035
ISBN (Print)978-1538634042
Publication statusPublished - 30 Oct 2017
Event2017 IEEE International Symposium on Systems Engineering - Vienna, Austria
Duration: 11 Oct 201713 Oct 2017


Conference2017 IEEE International Symposium on Systems Engineering


  • big data
  • containers
  • cloud computing
  • virtulization
  • data models
  • buildings


Dive into the research topics of 'Docker container-based big data processing system in multiple clouds for everyone'. Together they form a unique fingerprint.

Cite this