Data Flowchart Diagram
By Robin E-H. Hoard
Program Manager, H&S.CO – Climate
ID. https://orcid.org/0000-0001-7960-9780
This is a H&S.CO working paper
Number: 0000020 – Working copy
Date Issued: 11/18/2021
©
Table of Contents
STEPS in the Data Flowchart Diagram For the Intelligent Aide Climate (IAC) Model 12
Plate 1. Data Flowchart Diagram for the Intelligent Aide Climate (IAC) Model 19
Early works on this concept 20
Linux Hadoop Small Clusters 20
Open-Source & Copyrighted Documentation 21
Free and Open-Source Software 21
Other Licenses for use of software: 22
ABSTRACT
A way of developing vast amounts of targeted information for the project.
This system platform is a cross-breed, of a generic early warring system (EWS) changed to a climate and environmental changes warring system with a digital media reference library and limited communications subsystem.
In this data flowchart diagram plate, the Intelligent Aide Climate (IAC) Platform (Linux Ubuntu Hadoop cluster) is broken-down into four logical modules: Input and updates; Number crunching; Digital library and Graphical user interface for client interaction with the system.
INTRODUCTION
The following is an example of work I have been doing toward developing a climate and environmental change early warning system (EWS). This paper is meant to spark a discussion and I hope develop a community of interest on how a low cost system can fill a need for localized EWS to protect human settlements. To preform these tasks, I have been working on a system called, Intelligent Aide Climate (IAC).
This platform is intended to be a stand-alone system or work in concert with other IAC Systems to expand the area of coverage. To help harden human settlements where deployed systems are used against the ravages of climate and the accompanying environmental changes taking place around them.
If we are to develop, and then operationally deploy systems for civil defense in the age of climate change. Then we need to take both a long range view, of how climate change will affect many generations that will follow us. Developing a basic system that is workable now and can be added on to later as the technology and understanding about climate change develops more.
At the same time being pragmatic on the work that needs to be done now, on how to blend both recycled low cost PC hardware, in combination with open-source software to develop a stable platform. Adding to this, is how to classify great volumes of both academic and scientific literature by its appropriate value as targeted information to help settlements adjust to a changing environment.
The use of targeted information is only one of two ways to fight climate change by hardening human settlements. The other is to encourage vigorous micro-climates linking all natural habitats in and around the IAC deployed system. Making it a robust natural environment incorporating both native flora and fauna, and actively supported by the human settlements as a means of active defense against climate change.
What I am talking about the use of IAC To support the human settlement as an EWS for climate change. In which the primary purpose of the model is twofold:
- One, to answer three basic questions about a changing local climate and environmental conditions. The questions this system can answer are: Is the climate and environment changing?, What does it mean?, and What can we do? In so doing, help set up settlement wide preparedness plans; and
- Two, to back up community managers decisions with hard data for taking action for future trends and projections. The system is meant to be under local human control and does not incorporate two flavors of the day: Artificial Intelligence or the Internet of Things.
The criteria for this to operate in the field as a deployed system, at low cost, and that can be produced around the world in workshops and labs are.
- Open-Source by: specifications; design; fabrication; and with the use of FOSS Software.
- Hardware designed to be rugged, lightweight, modular, scalable, and suitable for bare-base or remote field deployment with an add-on energy sustainable power module.
- Circular economy thinking: reuse of common commodity computers into cluster platforms to mimic a server’s capabilities; that maintains a small carbon footprint; and reduce the cost for both producers and the end-user.
- Deployed sensor array system should include: sensor to monitor changes in air / water quality; non-source pollution; soil contamination and moisture content; weather – air temperature, relative humidity, wind speed and direction, atmospheric pressure and rain gauge. Other sensor system can be added: Radiation, agricultural, groundwater table.
- Low capital expenditure (CAPEX) and operating expenditure (OPEX) costs for end users.
- Ease of deployment and operation by local communities with a limited technical background.
- Incorporation of scientific and risk-reduction software, able to answer 3 questions: What is happening to our climate/environment? What does it mean?, And what can we do?
- Development of a Tier 1 and 2 support network by locally based groups.
In this paper, I point out the use of Open-Source preferable to proprietary software and hardware or articles behind a pay wall, the main reason, cost. Yes, proprietary systems and pay for literature can do just about ever thing in this paper but one, that is widespread adoption. With proprietary systems comes a high price tag for ownership and operational maintenance, only the rich can pay that bill. The more open-source systems produced, deployed and improved on, the more one generation after another has a fighting chance against climate change. Now you can say this is conjecture, but I an more then willing to prove this can be done. This is the first of three data flowcharts to be released. The other two are:
Clearing House for the Digital Library, information support services for the IAC Platform, and Telecommunications support services for the IAC Platform in the field.
Other up coming subjects to be covered:
Privacy and Security; Standardized data formats and Languages support.
GLOSSARY OF TERMS
Algorithm, the algorithms that could come from this data flowcharts diagram would predict how fast the code will run and how much space in memory it will require. Showing the running time, computer memory requirements, the hard-disk usage, power consumption and data-transmission speed in a network (if a network is used). No algorithm will accompany paper, as algorithms are crafted to each desired software and hardware configuration to be used. This paper is meant to be tried on all types of PC Clusters and to encourage experimentation.
Area of Operations (AO). A deployed system area of coverage. Example: small town, village or community.
Benchmark (B). A term use in the flowchart for the Synecology of the human settlement. Its ecology in dealing with the relations between natural communities and their natural surrounding environments.
Circular Economy, is an economic system aimed at eliminating waste and the continual use of resources. Example: Recycled personal computers (desktop, tower or laptop), turning them into commodity computers, with Hadoop software for a computer clusters needed by IAC Platform.
Clearing House (CH), is a supporting organization for the collection of targeted information from the public domain / open-source / creative commons copyright and license sources (documentation, magazine articles, books, papers, images, sound files, videos, and software) in organized electronic form, available on the IAC Platform system database. Put together as Digital Library reference information for a community AO Deployed or to be deployed IAC Platform system.
Commodity Computing, or commodity cluster computing, is the use of large numbers of already available computing components for parallel computing to get the greatest amount of useful computation at low cost. It is computing done in commodity computers as opposed to high-cost super micro computers or boutique computers.
Commodity computers are computer systems manufactured by multiple vendors, incorporating components based on open standards. Such systems are said to be based on commodity components, since the standardization process promotes lower costs and less differentiation among vendors.
Computer Cluster, consists of a set of loosely or tightly connected computers that work together so that, in many respects, they can be viewed as a single system. Unlike grid computers, computer clusters have each node set to perform the same task, controlled and scheduled by Hadoop software.
Data Flowchart Diagram (DFD), is a graphical diagram that represents the data flow of a system. This IAC plate also incorporates the flowchart graphical diagram that represents the sequence of steps used in a system to help solve the problems of civil defense in the age of climate change.
Digital Library (DL), is a collection of targeted information (documentation, magazine articles, books, papers, images, sound files, and videos) in organized electronic form, available on the IAC Platform system, DVD / CD-ROM (compact-desk read-only memory) disk / USB Drive or off site at the clearing house (CH) for support.
Footprint (F). A stand in term used in this flowchart for Autecology, ‘a term in Species Ecology, the study of the interactions of an individual organism or a single species with the living and nonliving factors of its environment. In this case the social and economic activity of the human settlement.
Harden Human Settlement (HHS). A vigorous micro-climate linking all natural habitats in and around the IAC deployed system. Making it a robust natural environment incorporating both native flora and fauna and activity supported by the human settlement as a means of active defense against climate change.
Hadoop, is an open-source software framework for storing data and running applications on clusters of commodity hardware. It provides massive storage for any kind of data, enormous processing power and the ability to handle virtually concurrent tacks or jobs.
Hadoop Ecosystem, software packages under consideration for use:
AMBARI is a software project of the Apache Software Foundation. Ambari enables system administrators to provision, manage and monitor a Hadoop cluster, and also to integrate Hadoop with the existing infrastructure. (remote administration for deployment and updates?)
CouchDB is a database that uses JSON for documents, JavaScript for MapReduce indexes, and regular HTTP for its API. CouchDB is a database that completely embraces the web. Store data with JSON documents. Access documents and query your indexes with web browser, via HTTP. Index, combine, and transform your documents with JavaScript. (Digital Library feed?)
FLUME is service designed for streaming logs into Hadoop environment. Flume is a distributed and reliable service for collecting and aggregating huge amounts of log data. (for sensor feed and weather logs?)
HBase is an column-oriented, NoSQL database built on top of Hadoop (HDFS, to be exact). It is an open-source implementation of Google’s Big-table.
HCatalog is a table storage management tool for Hadoop that exposes the tabular data of
Hive metastore to other Hadoop applications. It enables users with different data processing tools (Pig, MapReduce) to easily write data onto a grid.HIVE is similar to a SQL Interface in Hadoop. The data that is stored in HBase component of the Hadoop Ecosystem can be accessed through Hive.
Hive is of great use for developers who are not well-versed with the MapReduce framework for writing data queries that are transformed into Map Reduce jobs in Hadoop. (for sensor feed and weather logs?)
MapReduce v2. The core of Apache Hadoop consists of a storage part, known as Hadoop Distributed File System (HDFS), and a processing part which is a MapReduce programming model. Hadoop splits files into large blocks and distributes them across nodes in an cluster.
PIG is a platform for analyzing large sets of data.
SQOOP is a connectivity tool for moving data from non-Hadoop data stores – such as relational databases and data warehouses – into Hadoop. It allows users to specify the target location inside of Hadoop and instruct Sqoop to move data from relational databases to the target.
Sentry is a granular, role-based authorization module for Hadoop. Sentry provides the ability to control and enforce precise levels of privileges on data for authenticated users and applications on a Hadoop cluster. Sentry is designed to be a pluggable authorization engine for Hadoop components.
TEZ is an extensible framework for building high performance batch and interactive data processing applications, coordinated by YARN in Apache Hadoop. TEZ improves the MapReduce paradigm by dramatically improving its speed, while maintaining MapReduce’s ability to scale to petabytes of data.
Hadoop YARN is a programming model for processing and generating large sets of data. Yarn is the successor of Hadoop MapReduce. Yarn provides better scalability due to distributed life-cycle management and support for multiple MapReduce API’s single cluster.
History (H). Term used in the flowchart for the local and regional weather and climate records and documentation fed into the system to give it a base-line and historical perspective for all future changes.
Intelligent Aide Climate (IAC). A system that incorporates four process modules: input and updates; number crunching; digital library and graphical user interface (GUI) for client interactions with the system. With two techniques to fight climate and environmental changes taking place. The use of targeted information and the establishment of vigorous robust micro-climates in the IAC coverage area.
Manual Loop, the sensor array, sensors are on a 20-minute loop polling continuously till manually stopped by IAC Operation personnel. The automatic polling loop produces 72 sensor loop records per 24 hours, and is backed up at midnight local time.
open-source, in terms of software, it is denoting software for which the original source code is made freely available and may be redistributed and modified. It is suggested that for each piece of software, you look at its license. As with open-source software licenses, open-source documentation is handled by a number of organizations for the free use of copyrighted materials.
Phases of Climate Change. Climate Change or Global Warming are generalized concepts and pretty-much a blur to most. An understanding of how far-reaching each different effect of climate change is, is needed to be developed for use of the IAC System. Not one or any of the following human generations will live through all the different phases of climate change at the same time. But climate change is a multiple generational issue.
This is meant as a guide to how one phase will set the stage for another and then another. Climate change phases: 1. Adverse Weather; 2. Sea Level Rise & Storm Surges; 3. Rapid Reduction of Mountain Glaciers, North American & Asian zones of Permafrost, Arctic, Greenland and Antarctica snow and ice fields; 4. Continental (shelf) crust displacement by Ocean (basin) crust by the rapid removal of weight from land masses in phase three, triggering increased seismic and volcanic activity; 5. Changes in the Earth magnetic field; Magnetic pole locations and the South Atlantic Anomaly.
Sensor Array (SA). Is for the type, kinds and numbers of sensors deployed in and around the settlement to note and record the changes taking place. Example: sensor to monitor changes in air and water quality; non-source pollution; soil contamination and moisture content; weather – air temperature, relative humidity, wind speed and direction, atmospheric pressure and rain gauge. Other sensor system can be added are: Radiation, agricultural, groundwater table.
System Platform Configuration, is designed for use in Circular Economy thinking by the reuse of common commodity computers into a Hadoop cluster platform. To mimic a server’s capabilities, while maintain a small carbon footprint and reduce the cost to both the producers of systems and end-users. The Hadoop and software utilities stack is mounted on a Linux Ubuntu cluster.
STEPS in the Data Flowchart Diagram For the Intelligent Aide Climate (IAC) Model
1. START, IAC Module One
2. INPUT documentation under Footprint / Benchmark / History categories (F/B/H)
Comment: The most important element to the whole IAC System is obtaining a since of perspective that these three categories contain. Without them any charting, trend or projection made are useless. Example: Sea level has historically changed over time, but knowing at what dates and locations, sea level was fixed for a time. Gives us information, is it rising, falling or erratic.
Just as important is the information about how the settlement interacts with it’s surrounding environment and the nature of the social and economic activity within it. Without good information, it is an up-hill battle all the way.
3. DOCUMENTATION and reports use in a deployed IAC System
AO Human Settlement Footprint (AUTECOLOGY) Records Documents (F)– Color Blue
AO Human Settlement Benchmark (SYNECOLOGY) Records Documents (B) – Color Green
AO Weather / Climate History – local/regional – Records Documents (H) – Color Yellow
4. MANUAL INPUT digitize (F/B/H/) scanner and digital files/record input.
5. COLLATE information (F/B/H) into standard formats
6. INPUT AO Sensor Array (SA) – Color Red
Comment: The type and numbers of sensors deployed in the array can determine which phase of climate change the AO is in or about to enter. As well as the nature of the type of threat it well be under.
7. MANUAL LOOP (to stop), AO Sensor Array polling ever 20 minutes automatic till manual stopped – Color Red
8. COLLATE information from SA into standard formats to match F/B/H – Color Red
9. MERGE data streams (F/B/H/SA)
10. PROCESS data streams into one channel (F/B/H/SA)
11. SORT AO (F/B/H/SA) Data feeds into per-determined sequence for data storage
12. OUTPUT copy of data go to IAC Platform Database and Module Two Input step 16
13. DATA STORAGE AO F/B/H/SA backup Module One each midnight local time.
Comment: Optional, can go directly to step 14.
14. DATABASE – Module One backup each midnight local time.
15. PROCESS, Cluster Master and communications channel
Comment: The Cluster Master controls the access between the outside secure communications channel and the platform for Digital Library requests for updates and new media; Off site storage of copies of the databases and download stream responses from the Clearing house.
16. INPUT Module Two, process copy from Module One (F/B/H/SA)
17. COLLATE History and Sensor Array data streams from data storage (examine and compare) – Color Yellow / Red
18. SORT collated data by relationships and patterns as to seasons, month of the year, calendar dates, and hours of the day. Color Yellow / Red
19. PROCESS develop graph by year, season, month and week using history and sensor data. – Color Yellow / Red
20. PROCESS look for set relationships, patterns, trends and non-set trends. – Color Yellow / Red
21. PROCESS, take trends and non-set trends from graph when first noticed from processes 18 and 19. Double the time and projected path of trend. Triple it, then square it, noting each projected path and time for it to take. Note each projected non-set trend as future projections. Color Yellow / Red
22. OUTPUT, from Module Two – copies to platform database and Module Three input (new history / sensor – projections) – Color Yellow / Red
23. DATA STORAGE, Module Two (new history/sensor array – projections) Color Yellow / Red
Comment: Optional, can go directly to step 24.
24. DATABASE Module Two backup (new history/sensor array – projections). Color Yellow / Red
Comment: Backup Module Two each midnight local time.
25. INPUT, copy of Module Two (new history/sensor array projections and Footprint / Benchmark understanding at time of system deployment in AO or latest scheduled update to Footprint / Benchmark. Note, up date reading Footprint/Benchmark should be taken annually where possible and added to platform database with date and time stamp.
26. INPUT from Module One (Baseline and updates (H/SA) for Digital Library.
Comment: Note, up date reading of Footprint/Benchmark should be taken annually where possible and added to platform database with date and time stamp.
27. MERGE date streams from steps 25 and 26
28. COLLAGE baseline and updates with new historical information and sensor array projections.
29. SORT, match key words from projected tends with Digital Library categories key words.
Comment: Request update from Clearing House of Digital Library. Update Digital Library categories to projected tends key words
30. DATA STORAGE of both keys words from projected tends and Digital Library categories. Copy to Digital Library database.
Comment: Optional, can go directly to step 31.
31. DATABASE (Digital Library).
32. DATABASE (platform) Copy of Module Three, backup each midnight local time.
33. INPUT (Module Four) add Module Two information, F/B/H/SA with new projections
34. COLLATE History and update Sensor Array data streams (examine and compare) – Color Yellow / Red
35. SORT AO History and update sensor information.
36. DECISION, first of three main questions for IAC. “Is Our Climate & Environment Changing?”
Comment: Answers to this first question is highly dependent on: the number and types of sensors deployed in the AO Sensor array; The dept and breadth of the historical local/regional weather and climate data inputted into the system and the interpretation processes in steps 18, 19 and 20 also comes into play for greater accuracy.
37. DISPLAY (screen monitor) show the following information: No decision, end of flowchart.
Comment: Give following answer: “No Change in Our Climate & Environment Is Detectable” With date and time stamp.
38. DISPLAY (screen monitor) show the following information from the Yes decision.
Comment: Giving the following answer: “Yes there is a change detectable in climate & environment”. With date and time stamp. The screen will display which climate change phase the AO has just entered and or is in now: 1. Adverse Weather; 2. Sea Level Rise & Storm Surges; 3. Rapid Reduction of Mountain Glaciers, North American & Asian zones of Permafrost, Arctic, Greenland and Antarctica snow and ice fields; 4. Continental (shelf) crust displacement by Ocean (basin) crust by the rapid removal of weight from land masses in phase three, triggering increased seismic and volcanic activity; 5. Changes in the Earth magnetic field; Magnetic pole locations and the South Atlantic Anomaly.
39. DECISION, the second of the three main questions for IAC is: “What does it mean?”
Comment: Again the answers to this question as with the first is highly dependent on: the number and types of sensors deployed in the AO Sensor array; The dept and breadth of the historical local/regional weather and climate data inputted into the system and the interpretation processes in steps 19, 20 and 21 also comes into play for greater accuracy.
40. DISPLAY (screen monitor) show the following information: No decision, end of flowchart, Give the following answer: “Nothing, No Change in Climate & Environment Is Detectable?” With date and time stamp.
41. DISPLAY (screen monitor) show the following information from the Yes decision. Giving the following answer: “Yes there is a change detectable in climate & environment” With date and time stamp.
Comment: The screen will display which climate change phase the AO has just entered and or is in now. Once more we need to refer back to the number crunching done in module two for data modeling and future projections and the standard formats developed in module one as to if it will get: hotter, colder, wetter, dryer or more flooding with sea level change. Based on the pattern trends and non-set trends point out.
42. DECISION, the last of three main questions for IAC. “What Can We Do?”
43. DISPLAY (screen monitor) show the following information: No decision, Give the following answer: “Nothing” With date and time stamp.
44. DISPLAY (screen monitor) show Digital Library media response to query.
45 OUTPUT of Module Four.
46. DATA STORAGE
Comment: Optional, can go directly to step 46.
47. DATABASE Backup Module Four, backup each midnight local time.
15. Back to Cluster Master / Comms channel.
48. END
Plate 1. Data Flowchart Diagram for the Intelligent Aide Climate (IAC) Model
KEY:
Data flowchart diagram plate color code for the five data streams in use are:
Human Settlement “Footprint” (AUTECOLOGY) Records – Documents – Color Blue
Human Settlement “Benchmark” (SYNECOLOGY) Records – Documents – Color Green
AO Weather / Climate “History” – local/regional – Records – Documents – Color Yellow
AO Sensor Array – Near Real time input – Color Red
IAC System decision, display screen, processes, steps or actions – Color Back
REFERENCES
Nerilie J. Abram, Helen V. McGregor, Jessica E. Tierney, Michael N. Evans, Nicholas P. McKay, Darrell S. Kaufman & the PAGES 2k Consortium (25 August 2016) Early onset of industrial-era warming across the oceans and continents, Nature volume 536, pages 411–418
Early works on this concept
Robin EH. Hoard (Jan. 15, 2018) A Course of Action
http://independent.academia.edu/RobinHaberman
Robin EH. Hoard (Oct. 8, 2016) Civilian Defense in the Age of Climate Change
http://independent.academia.edu/RobinHaberman
Robin EH. Hoard (March 24, 2016) LET’s START WITH WHY
http://independent.academia.edu/RobinHaberman
Robin EH. Hoard (Sept. 13, 2015) Earth as a Library
http://independent.academia.edu/RobinHaberman
Robin EH. Hoard (July 1, 2015) Results from a Survey on Climate Change and
Local Responses To It
https://www.linkedin.com/in/robin-haberman-8718998/detail/recent-activity/posts/
Linux Hadoop Small Clusters
Michael G. Noll (August 5, 2007) Running Hadoop On Ubuntu Linux (Single-Node Cluster)
https://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
Michael G. Noll (August 2007) Running Hadoop On Ubuntu Linux (Multi-Node Cluster)
https://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
Michael G. Noll (August 2007) Writing An Hadoop MapReduce Program In Python
https://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/
TRADE NAMES
Open-Source & Copyrighted Documentation
Under Creative Commons, (documentation, magazine articles, books, papers, images, sound files, and videos).
Creative Commons (CC) – https://creativecommons.org/
Science Commons – https://creativecommons.org/sciencecommons/
Free and Open-Source Software
(FOSS) licenses are held under:
Apache License 2.0,
Apache Software Foundation – https://www.apache.org/licenses/LICENSE-2.0
BSD 3-Clause “New” or “Revised” license,
open-source Initiative – https://opensource.org/organization
BS666D 2-Clause “Simplified” or “FreeBSD” license;
open-source Initiative – https://opensource.org/organization
GNU General Public License (GPL); Free Software Foundation – https://www.fsf.org/about/
open-source Initiative – https://opensource.org/organization
GNU Library or “Lesser” General Public License (LGPL);
Free Software Foundation – https://www.fsf.org/about/
MIT license; Mozilla Public License 2.0.
MIT License – https://www.opensource.org/licenses/MIT
Other Licenses for use of software:
Canonical – https://canonical.com/
The Document Foundation – https://www.documentfoundation.org/
Eclipse Foundation – https://www.eclipse.org/
Free software movement
https://en.wikipedia.org/wiki/Free_software_movement
GNOME Foundation – https://www.gnome.org/foundation/
GNU General Public License (GPL) – https://www.gnu.org/licenses/lgpl-3.0.en.html
Linux Foundation – https://www.linuxfoundation.org/about/
Mozilla Foundation – https://foundation.mozilla.org/en/
Open-Source Geospatial Foundation – https://www.osgeo.org/about/
Python Software Foundation – https://www.python.org/psf/
Software Freedom Conservancy – https://sfconservancy.org/