Central Design Challenges in Designing a Energy Monitoring System

Introduction


This section will look at the central design challenge of designing a energy monitoring system which can monitor all IP devices on a network. From the design challenge the chapter will discuss broad principles which will guide product design and development. Broad principles will be used to inform choices of existing technologies e.g. operating system, scripting language. The selected technologies will be discussed within a high-level approach section. The high level approach will describe delivery milestones and desired functionality within each milestone.

Design Challenge

Designing an energy monitoring system which can monitor individual device level energy usage is difficult because enterprise networks often incorporate equipment from various vendors, over several years, operating on different operating systems (OS). Energy monitoring capability differs dependent on the vendor, model, age, operating system and configuration. The following design principles will be adopted to produce the product:-

Principle 1, only phase 1 monitoring. For design, development and deployment simplicity, the solution being proposed will only support phase 1 i.e. passive monitoring of an estate. The design will explicitly exclude phase 2, real time pricing information and also exclude phase 3, active management.

Principle 2, wide device coverage. Passive monitoring will have the capacity to report energy information for all IP connected devices.

  • For devices which have the capacity to directly report energy usage via a SNMP MIB, the solution will collect power utilisation information and report on energy usage.
  • For devices which are SNMP enabled but do not have a MIB to directly report energy usage, the solution will obtain system information i.e. device type and (where possible) connected devices. The solution will then reference an internal static list to report on power use. For simplicity it will be assumed these devices will have two states of operation (ON/OFF), rather than finite number of operating states. The reason for this is, a single ON power measurement will be used when the device is reachable.1
  • For devices without SNMP their media access control (MAC) address will be obtained from the network. The MAC address contains 12 digits, the system uses this to identify the device manufacturer and if possible the device type. The solution will then reference an internal static list to report on power use and provide a two-state power operation.

Principle 3, system training1. System training refers to manual customisation of a platform to monitor energy of devices which do not provide direct energy statistics. Energy monitoring capability differs dependent on the vendor, model, age, operating system and configuration. To accommodate these differences, the system will need to be manually customised with device types and their energy consumption information. For instance:-

  • For devices that do not provide SNMP direct power measurements and have not been validated. A manual check will be required whereby a manual ILM device will directly measure a sample device. These real power usage figure will be used and a internal static list will be updated. This manual check and system update will be used to teach the system and make it more accurate.
  • For those devices which cannot be identified by their MAC address, an unknown device log will be maintained. The log will be manually reviewed, and devices identified will be validated. Physical audits may be required to establish what these unidentified devices are and power utilisation either obtained from the manufacturers specification or via a manually measurement. This information could then be added into a manually developing list of devices and energy use.

Principle 4, reporting. Devices being monitored will be grouped into logical groups and given meaningful names. Device groups will be dependent on the customer’s environment. The principle of simple graphical representation of energy used, grouped by:-

  • Location and up to three sub-locations. e.g. company-wide energy use, site energy use and floor/subnet.
  • Device type. e.g. Dell Laptops, HP Laptops, Cisco Routers, Cisco Switches
  • Device group. e.g. a sum total of energy use for all laptops and another a sum total of energy use for all Cisco routers and switches.
  • Reports to collate information and display the last 24 hours, last month and last 12 months energy usage in a graphical form. Reports to be made available via a website.
  • \

Principle 5, light touch. Minimize the impact of implementing the energy monitoring system by not needing any specific software to be loaded on end devices. Specifically no agents or widgets required to be installed on monitored nodes to obtain energy consumption information. Where appropriate the solution will use existing SNMP functionality to obtain power consumption information. Or use the MAC address information to estimate power utilisation information.

Principle 6, scalability. The initial system is able to obtain power statistics for 256 simultaneous IP devices and ensure the system has the capacity to retain reports for 1 day, 1 week, 1 month and 12 months. In future iterations the system should be able to be scalable to monitor more than 256 devices.

Technology and Implementation

This section will discuss each design principle. The Agile2 approach to product development will be adopted, as part of that approach, each principle will be broken down into requirements. Where possible these requirements will be met using existing products or solutions.

  • Principle 1, phase 1 monitoring – the solution will only support passive monitoring. This principle requires a server to poll devices and obtain, store and display power statistics. The server-based energy monitoring platform will require an operating system (OS) which is able to run a scheduled monitoring service, data storage and a webservice.

The three most popular open source Linux distributions all support scheduling, monitoring, storage and webservices. Ubuntu is the most popular service with twice as many server deployments than its nearest rivals, Linux and Redhat3.

Ubuntu is selected as the OS of choice as it is popular and well supported by frequent releases and security patches4.

Principle 2, wide device coverage – the solution should be able to schedule and poll all IP devices in a network. This principle requires a polling agent or process.

Ubuntu has established tools which will be installed to support polling devices. SNMP will be installed to provide MIB polling, SNMP-MIBS-DOWNLOADER will be installed to provide standard MIB library, NMAP and IPUTIL-ARPING will be installed to poll IP addresses and obtain MAC addresses. Scheduling software, Ubuntu (and all Linux distributions) natively supports scheduling via the cron feature.

Principle 3, system training – The system should be manual customisable to ensure the platform is able to monitor energy of devices which do not provide direct energy statistics. The platform should be able to be scripted so that it can poll IP devices, log those devices which the system cannot obtain power statistics. In principle the platform should then be manually customisable with static energy usage information.

Python will be used as the scripting language. Python will poll devices by pinging them and obtaining their MAC address. The system will then create logs of devices which it cannot obtain or infer energy usage. These logs could then be manually reviewed so that the system could be customised.

Polling Logic

Refer to figure above for an illustration of the logic to be implemented. As further narrative, system training refers to the process of customising the system.

  • For devices which support SNMP MIBs providing direct power measurements, the system will need to be configured with the specific MIB to obtain power stats.
  • For devices which support SNMP but do not provide specific MIBS for power measurements, the system will need to be configured to recognise the device type and use static power usage stats.
  • And for devices which do not support SNMP, the system will need to be configured to recognise the MAC address and associate it to a power usage.

Principle 4, reporting – devices being monitored will be grouped into logical groups, given meaningful names and be available to the end user. Device groups will be dependent on the customer’s environment.

The Ubuntu operating system is able to host a Web Server to display energy monitoring data. Three common web server solutions are available to run on Ubuntu, Lighttpd, Nginx and Apache HTTP Server. All three solutions provide the functionality required, Apache is the most popular Web server for websites as such will be used within this solution.5. Apache will be used to host a website to display energy usage statistics. Refer to the Polling Logic diagram for an illustration of the report types to be produced.

Principle 5, light touch. Minimize the impact of implementing the energy monitoring system by not needing any specific software to be loaded on end devices.

The solution will remotely monitor devices and will not install an agent on the monitored device. Monitoring will either use SNMP or PING. Where available SNMP will be used to remotely poll power usage. Where SNMP power utilisation MIBS are not available PING will be used to ascertain whether the device is live. This process will allow for remote monitoring without the need to install a agent on the monitored device.

Principle 6 – Scalability. The initial system is able to obtain power statistics for 256 simultaneous IP devices and ensure the system has the capacity to retain reports for 1 day, 1 week, 1 month and 12 months.

The system will use existing libraries to obtain power statistics from 256 devices, examples of Python libraries to be used,6 which can poll up to 65,000 devices, and pysnmp6 which can obtain power information from specific SNMP MIBs. MRTG7 will be used to display and store data. Data retention will be supported through progressive data summarisation. Energy usage data to be polled every 5 minutes. This data to be retained for 1 week and displayed. Data after one week to be summarised to every 30 minutes and displayed. Data over one month old to be summarised to 2 hours and displayed and retained for 12 months. No data to be retained over 12 months. All data to be stored in individual device data files.

Summary

The solution being developed will use existing freeware or shareware software. Ubuntu will be used as the baseline operating system. Python will be used as the scripting language. And Apache will be used as the Web server. All services will reside within the same VM instance.

The solution will remotely monitor devices without the need to install an application on the monitored device. The solution will be developed to use existing SNMP MIBs to obtain power stats or estimate power utilisation from the MAC address of the device. It will report energy usage in a graphical form.

Following posts will document the build of a PoC enviroment that will reside within a virtual environment. This environment is intended to prove that a polling agent is able to poll devices and display basic statistics on a website. The estate will be configured to poll three types of devices, devices which support SNMP MIB providing power usage, device which support SNMP but not provide a power utilisation MIB and a device which not supporting power utilisation.

  1. Zoha, A., Gluhak, A., Imran, M. A. \& Rajasegarar, S., 2012. Non-Intrusive Load Monitoring Approaches for Disaggregated Energy Sensing: A Survey. Sensors, Issue 12, pp. 38-66. DOI: 10.3390/s121216838 [↩] [↩]
  2. Agilemanifesto.org. (2018). Manifesto for Agile Software Development. [online] Available at: \href{www.agilemanifesto.org}{www.agilemanifesto.org} [Accessed 20 Apr. 2018] [↩]
  3. TCM, 2018. The Cloud Market: EC2 Statistics. [Online] Available at: \href{http://thecloudmarket.com/stats#/totals}{http://thecloudmarket.com/stats#/totals} [Accessed 10 April 2018] [↩]
  4. Ubuntu, 2018. Ubuntu Release. [Online] Available at: \href{https://wiki.ubuntu.com/Releases}{https://wiki.ubuntu.com/Releases} [Accessed 15 April 2018 [↩]
  5. W3Techs, 2018. Web Server Comparison. [Online] Available at: \href{https://w3techs.com/technologies/overview/web_server/all}{https://w3techs.com/technologies/overview/web_server/all} [Accessed 5 April 2018 [↩]
  6. Python library to monitor one or many IP addresses via ICMP echo (ping) requests [↩] [↩]
  7. Multi Router Traffic Graphee [Online] Available at: \href{https://oss.oetiker.ch/mrtg/doc/mrtg-reference.en.html}{https://oss.oetiker.ch/mrtg/doc/mrtg-reference.en.html} [Accessed 1 Feb 2018] [↩]