SCHOOL OF INFORMATICS AND ENGINEERING
REAL-TIME WEARABLE AND WIRELESS BIOFEEDBACK SENSING
A report submitted in partial fulfillment of the requirements for the degree
BACHELOR OF ENGINEERING (HONOURS)
SUPERVISOR: DR. CATHERINE DEEGAN
SUBMISSION DATE: 21/01/2017
The work submitted in this report is the results of the candidate’s own investigations and has not been submitted for any other award. Where use has been made of the work of other people it has been fully acknowledged and referenced.
TABLE OF CONTENTS
TABLE OF CONTENTS. iii
LIST OF ABBREVIATIONS. v
LIST OF FIGURES. v
LIST OF TABLES. vi
Chapter 1 Introduction. 1
1.1 Background. 1
1.2 Problem.. 1
1.3 Scope and Objectives. 1
Chapter 2 Literature Review.. 2
2.1 Body Health Signals 2
2.1.1 Galvanic Skin Response. 2
2.1.2 Temperature. 4
2.1.3 Heart Rate Variability. 5
2.1.4 Measurements. 5
2.1.5 Signal Registration. 5
2.1.6 Problems with wearable device. 5
2.2 Types of Sensors. 5
2.2.1 Accelerometer 6
2.2.2 Blood Volume Pulse. 6
2.2.3 Galvanic Skin Response. 6
2.2.4 Interbeat Interval 6
2.2.5 Human body temperature. 6
2.2.6 Device Battery. 6
2.3 Empatica E4 Wristband. 6
2.4 E4 Real-Time Data Stream.. 6
2.5 Bluetooth Low Energy Server 6
2.5.1 Bluetooth Low Energy USB Dongle. 6
2.5.2 BLE Server Operation. 6
2.5.3 BLE Server Commands. 6
2.5.4 BLE Server Data Structure. 7
2.5.5 Parser for BLE Server 7
2.6 Proof of Concept 7
2.6.1 Types of Stress 7
2.6.2 Cognitive Stress. 7
Chapter 3 Design and Implementation. 8
3.1 Software System Overview.. 8
3.2 Design Consideration. 8
3.3 Software Tools. 8
3.4 Architecture Design. 9
3.5 Android Design and Implementation. 9
3.6 Android Graphical User Interface. 9
3.7 Desktop Application Interface. 9
3.8 Desktop Application Design. 9
3.9 Challenges 9
Chapter 4 Results and Discussion. 10
4.1 Extracted Data Streams from E4. 10
4.2 Verification of E4 wristband. 10
4.3 Discussion. 10
Chapter 5 Conclusion and Future work. 10
5.1 Conclusion. 10
5.2 Future Work. 10
APPENDIX A: 12
LIST OF ABBREVIATIONS
EDA Electro Dermal Activity
GSR Galvanic Skin Response
SCL Skin Conductance Level
IBI Inter Beat Interval
BVP Blood Volume Pulse
HR Heart RateHRV Heart Rate Variability
AHR Average Heart Rate
PPG Photo Plethysmography
LIST OF FIGURES
LIST OF TABLES
Chapter 1 Introduction
Biofeedback systems begin its existence from 20th century when first computer where invented to be used in complex calculations in medicine and science. People tried to use first invented biofeedback sensors and tried to use computation power to analyze and explain physiological processes in human body. In 21st century, technology has advanced high rapid manor resulting in high in high power efficiency, wireless technology, battery capacitance, high biofeedback sensor accuracy with low product cost. It has given ability to create consumer available products in real-time biofeedback sensing using Bluetooth Low Energy (BLE) wireless to connect and cloud based technologies to store biofeedback data.
The wearable and wireless product Empatica E4 released in 2015 uses cloud-base system that does not support real-time desktop biofeedback sensing application.
It presumes to be store on cloud-base website and has to be uploaded either using BLE connection to Mobile Application and upload data using device internet connection or using straight forward USB connection to desktop Empatica Manager upload application which reads autonomously recorded data, uploads it if internet connection on the desktop is available, and cleans the storage of the device from previously recorded data.
Unfortunately this method of reading data can not be used to analyze data in real-time so development of real-time desktop is needed.
1.3 Scope and Objectives
This project involves the desktop application design and development for real-time record and analysis, primarily in stress detection analysis.
To create this application will be used literature review of the subject, BLE solution from desktop to Empatica E4 device, parse solution of the data, data analysis and solution of the stress detection, solution to data time records, final real-time desktop application with application public interface (API) for future real-time Empatica E4-based desktop applications and application tests based on stress induction proof of concept experiment.
Chapter 2 Literature Review
2.1 Body Health Signals
Body health signals are biofeedback signals that requires a biological creature to connect different types of sensors that can sense different types of biophysical state of the body.1
The types of body health signals include:
Electromyography (EMG) – measures charge of muscle tensions.
Temperature (TEMP) – measures body temperature.
Electroencephalography (EEG) – measures brain wave type activity.
Galvanic Skin Response (GSR) – measures skin body resistance (sweat of the skin).
Heart Rate Variability (HRV) – measure heart rate and pulse variability.
2.1.1 Galvanic Skin Response
Galvanic Skin Response (GSR) refers to Electro Dermal Activity (EDA) and Skin Conductance (SC), which modulates the amount of sweat secretion from sweat glands, which increases the conductance when either external or internal stimuli occur. The internal stimuli often refer to psychological arousal.2
Figure 1 – Illustration of smooth skin section from the sole of the foot 3
Since the main purpose of the skin is to protect the body from the environment it also has many other functions that depend on layers of the skin tissues, which prevent the loss of water by the body by the stratum corneum as a barrier layer that prevents the loss of water through the sweat glands. These sweet glands, in turn, are controlled by the autonomic nervous system provides where a measurement of EDA is a simple gauge of the level activity of them.4
Temperature is a non-linear complex variable that has sources in the human body as internal and external, which complicates interpretation of the measured data of the body. The normal temperature for the healthy human body is 37° C/98.6° F, which can vary between 97.2° F/36.2° C and 99.5° F/37.5° C depending on the individual.5
Figure 2 – Hypothalamus thermal regulation process6
The temperature is regulated by controlling part of the hypothalamus, which has neural biofeedback-controlled mechanisms, and sensors that are necessary to control thermal regulations.7
2.1.3 Heart Rate Variability
Heart rate variability is an analysis of the heart’s beat-to-beat variation in the heart rhythm.
“By accurately measuring the time interval between heartbeats, the detected variation can be used to measure the psychological and physiological stress and fatigue on the body …”.8
Figure 3 – Method of measuring heart rate variability the time between R spikes9
General description of expected biofeedback values and their units.
2.1.5 Signal Registration
Brief concept of the biofeedback signal registration.
2.1.6 Problems with wearable device
The senser position, small amplitude of signals, low energy consumption.
2.2 Types of Sensors
Brief overview of the types of sensors are used to sense the biofeedback signals.
Full explanation of what is accelerometer sensor. How it works.
2.2.2 Blood Volume Pulse
Full explanation of what is blood volume pulse sensor. How it works.
2.2.3 Galvanic Skin Response
Full explanation of what is galvanic skin response sensor. How it works.
2.2.4 Inter Beat Interval
Full explanation of what is inter beat interval sensor. How it works.
2.2.5 Human body temperature
Full explanation of what is temperature sensor. How it works.
2.2.6 Device Battery
Full explanation of device battery used in devices to maintain wear ability.
2.3 Empatica E4 Wristband
Clear structured description of device used in Project.
2.4 E4 Real-Time Data Stream
Detailed description of the real-time data concept and structure.
2.5 Bluetooth Low Energy Server
Detailed look at available software for E4.
2.5.1 Bluetooth Low Energy USB Dongle
Description of the product and its compatibility for BLE Server.
2.5.2 BLE Server Operation
Illustration BLE server process to identify common patterns for application.
2.5.3 BLE Server Commands
List of commands and its functions to be used in application.
2.5.4 BLE Server Data Structure
Types of data and stream representation for client application.
2.5.5 Parser for BLE Server
Design of parser class for BLE client to retrieve data from BLE Server streams.
2.6 Proof of Concept
Description of concept to proof verification of the device.
2.6.1 Types of Stress
Brief list of types of the stress and identify core the type of the stress is the most appropriate to be used in experiment.
2.6.2 Cognitive Stress
Detailed explanation of cognitive stress and ways it is induced in human body.
Chapter 3 Design and Implementation
3.1 Software System Overview
The desktop software application is communicating with Empatica E4 wristband using BLE connection. To establish communication between desktop and Empatica E4 device the software first need to successfully connect and configure streams of data from the BLE closed source serve, which natively connects to E4 wristband via Bluetooth and creates a daemon to the output stream to the software. To establish communication between E4 and software, the configuration command has to be sent to Server. The type of streams and data will depend on the setup configuration message to the server.
3.2 Design Consideration
The primary considerations are physical and ethical for this project.
The physical consideration was based on the connectivity of the Empatica E4 wristband of the GSR sensor and conductivity of the skin. The skin might have a very high level of resistance of the dry hands. To improve conductivity and results of the skin conductance sensor, the additional conducting substance will be applied on the surface of the skin.
The ethical consideration was based on the fact that the arousal stimuli to trigger the autonomic neural system into stress is very challenging that has to be based on cognitive stress situation that has to give enough stimulus to increase the activity of sweet gland secretion to a certain point the skin conductance will be significantly changed.
3.3 Software Tools
The C# has been chosen as primary programming language based on the software design of the BLE server.
The benefits of this language are based on integration ability of the C# language main framework – a .NET framework that gives high-level classes and libraries that have an unprecedented level of optimization on Windows-based machines.
The integration of the framework between different based on its software will enable high-level integration into more sophisticated projects that are based on the .NET framework.
The similarity of the C# and Java languages will give an easy porting ability to the other operating systems.
3.4 Architecture Design
A detailed description of the architecture. Use cases, dataflow and classes.
3.5 Android Design and Implementation
Description of communication using BLE, parsing, analysis.
3.6 Android Graphical User Interface
Description of GUI interface and application design.
3.7 Desktop Application Interface
Description of the application public interface with Bluetooth driver.
3.8 Desktop Application Design
Description of BLE Server, connection to the device, parser, and graphical user interface.
Brief list of challenges was faced during the design and software implementation.
Chapter 4 Results and Discussion
4.1 Extracted Data Streams from E4
List of detailed results.
4.2 Verification of E4 wristband
A conclusion to verify or not the Empatica E4 device based on the results.
Discussion and interpretation of the results of the project.
Chapter 5 Conclusion and Future work
Confirmation of the results of the device verification and consistency of the data for stress level analysis.
5.2 Future Work
The future work on the analysis using other methods, etc.
1 “Biofeedback: Types, Purpose, and Risks”, Healthline, 2018. Online. Available: https://www.healthline.com/health/biofeedback#types. Accessed: 21- Jan- 2018.
2 R. Salim, P. Mahler and P. Bryn Farnsworth, “What is GSR (galvanic skin response) and how does it work?”, iMotions, 2018. Online. Available: https://imotions.com/blog/gsr/. Accessed: 21- Jan- 2018.
3 Bem.fi, 2018. Online. Available: http://www.bem.fi/book/27/fi/2701.gif. Accessed: 21- Jan- 2018.
4 “27. The Electrodermal Response”, Bem.fi, 2018. Online. Available: http://www.bem.fi/book/27/27.htm. Accessed: 21- Jan- 2018.
5 “Body Temperature Variability (Part 1): A Review of the History of Body Temperature and its Variability Due to Site Selection, Biological Rhythms, Fitness, and Aging”, Altmedrev.com, 2018. Online. Available: http://www.altmedrev.com/publications/11/4/278.pdf. Accessed: 21- Jan- 2018.
6 “Hypothalamus stock illustration. Illustration of hypothalamus – 19173185”, Dreamstime.com, 2018. Online. Available: https://www.dreamstime.com/royalty-free-stock-photo-hypothalamus-image19173185. Accessed: 21- Jan- 2018.
7 “Temperature Regulation of the Human Body”, Hyperphysics.phy-astr.gsu.edu, 2018. Online. Available: http://hyperphysics.phy-astr.gsu.edu/hbase/thermo/heatreg.html. Accessed: 21- Jan- 2018.
8 “Heart Rate Variability – Myithlete”, Myithlete, 2018. Online. Available: https://www.myithlete.com/what-is-hrv/. Accessed: 21- Jan- 2018.
9 9097bca1a75e8bb2e831365d-hrvfitltd.netdna-ssl.com, 2018. Online. Available: https://9097bca1a75e8bb2e831365d-hrvfitltd.netdna-ssl.com/wp-content/uploads/2014/07/ECG-Trace.jpg. Accessed: 21- Jan- 2018.