File Type: MS Word (DOC) & PDF

File Size: 1,178KB

Number of Pages:79

**ABSTRACT**

The artificial intelligence (AI) domain grows every day with new algorithms and architectures. Artificial Neural Networks (ANNs), a branch of AI has become a very interesting domain since the eighties when the back-propagation learning algorithm and the feed-forward architecture were first introduced. As time passed, ANNs were able to solve non-linear problems, and were being used in classification, prediction, and representation of complex systems. However, ANN uses a black box learning approach – which makes it impossible to interpret the relationship between the input and the output. Discrete Event System Specification (DEVS) is a mathematical well-defined formalism that can be used to model dynamic systems in a hierarchical and modular manner; it can automatically generate simulators for the described DEVS models. Combining ANN and DEVS, we can model the complex configuration of ANNs and express its internal workings. In this thesis, we are extending the DEVS-Based ANN proposed by Toma et al [1] for comparing multiple configuration parameters and learning algorithms. The DEVS model is described using a visual modeling language known as High Level Language Specification (HiLLS) for a clear understanding. This approach will help users and algorithm developers to test and compare different algorithm implementations and parameter configurations of ANN.

**TABLE OF CONTENTS**

1.0 INTRODUCTION ………………………………………………………………………………………………… 1

1.1. Context …………………………………………………………………………………………………………….. 1

1.2. Research Objectives …………………………………………………………………………………………… 2

1.3. Related Works …………………………………………………………………………………………………… 2

1.3.1. Abstraction of Continuous System to Discrete Event System Using Neural Network……………………………………………………………………………………………………… 3

1.3.2. Identification of Discrete Event Systems: Using the Compound Recurrent Neural Network: Extracting DEVS from Trained Network …………………………………………. 4

1.3.3. Neuro-DEVS, an Hybrid Methodology to describe Complex Systems ……………….. 4

1.3.4. Dynamic Neuronal Ensembles (DNE): Neurobiologically Inspired Discrete Event Neural Networks …………………………………………………………………………………………. 5

1.3.5. A New DEVS-Based Generic Artificial Neural Network Modeling Approach ……. 6

1.4. Approach Adopted …………………………………………………………………………………………….. 7

1.5. Organization of Work…………………………………………………………………………………………. 7

2.0 STATE OF THE ART …………………………………………………………………………………………… 8

2.1. Artificial Neural Networks (ANN) ………………………………………………………………………….. 8

2.1.1. History …………………………………………………………………………………………………………… 8

2.1.2. Architecture ………………………………………………………………………………………………….. 10

2.1.3. Activation Functions………………………………………………………………………………………. 12

2.1.4. Learning Algorithms………………………………………………………………………………………. 15

2.1.4.1. Standard Back Propagation (BP) Algorithm………………………………………………… 16

2.1.4.2. Back Propagation with Momentum Algorithm…………………………………………….. 18

2.1.4.3. Silva and Almeida (SA) Algorithm ……………………………………………………………. 19

2.1.4.4. Delta-Bar-Delta Algorithm ……………………………………………………………………….. 20

2.1.4.5. Quickprop Algorithm ……………………………………………………………………………….. 21

vii

2.1.4.6. Resilient Back propagation ……………………………………………………………………….. 22

2.1.5. Applications of ANN ……………………………………………………………………………………… 24

2.2. Discrete Event System Specification (DEVS) …………………………………………………………. 25

2.2.1. The DEVS Modeling ……………………………………………………………………………………… 25

2.2.1.1. Atomic Classic DEVS Model ……………………………………………………………………. 26

2.2.1.2. Coupled Classic DEVS Model…………………………………………………………………… 27

2.2.2. The DEVS Simulation ……………………………………………………………………………………. 29

2.2.3. DEVS SimStudio Simulation Package ……………………………………………………………… 30

2.3. High Level Language for System Specification (HiLLS) ………………………………………….. 32

2.3.1. HiLLS Architecture ……………………………………………………………………………………….. 32

2.3.2. Concrete Syntax of HiLLS ……………………………………………………………………………… 33

2.3.3. HiLLS Example: Single Lane Road Model ……………………………………………………….. 36

3.0 DEVS- BASED ANN ………………………………………………………………………………………….. 38

3.1. DEVS-Based ANN Approach ……………………………………………………………………………….. 38

3.1.1. DEVS-Based ANN Design ……………………………………………………………………………… 38

3.1.2. Feed-Forward Calculations Model Set ……………………………………………………………… 39

3.1.2.1. Non-Calculation Layer Atomic Model ……………………………………………………….. 39

3.1.2.2. Calculation Layer Atomic Model ………………………………………………………………. 40

3.1.3. Back-Propagation Learning Model Set …………………………………………………………….. 40

3.1.3.1. Error-Generator Atomic Model …………………………………………………………………. 40

3.1.3.2. Delta-Weight Atomic Model …………………………………………………………………….. 41

3.2. HiLLS Description of DEVS-Based ANN………………………………………………………………. 41

3.2.1. Input Generator (IGEN)………………………………………………………………………………….. 42

3.2.2. Non-Calculation Layer Atomic Model (NC)……………………………………………………… 42

3.2.3. Calculation Layer Atomic Model (CAL) ………………………………………………………….. 44

3.2.4. Target Generator (TGEN) ………………………………………………………………………………. 45

viii

3.2.5. Error Generator (ERR) …………………………………………………………………………………… 46

3.2.6. Delta-Weight Atomic Model (DW) ………………………………………………………………….. 47

3.2.7. DEVS-Based ANN Coupled Model …………………………………………………………………. 48

3.3. SimStudio Implementation ……………………………………………………………………………………. 49

3.4. User Interface ……………………………………………………………………………………………………… 50

4.0 APPLICATION OF DEVS-BASED ANN ……………………………………………………………… 56

4.1. Presentation of the Case Studies ……………………………………………………………………………. 56

4.2. Data Extraction ……………………………………………………………………………………………………. 57

4.2.1. Raw Data Presentation……………………………………………………………………………………. 57

4.2.2. Data Normalization ………………………………………………………………………………………… 58

4.2.2.1. Statistical or Z-Score Normalization ………………………………………………………….. 58

4.2.2.2. Min-Max Normalization …………………………………………………………………………… 58

4.3. Results ……………………………………………………………………………………………………………….. 60

5.0 CONCLUSION …………………………………………………………………………………………………… 63

5.1. Summary of Work Done ………………………………………………………………………………………………. 63

5.2. Pros and Cons …………………………………………………………………………………………………………….. 63

5.3. Future Works ……………………………………………………………………………………………………………… 64

**CHAPTER ONE**

1.0 INTRODUCTION

1.1. Context

Modeling and Simulation (M&S), the third pillar of science is a paradigm that provides a way of obtaining the behavior of the representation of an object in real life without doing physical experiments. As introduced by the theory of Modeling and simulation [2], there are four major important concepts of M&S. The concepts are defined below:

a) System: is a well-defined object in the real world under specific conditions that we are interested in modeling.

b) Experimental Frame (EF): is a specification of the conditions within which the system is observed or experimented. It is realized as a system (with generators, acceptors and transducers) that interacts with the source system to obtain data of interest under specified conditions.

c) Model: is an abstract representation of the structure and properties of a system at some particular point in time or space intended to promote understanding of the real system.

d) Simulation: is the execution of a model over time in order to get the information about the changes in the behavior of the system during executions.

Modeling complex systems requires a robust formalism. The Discrete Event System Specification (DEVS) formalism [3] that was introduced in the early 70’s is a theoretically well-defined formalism for modeling discrete event systems in a hierarchical and modular manner. It allows the behavior modeling of complex systems.

Artificial Neural Networks (ANN) is a branch of artificial intelligence that became popular in the eighties when the back-propagation algorithm [4] for multilayer feed-forward architectures was introduced. It is widely known that classical neural networks, even with one hidden layer, are universal function approximators [5]. ANNs became widely applicable for real applications when it had the capabilities to solve non-linear problems. It is used for modeling of complex optimization problems such as classification, prediction and pattern recognition.

2

Artificial neural network is capable of modeling complex non-linear systems using adaptive learning mechanism to derive meaning from complicated or imprecise data with a high degree of accuracy. However, ANN uses a black box learning approach – when the general architecture is defined, you almost don’t have an idea of how the output is produced. To overcome this, DEVS is combined with ANN to express the relationship between the input and output. Combining DEVS and ANN is possible because ANNs are by default using discrete events i.e. the network is always waiting to an input event to generate an output one. Toma et al [1] proposed an approach for the describing the structure of ANN with DEVS known as DEVS-Based ANN. This approach was said to be able to facilitate the network configuration that depends a lot on ANN.

We propose to extend the work in [1] for comparing multiple configuration parameters and learning algorithms. The configuration parameters for ANNs are number of hidden layers, output neurons for each layer and stopping condition (minimum error or maximum number of iterations) to avoid over-training. This will help users and developers test and compare different algorithm implementations and parameter configurations. A new visual modeling language, High Level for System Specification [6] (HiLLS) which is an extension of DEVS Driven Modeling Language (DDML) [7] will be used to describe the approach for clear understanding.

1.2. Research Objectives

The objective of this work is to achieve a DEVS-based Modeling and Simulation (M&S) platform that will allow the specification of ANNs and their configuration (training) and use for prediction as well. We aim to build a generic systems dynamic-based model for ANNs specification and training with HiLLS – a graphical representation of DEVS and implement the DEVS models with the SimStudio M&S package for ANN learning algorithms comparisons. The built DEVS-Based ANN platform will be benchmarked with study cases.

1.3. Related Works

Neural networks have been used extensively to determine the behavior of discrete event systems because it can learn from empirical data but little research is focused on using discrete event modeling to specify the structure and components of a neural network.

3

1.3.1. Abstraction of Continuous System to Discrete Event System Using Neural Network

Sung Hoon Jung and Tag Gon Kim in [8] explored the use of Artificial Neural Networks in modeling of a hybrid system which consists of continuous systems and discrete event systems. The continuous system cannot directly communicate with the discrete event system. Therefore, they proposed neural networks as an interface for communication between the two disjoint systems. To achieve this, the continuous system was first represented by a timed state transition model and the model is then mapped into a neural network by learning capability of the network.

The timed state transition model was obtained through 3 steps – output partitioning, input sampling and measuring of delay. From the information gotten they specified a 6 tuple model based on DEVS [9] model.

is the input events set obtained from input sampling

is the output events set obtained from output partitioning

is the states set

is the time delay function

is the output function

is the initial state in

Figure 1.1: State Trajectories of Differential Equation Model and Neural Network Model [8]

4

The table function is a tedious and difficult one, hence, they trained a neural network to obtain the state transition rules using state variables, inputs and outputs as the neural networks input and time delay (since it is a real number) as the neural network output. The approach was applied to Water Tank Continuous System and the results of the table function were compared to that of differential equations which shows an impressing result. (See figure 1.1).

1.3.2. Identification of Discrete Event Systems: Using the Compound Recurrent Neural Network: Extracting DEVS from Trained Network

Si Jong Choi and Tag Gon Kim [10] identified Discrete Event Systems (DES) from a compound recurrent neural network (CRNN) by 2 major steps: behavior learning using a specially signed neural network and extraction of a DEVS model from the neural network. The neural network was trained using observed input/ output events of an unknown DES.

After training of CRNN, it is behaviorally equivalent to the explored DEVS model. The extracted DEVS model is validated against the timed I/O event sequences of the target DES. As a result of identification process, an unknown target DES is identified as a single DEVS model or equivalent DEVS model

1.3.3. Neuro-DEVS, an Hybrid Methodology to describe Complex Systems

Jean-Baptiste Filipi, Paul Bisgambigia and Marielle Delhom [11] presented an Object Oriented Modeling and Simulation (OOMS) of Complex Systems based on DEVS and neural networks. For OOMS to be implemented efficiently, the model behavior has to be well known. Neural networks can learn from empirical data to obtain a systems behavior. Therefore, they extended

Target DES

DEVS Model

Validation

CRNN

CRNN Training

Rule Extraction

Figure 1.2: Identification of Discrete Event

5

the OOMS environment with neural network objects known as adaptive models or neural net sub-models.

They proposed Neuro-DEVS extended the DEVS atomic model with some neural networks essential functions like activation function, learning function and connection links. The OOMS coupled model now has Neuro-DEVS has one of its atomic models. The hybrid system offers better simulation in the following ways:

a) Concurrent simulation: useful for comparing neural network outputs with the output of a simple model. This helps to validate the results of neural networks.

b) Adaptive models can be used to modify the neural network runtime according to an error feedback [12]. The error is the difference between the model’s forecast and the real world data collected afterwards.

c) ANN as a sub-component can be used if Neural Networks provides better results for only a piece of the whole system.

1.3.4. Dynamic Neuronal Ensembles (DNE): Neurobiologically Inspired Discrete Event Neural Networks

S. Vahle [13] made use of discrete event modeling formalism to model the neurobiological components of the neural network in order to increase the computational power, adaptability and dynamic response of ANNs. He proposed a DEVS model called Dynamic Neuronal Ensembles (DNE). The DNE is a coupled model of components that are themselves coupled models called dynamic neurons (DN).

Figure 1.3: The Dynamic Neuron [13]

6

The proposed dynamic neuron coupled model is composed of three basic atomic models: the dendrite, the cell body and the axon. The dendrite receives the inputs from axonal output connections, adjusts each according to its weighting requirements and requests connection modification. The cell body adds the input signals received from its attached dendrites to its decayed potential level. A fire signal is sent to the cell body’s attached axons if the potential level meets or exceeds the thresholds. The axon produces an output when it receives a fire signal.

1.3.5. A New DEVS-Based Generic Artificial Neural Network Modeling Approach

S. Toma, L. Capocchi, and D. Federici [1] described the structure of ANN with DEVS atomic and coupled models. They pointed out that ANNs are by default using discrete event; the network is always waiting to an input event to generate an output one. S. Toma [14] described three possible ways by which message can be transferred in ANN/DEVS mapping. In figure below, we have (a) neuron architecture level, (b) layer architecture level and (c) and reduced-layer architecture level. In the neuron architecture, each neuron is an atomic model – this offers good visual representation of ANN but the simulation will be slow because of large number of messages sent between neurons. On the other hand, in layer architecture, each layer is an atomic model with less visual representation of the neuron connections; it clearly shows the number of outputs from one layer to another and lesser messages is transferred during simulation. The reduced-layer architecture has the least number of messages transferred between atomic models which ensure fast simulation but no graphical representation for the number of neurons in the each layer. The layer architecture level is preferred.

Figure 1.4: The Dynamic Neuron [14]

7

They used four atomic models to describe a multilayer ANN: non-calculation layer (input layer), calculation layer (hidden and output layers), error generator and delta weight. The non-calculation layer forwards inputs to the first hidden layer through linear activation function ( ) . The calculation layer will compute the neuron output using non-linear activation functions. The calculated output is compared with the target output in the error generator model, and the error difference is forwarded to the delta weight model for error correction. This approach will be able to facilitate the network configuration that depends a lot on the application. In other words the new model will be able to give the space to implement algorithms and plug-ins to automate the network configuration as the network efficiency.

,

1.4. Approach Adopted

In this work, we reviewed Artificial Neural Networks learning algorithms based on gradient descent such as Back propagation, Back propagation with momentum, Silva & Almeida, Delta-Bar-Delta, Quickprop and Resilient Propagation. Common activation functions that are continuous, non-linear and differentiable such as Binary sigmoid function, Bipolar sigmoid function, Hyperbolic tangent function and Gaussian functions were also reviewed.

HiLLS visual modeling language is then used to describe the DEVS-Based ANN model taking into consideration the parameter configurations such as number of hidden layers, output neurons for each layer and stopping condition (minimum error or maximum number of iterations).

The DEVS model is then implemented with SimStudio [15] package and Java programing language. The Graphical DEVS-Based ANN platform built is able to compare different learning algorithms and activation functions. It will also be benchmarked with case studies.

1.5. Organization of Work

This work is organized as follows: Chapter 2 introduces the Artificial Neural Networks architecture and learning algorithms; DEVS formalism; and HiLLS visual modeling language. Chapter 3 presents the DEVS-Based ANN approach, HiLLS specification and SimStudio implementation. Chapter 4 presents the case studies, data extraction and results. Chapter 5 provides the summary of the work done, pros and cons and future works.

8