diff options
Diffstat (limited to 'doc/tex/systemDesign.tex')
| -rw-r--r-- | doc/tex/systemDesign.tex | 315 |
1 files changed, 229 insertions, 86 deletions
diff --git a/doc/tex/systemDesign.tex b/doc/tex/systemDesign.tex index 80a2ccb..e166955 100644 --- a/doc/tex/systemDesign.tex +++ b/doc/tex/systemDesign.tex @@ -1,5 +1,8 @@ \section{System Design} +This section explains the design of the component systems of \program{}, the +algorithms used and how they are to be implemented. + \subsection{Class Diagram} The full class diagram is shown in \fref{fig:fullClasses}. @@ -15,7 +18,7 @@ The full class diagram is shown in \fref{fig:fullClasses}. \end{figure} The design of each system of the diagram is explained after this section -together with diagrams for each subsystem, since the full class diagram can be +along with diagrams for each subsystem, since the full class diagram can be too big to be comfortably analyzed. \subsection{Game} @@ -30,24 +33,24 @@ too big to be comfortably analyzed. \end{figure} A regular Go match is composed of a list of moves. But since game review and -variants exploration is an important part of Go learning, \program{} and most +exploration of variants is an important part of Go learning, \program{} and most playing and analysis existing programs allow for navigation back and forth through the board states of a match and for new variants to be created from each of these board states. Therefore, a match is represented as a tree of moves. The -GameMove class has the information about a specific move and also a reference to -the previous move and to a list of following moves, implementing this tree -structure and allowing for navigating both forward and backwards in the move -history. +\texttt{GameMove} class has the information about a specific move and also a +reference to the previous move and to a list of following moves, implementing +this tree structure and allowing for navigating both forward and backwards in +the move history. The state of the board at any given move must be stored so liberties, captures count and legality of moves can be addressed, so it is represented with the -GameState class, which holds a reference to the current move. +\texttt{GameState} class, which holds a reference to the current move. Moves depend on a representation of the game board to have access to its current layout and count of captured stones. There are also many logic operations needed to be performed on the board, such as getting the stones in a group, counting their liberties or checking if a move is playable. The layout of the board and -these operations are implemented as the GameBoard class. +these operations are implemented as the \texttt{GameBoard} class. A game can be started by the executable \texttt{go.py}. @@ -58,15 +61,15 @@ These classes and their relationships can be seen in \fref{fig:game}. \begin{figure}[h] \begin{center} \includegraphics[width=0.8\textwidth]{diagrams/engineModule.png} - \caption{Design of the GTP engine.}\label{fig:engine} + \caption{Design of the \acrshort{gtp} engine.}\label{fig:engine} \end{center} \end{figure} -An implementation of GTP, that is, the piece of software which offers the GTP -interface to other applications. It is designed to be used by a software -controller but can also be directly run, mostly for debugging purposes. Its -design is shown in \fref{fig:engine}. The core of the engine is related with -three components, each with a separate responsibility: +This will be an implementation of \acrshort{gtp}, that is, the piece of software +which offers the \acrshort{gtp} interface to other applications. It is designed +to be used by a software controller but can also be directly run, mostly for +debugging purposes. Its design is shown in \fref{fig:engine}. The core of the +engine is related with three components, each with a separate responsibility: \begin{itemize} @@ -74,63 +77,68 @@ three components, each with a separate responsibility: applications and offers the text interface. It reads and processes input and calls corresponding commands from the core of the engine. - \item The GameEngine contains the logic of the commands available from the - IO component. It uses a GameState to keep a record of the game and uses - a DecisionAlgorithm to generate moves. + \item The \texttt{GameEngine} contains the logic of the commands available + from the IO component. It uses a \texttt{GameState} to keep a record of + the game and uses a \texttt{DecisionAlgorithm} to generate moves. - \item The DecisionAlgorithm component is responsible of analyzing the match - and generate moves. The engine core uses it when a decision has to be - made by the AI, such as when a move needs to be generated by the engine. + \item The \texttt{DecisionAlgorithm} component is responsible of analyzing + the match and generate moves. The engine core uses it when a decision + has to be made by the \acrshort{ai}, such as when a move needs to be + generated by the engine. \end{itemize} -Two implementations of DecisionAlgorithm have been made: one for the Monte -Carlo Tree Search algorithm (on the MCTS class) and the other for neural -networks (on the Keras class). +Two implementations of \texttt{DecisionAlgorithm} have been made: one for the +Monte Carlo Tree Search algorithm (on the \texttt{MCTS} class) and the other for +neural networks (on the \texttt{Keras} class). -The Keras class also makes use of the NeuralNetwork class, which offers -functions for creating, training, saving and using neural network models. The -designs of the network are implemented in the subclasses DenseNeuralNetwork and -ConvNeuralNetwork as examples of dense and convolutional networks, respectively. +The \texttt{Keras} class also makes use of the \texttt{NeuralNetwork} class, +which offers functions for creating, training, saving and using neural network +models. The designs of the network are implemented in the subclasses +\texttt{DenseNeuralNetwork} and \texttt{ConvNeuralNetwork} as examples of dense +and convolutional networks, respectively. The engine can be started with the executable \texttt{imagocli.py}. \subsubsection{Monte Carlo Tree Search Explained} Monte Carlo Tree Search is an algorithm that can be useful for exploring -decision trees. It was used by AlphaGo in conjunction with neural networks as -explained in the AlphaGo 2016 paper\cite{natureAlphaGo2016}. +decision trees. It has a history of use in Go engines, not being able to reach +strong levels of play until it was paired with neural networks by AlphaGo as +explained in the AlphaGo 2016 paper \cite{natureAlphaGo2016}. -The algorithm assigns a score to each explored node based on how likely the -player who makes the corresponding move is to win and updates this score on each -exploration cycle. +The algorithm assigns a score to each explored node of the game tree based on +how likely the player who makes the corresponding move is to win and updates +this score on each exploration cycle. The exploration of the tree has 4 steps: \begin{enumerate} - \item \textbf{Selection}: The most promising move with unexplored children - is selected for exploration. Unexplored children are viable moves which - are not yet part of the tree. + \item \textbf{Selection}: The most promising move with at least one + unexplored children is selected for exploration. Unexplored children are + viable moves which are not yet part of the tree. \item \textbf{Expansion}: An unexplored children of the selected move is added to the tree. This children is selected at random. \item \textbf{Simulation}: The score of the new move is evaluated by playing - random matches from it. + different matches from it. How this matches are played varies: they can + be totally random, but here is where AlphaGo introduces one of its + neural networks so these simulation matches are more valuable. - \item \textbf{Backpropagation}: The score of the new move, as well as its - previous moves up to the root of the tree, is updated based on the - results of the simulation. + \item \textbf{Backpropagation}: The scores of the new move and its previous + moves up to the root of the tree are updated based on the results of the + simulation. \end{enumerate} The suggested move is the children of the current move with the best score from the perspective of the player which has to make the move. -The implementation of the algorithm will use the existing GameMove class from -the Game System to access the game logic it needs, such as to get the possible -children from a node or to simulate random games. +The implementation of the algorithm will use the existing \texttt{GameMove} +class from the Game System to access the game logic it needs, such as to get the +available children from a node or to simulate random games. \subsubsection{Neural Networks Explained} @@ -154,13 +162,13 @@ Several kinds of layers have been used in this project: \item \textbf{Dense layers}, which connects each of its nodes to each of the nodes of the previous layers. - \item \textbf{Convolutional layers}, which process their input by applying - a filter function to adjacent values. In the case of this project, the + \item \textbf{Convolutional layers}, which process their input by applying a + filter function to adjacent values. In the case of this project, the board is filtered by grouping its vertices in 3x3 squares. The aim of these layers is to detect patterns in the input, such as curves, edges - or more complex shapes, so they are used a lot on neural networks - processing images. They are used in this project because a configuration - of the Go board is not that different from a two-dimensional image. + or more complex shapes, so they are used a lot on image processing. They + are used in this project because a configuration of the Go board is not + that different from a two-dimensional image. \item \textbf{Max pooling layers}, which process their input in a similar way to convolutional layers, but reducing the size of the input by @@ -176,19 +184,19 @@ Several kinds of layers have been used in this project: Combinations of these layers have been used to define two neural networks. First, a network using mainly dense layers as an example of a more general -purpose design of a network. +purpose and baseline design of a network. Then, a network with convolutional and max pooling layers to compare the -approach used on image processing to the more general one and studying its -utility on the analysis of the Go board. +approach used on image processing to the more general one and study its utility +on the analysis of the Go board. -These networks have been implemented on the DenseNeuralNetwork and -ConvNeuralNetwork classes, respectively. +These networks have been implemented on the \texttt{DenseNeuralNetwork} and \\ +\texttt{ConvNeuralNetwork} classes, respectively. The networks have been designed to process boards of size 9x9, which is the introductory size to the game. It is the easiest both for the hardware to handle -and for the analysis of results while keeping able to support meaningful -matches. +and for the analysis of results while still being big enough to support +meaningful matches. Both networks have the same design for their input and output. @@ -224,7 +232,7 @@ been selected so each node can have as input each of the vertices of the board. A flatten layer acts then to make the output one-dimensional, and a final dense layer provides the vector containing the likelihood of each possible move. -The design of this network is shown in \flist{code:denseModel} and +The design of this network is shown in \lref{code:denseModel} and \fref{fig:denseNN} as provided by Keras' summary and plot\_model functions respectively. @@ -249,7 +257,7 @@ of being trained to recognize patterns on the board. A flatten layer acts then to make the output one-dimensional, and a final dense layer provides the vector containing the likelihood of each possible move. -The design of this network is shown in \flist{code:convModel} and +The design of this network is shown in \lref{code:convModel} and \fref{fig:convNN} as provided by Keras' summary and plot\_model functions respectively. @@ -257,8 +265,8 @@ respectively. \begin{figure}[h] \begin{center} - \includegraphics[width=\textwidth]{diagrams/trainingModule.png} - \caption{Components of the SGF file parsing module.} + \includegraphics[width=0.7\textwidth]{diagrams/trainingModule.png} + \caption{Components of the \acrshort{sgf} file parsing module.} \label{fig:trainingModule} Components not showing a capital C are not classes, as in they not follow the object-oriented paradigm and do not implement any classes, @@ -267,19 +275,20 @@ respectively. \end{figure} Neural networks can be powerful machine learning algorithms, but they have to be -trained first so they can provide meaningful results. For a Go AI it makes sense -to have its algorithms trained on Go games. There exists a common text format to -store Go games: SGF. If the system is able to process SGF files, it can provide -the games stored on them to the neural networks for training. And so the need -for an SGF parser arises. - -To parse SGF files a lexer and parser have been implemented using PLY.\@ The -result of the parsing is an AST (Annotated Syntax Tree) reflecting the contents -of the text input, each node with zero or more properties, and with the ability -to convert themselves and their corresponding subtree into a GameMove tree. This -is done for the root node, since from the SGF specification there are some -properties only usable in the root node, like those which specify general game -information and properties such as rank of players or komi. +trained first so they can provide meaningful results. For a Go \acrshort{ai} it +makes sense to have its algorithms trained on Go games. There exists a common +text format to store Go games: \acrshort{sgf}. If the system is able to process +\acrshort{sgf} files, it can provide the games stored on them to the neural +networks for training. And so the need for an \acrshort{sgf} parser arises. + +To parse \acrshort{sgf} files a lexer and parser have been implemented using PLY +\cite{ply}. The result of the parsing is an \acrfull{ast} reflecting the +contents of the text input, each node with zero or more properties, and with the +ability to convert themselves and their corresponding subtree into a +\texttt{GameMove} tree. This is done for the root node, since from the +\acrshort{sgf} specification there are some properties only usable in the root +node, like those which specify general game information and properties such as +rank of players or \gls{komi}. Here follows an explanation of the role and motivation before each component of the Training module to show how these previous concerns have been addressed and @@ -287,24 +296,25 @@ solved. These components are shown in \fref{fig:trainingModule}. \begin{itemize} - \item \textbf{SGF}: Provides a high-level method to convert a path to a SGF - file to a GameMove tree. + \item \textbf{\texttt{\acrshort{sgf}}}: Provides a high-level method to convert a path to a \acrshort{sgf} + file to a \texttt{GameMove} tree. - \item \textbf{sgfyacc}: The implementation of a SGF parser using PLY. Takes - the tokens generated by \textbf{sgflex} and creates an ASTNode tree from - them. + \item \textbf{\texttt{sgfyacc}}: The implementation of a \acrshort{sgf} parser using PLY. Takes + the tokens generated by \texttt{sgflex} and creates an \texttt{ASTNode} + tree from them. - \item \textbf{sgflex}: The implementation of a SGF lexer using PLY.\@ Takes - text input and generates the tokens of the SGF language from them. + \item \textbf{\texttt{sgflex}}: The implementation of a \acrshort{sgf} lexer using + PLY.\@ Takes text input and generates the tokens of the \acrshort{sgf} + language from them. - \item \textbf{ASTNode}: The AST resulting from the parsing of a SGF file. - Has a method to convert it to a tree of GameMove and so obtain the - contents of the SGF in the internal representation used by the project's - systems. + \item \textbf{\texttt{ASTNode}}: The AST resulting from the parsing of a + \acrshort{sgf} file. Has a method to convert it to a tree of + \texttt{GameMove} and so obtain the contents of the \acrshort{sgf} in + the internal representation used by the project's systems. - \item \textbf{Property}: The representation of a property of an ASTNode - tree. Each property is made of a name and one or more values and this - class helps handling this specific situation. + \item \textbf{\texttt{Property}}: The representation of a property of an + \texttt{ASTNode} tree. Each property is made of a name and one or more + values and this class helps handling this specific situation. The training can be started with the executable \texttt{train.py}. @@ -313,7 +323,7 @@ The training can be started with the executable \texttt{train.py}. %\subsection{Modules} % %One module to store the state of the game and the game tree. One module to parse -%moves. One module to read and write SGF files. Modules are shown in +%moves. One module to read and write \acrshort{sgf} files. Modules are shown in %\fref{fig:modules}. % %\begin{figure}[h] @@ -322,3 +332,136 @@ The training can be started with the executable \texttt{train.py}. % \caption{Modules.}\label{fig:modules} % \end{center} %\end{figure} + +\subsection{Technical Testing Plan Specification} + +This section lists and explains the exact testing plan. + +\subsubsection{Unitary Testing} + +Tests for the Python code will be developed using the unittest +\cite{python_unittest} testing framework. It has been chosen by virtue of being +thoroughly documented and widely used. + +The coverage of unit testing will be checked with Coverage.py +\cite{python_coverage}, which can by itself run the unittest tests and generate +coverage reports based on the results. + +\subsubsection{Integration Testing} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.4\linewidth}p{0.5\linewidth}} + \toprule + \multicolumn{2}{c}{\textbf{Engine and Game modules}} \\ + \midrule + \textbf{Test} & \textbf{Expected behaviour} \\ + \midrule + The GTP interface of the engine is used to play a match & The module handles + the game and can show its state. \\ + \bottomrule +\end{tabular} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.4\linewidth}p{0.5\linewidth}} + \toprule + \multicolumn{2}{c}{\textbf{Training and Engine module}} \\ + \midrule + \textbf{Test} & \textbf{Expected behaviour} \\ + \midrule + The training process is started & The training uses the network defined on + the Engine module. \\ + \bottomrule +\end{tabular} + +\subsubsection{System Testing} + +These are the tests to check the correct working of the system as a whole. The +tests are grouped by the interface they are run against. + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.4\linewidth}p{0.5\linewidth}} + \toprule + \multicolumn{2}{c}{\textbf{Game interface}} \\ + \midrule + \textbf{Test} & \textbf{Expected behaviour} \\ + \midrule + Play a game of Go with no engine & The game can be played until the end. \\ + \midrule + Provide a wrong move & The interface shows it is wrong and the game + continues without a change of state. \\ + \midrule + Close the game & The interface closes. \\ + \bottomrule +\end{tabular} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.4\linewidth}p{0.5\linewidth}} + \toprule + \multicolumn{2}{c}{\textbf{Engine interface}} \\ + \midrule + \textbf{Test} & \textbf{Expected behaviour} \\ + \midrule + Ask for the available commands & The interface outputs the available + commands. \\ + \midrule + Provide a move & The state of the engine updates with the new move. \\ + \midrule + Ask for a move & The engine suggests a move without changing the state of + the current game. \\ + \bottomrule +\end{tabular} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.4\linewidth}p{0.5\linewidth}} + \toprule + \multicolumn{2}{c}{\textbf{Training interface}} \\ + \midrule + \textbf{Test} & \textbf{Expected behaviour} \\ + \midrule + Provide some games to train on & A neural network model is created. \\ + \midrule + Start the training without providing games & An error message is shown and + the execution terminated. \\ + \bottomrule +\end{tabular} + +\subsubsection{Usability Testing} + +Two different human users will be exposed to the interfaces of the project and +asked to answer a questionary about their experience. The aim of this process is +to identify and address any problems end users could have when using the +system. + +As the training of the neural networks is part of the preparation of the system, +its usability will not be tested for end users. + +These are the questions provided to the testers. + +\begin{itemize} + + \item Playing against a human: + \begin{itemize} + \item Were you able to start the interface? + \item How hard was the interface of the game to understand? + \end{itemize} + + \item Playing against the engine: + \begin{itemize} + \item Were you able to start the interface? + \item How hard was the interface of the game to understand? + \item How strong did you find the engine? + \end{itemize} + + \item Playing against the interface through a third-party \acrshort{gui}: + \begin{itemize} + \item Were you able to start the interface? + \item Did you find any problems when setting up the engine? + \item Do you think this tool has value for studying Go? + \end{itemize} + +\end{itemize} |