\section{System Analysis} \subsection{System Reach Determination} These are the main goals the final product must reach. \begin{enumerate} \item The implementation, analysis and comparison of different decision algorithms for genarating moves. This is the main goal and the following ones are derived from the need of reaching it. \item A library for representing the game of Go. It can be used for the decision algorithms to keep the state of the game and can also be used in an interactive application for a user to play the game and test the code. \item An engine program as a way of presenting an interface for using these algorithms. The engine will use the GTP so it can be used with an existing GUI or other tools. \item A parser for SGF files so they can be processed in the training of neural networks. \end{enumerate} \subsection{System Requirements} The requirements for the system are expressed here in a nested list way, each of them with a textual and numeric reference for them to be traceable. The functional requirements are exposed first, followed by the other kinds of requisites needed for the system. \setlist[enumerate,2]{label*=\arabic*.} \setlist[enumerate,3]{label*=\arabic*.} \subsubsection{Functional Requirements} \paragraph{Game Requirements} \setlist[enumerate,1]{label=FRG \arabic*.} \begin{enumerate} \item The game program is interactive. \item Movements can be introduced to be played on the board. \begin{enumerate} \item A move is introduced as the textual representation of the coordinates of the vertex to play on or as ``pass''. \begin{enumerate} \item The text introduced for the move must follow the regular expression \texttt{([A-Z][0-9]+|pass)} \item If the move is not valid it must be notified to the user and another move asked for. \end{enumerate} \end{enumerate} \item The state of the board can be shown to the user. \begin{enumerate} \item A text representation of each cell is printed. \begin{enumerate} \item A different character is used for each different state of a cell. \end{enumerate} \item The coordinates system is shown around the board. \begin{enumerate} \item Columns are shown as capital letters left to right starting with ``A'' and skipping ``I''. \item Rows are shown as numbers starting with 1 on the lowest row and increasing upwards. \end{enumerate} \end{enumerate} \item The board will behave according to the Japanese rules of Go. \end{enumerate} \paragraph{Engine Requirements} \setlist[enumerate,1]{label=FRE \arabic*.} \begin{enumerate} \item The engine program is interactive. \item The engine implements the GTP (\textit{Go Text Protocol}) for its interface. \begin{enumerate} \item Commands are read from standard input. \item Responses are provided via standard output. \item There exist commands to set up the conditions of the match. \begin{enumerate} \item The size of the board can be set. \item The komi can be set. \end{enumerate} \item There exist commands to manipulate the internal representation of the match. \begin{enumerate} \item It is possible to indicate a move being played. \item It is possible to clear the board. \end{enumerate} \item There exists a command to generate a move. \begin{enumerate} \item The generated move must be a playable move. \item Generating a move does not change the internal representation of the match. \end{enumerate} \item There exist commands to ask for information about the engine. \begin{enumerate} \item It is possible to ask for the protocol version implemented. \item It is possible to ask for the name of the engine. \item It is possible to ask for the version of the engine. \item It is possible to ask whether a specific command is known to the engine. \item It is possible to ask for a list of the known commands. \end{enumerate} \item There exists a command to stop the engine. \end{enumerate} \item The engine can be executed from the command line. \begin{enumerate} \item The engine can be executed directly from an interactive shell. \item The engine can be executed by another program to be used as backend. \end{enumerate} \end{enumerate} \paragraph{Trainer Requirements} \setlist[enumerate,1]{label=FRT \arabic*.} \begin{enumerate} \item The trainer program is non-interactive. \item The trainer can be executed from the command line. \begin{enumerate} \item The trainer can be executed directly from an interactive shell. \end{enumerate} \item The trainer can interact with stored neural network models. \begin{enumerate} \item The trainer can read stored models to continue training them. \item The trainer can store model files after their training. \end{enumerate} \item The trainer can import existing games. \begin{enumerate} \item Records of games stored as SGF can be imported. \item Files containing records of games are provided as arguments to the trainer. \end{enumerate} \end{enumerate} %\subsubsection{Security Requirements} % %\setlist[enumerate,1]{label=SR \arabic*.} \subsubsection{Usability Requirements} \setlist[enumerate,1]{label=UR \arabic*.} \begin{enumerate} %TODO: Implement this \item The engine executable will include a help option with the different modes of execution. \end{enumerate} \subsubsection{User Requirements} \setlist[enumerate,1]{label=USR \arabic*.} \begin{enumerate} \item For understanding the workings of the application the user needs to be familiar with the basics of the game of Go. \item For directly using the engine the user needs to be familiar with command line interfaces. \item For directly using the trainer the user needs to know the different network models available. \end{enumerate} \subsubsection{Technological Requirements} \setlist[enumerate,1]{label=TR \arabic*.} \begin{enumerate} \item The game program will be a python file able to be executed by the python interpreter. \item The game program will make use of standard input and standard output for communication. \begin{enumerate} \item Standard input will be used for reading moves. \item Standard output will be used for showing the board. \item Standard output will be used for messages directed to the user. \end{enumerate} \item The engine program will be a python file able to be executed by the python interpreter. \item The engine program will make use of standard input and standard output for communication. \begin{enumerate} \item Standard input will be used for reading commands. \item Standard output will be used for showing the result of commands. \end{enumerate} \item The trainer program will be a python file able to be executed by the python interpreter. \item The engine program will make use of standard input and standard output for communication. \begin{enumerate} \item Standard input will be used for reading commands. \item Standard output will be used for showing the result of commands. \end{enumerate} \end{enumerate} \subsubsection{Response Time Requirements} \setlist[enumerate,1]{label=RTR \arabic*.} \begin{enumerate} %TODO: Check and update this to something feasible \item The maximum thinking time of the engine will be configurable. \begin{enumerate} \item It will be possible to pass the maximum time as a launch argument. \item It will be possible to store the maximum time as a setting in a configuration file. \end{enumerate} \end{enumerate} \setlist[enumerate,1]{label=\arabic*.} \subsection{Subsystems} There will be three main subsystems. \subsubsection{Game System} The Game System will be in charge of storing all the state information regarding a Go match, such as the history of moves, the possible variations, the state of the board at any given time or the current number of captured stones. This system will include a command-line interface with which to play Go matches between human players to show and test its capabilities. \subsubsection{Engine System} The Engine System will implement the GTP interface and use the Game System to analyse positions and generate moves via decision algorithms. This system can be directly called to manually set up game states and ask for moves or can be called by other programs which use GTP to be used as backend for playing matches against a computer player. \subsubsection{Training System} The Training System will process SGF files storing records of games, train the neural network models over those games and store the result. These models can then be imported by the engine and be used to generate moves. \subsubsection{Interface Between Subsystems} The Training System depends on the NeuralNetwork interface of the Engine System and uses it to train and store the neural network models. Both the Engine and Training systems depend on the GameMove class of the Game System. The Engine System uses it to store the state of a game and provide it to the decision algorithms. The Training System uses it to create the internal representation of a game resulting from the processing of an SGF file. \subsection{Class Analysis} \subsubsection{Class Diagram} The classes resulting from the analysis phase are shown in \fref{fig:analysisClasses}. \begin{figure}[h] \begin{center} \includegraphics[width=\textwidth]{diagrams/analysisClasses.png} \caption{General classes obtained from the analysis phase.}\label{fig:analysisClasses} \end{center} \end{figure} \subsubsection{Class Description} \newcommand{\interclassSpace}{30pt} \paragraph{Engine System} \indent \\ \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{EngineIO} \\ \midrule \textbf{Description} \\ Offers the interface with the engine. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Read input.} \\ \tabitem{Do some preprocessing.} \\ \tabitem{Forward commands to the engine logic component.} \\ \midrule \textbf{Proposed attributes} \\ \midrule \textbf{Proposed methods} \\ \tabitem{\textbf{start()}: Starts reading standard input in a loop.} \\ \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{EngineLogic} \\ \midrule \textbf{Description} \\ Does the core logic and connects the different components of the engine. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Processes the commands and arguments forwarded by the IO component.} \\ \tabitem{Handles the logic of the game by using components from the game module.} \\ \tabitem{Calls a decision algorithm to generate moves.} \\ \midrule \textbf{Proposed attributes} \\ \midrule \textbf{Proposed methods} \\ \bottomrule \end{tabular} \vspace{\interclassSpace} \newcommand{\decisionAlgorithmMethods}{ \tabitem{\textbf{pickMove()}: Gives a move to play.} \\ \tabitem{\textbf{forceNextMove(coords)}: Notifies the system of a played move so it can update its state accordingly.} \\ \tabitem{\textbf{clearBoard()}: Empties the move history. The algorithm will now generate moves as from a new game.} \\ } \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{DecisionAlgorithm} \\ \midrule \textbf{Description} \\ Interface for the decision algorithms to be used by the engine. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Analyzing game states and generating moves.} \\ \midrule \textbf{Proposed attributes} \\ \textit{(Depends on the algorithm.)} \\ \midrule \textbf{Proposed methods} \\ \decisionAlgorithmMethods \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{MonteCarloTreeSearch} \\ \midrule \textbf{Description} \\ Implements the Monte Carlo Tree Search algorithm for exploring the tree of the game and deciding on a move. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Analyzing game states and generating moves.} \\ \midrule \textbf{Proposed attributes} \\ \tabitem{\textbf{root}: The root node of a tree representing of the current game state and the explored possible moves from it.} \\ \midrule \textbf{Proposed methods} \\ \decisionAlgorithmMethods \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{MCTSNode} \\ \midrule \textbf{Description} \\ A node of the tree used by the Monte Carlo Tree Search algorithm. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Storing a specific state of a match.} \\ \midrule \textbf{Proposed attributes} \\ \tabitem{\textbf{visits}: How many times the node has been visited.} \\ \tabitem{\textbf{score}: The number of explorations of the node resulting in victory.} \\ \tabitem{\textbf{move}: A GameMove for accessing game state and logic.} \\ \tabitem{\textbf{parent}: This node's parent in the tree.} \\ \tabitem{\textbf{children}: The nodes following from this node in the tree.} \\ \tabitem{\textbf{unexploredVertices}: The plays which could be explored from this node.} \\ \midrule \textbf{Proposed methods} \\ \tabitem{\textbf{ucbForPlayer()}: Computes the Upper Confidence Bound of the node from the perspective of the player making the move stored in the node.} \\ \tabitem{\textbf{selection()}: Monte Carlo Tree Search selection step. Selects the most promising node which still has some unexplored children.} \\ \tabitem{\textbf{expansion()}: Monte Carlo Tree Search expansion step. Picks an unexplored vertex from the node and adds it as a new MCTSNode.} \\ \tabitem{\textbf{expansionForCoords()}: Performs an expansion for the given coordinates. This represents forcing a move on the algorithm.} \\ \tabitem{\textbf{simulation()}: Play random matches to accumulate reward information on the node.} \\ \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{Keras} \\ \midrule \textbf{Description} \\ Implements the DecisionAlgorithm interface to give access to a neural network. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Analyzing game states and generating moves.} \\ \midrule \textbf{Proposed attributes} \\ \tabitem{\textbf{currentMove}: A GameMove for accessing game state and logic.} \\ \tabitem{\textbf{neuralNetwork}: A NeuralNetwork instance for generating moves.} \\ \midrule \textbf{Proposed methods} \\ \decisionAlgorithmMethods \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{NeuralNetwork} \\ \midrule \textbf{Description} \\ Manages the neural networks used by the engine. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Analyzing game states and generating moves.} \\ \tabitem{Generating a new neural network.} \\ \tabitem{Loading a model file to use an existing trained neural network.} \\ \midrule \textbf{Proposed attributes} \\ \tabitem{\textbf{currentMove}: A GameMove for accessing game state and logic.} \\ \tabitem{\textbf{neuralNetwork}: A NeuralNetwork instance for generating moves.} \\ \midrule \textbf{Proposed methods} \\ \tabitem{\textbf{pickMove()}: Uses the current internal model to pick a move given a game state.} \\ \tabitem{\textbf{trainModel()}: Receives a list of games, with each game being a list of moves, and trains the network on them.} \\ \tabitem{\textbf{saveModel()}: Saves the current internal neural network model.} \\ \tabitem{\textbf{saveHeatmap()}: Saves an image of a heatmap of move likelihood.} \\ \tabitem{\textbf{saveModelPlot()}: Saves an image of a plot of the model configuration.} \\ \bottomrule \end{tabular} \vspace{\interclassSpace} \paragraph{Game System} \indent \\ \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{GameState} \\ \midrule \textbf{Description} \\ Stores the state of a match. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Store state information.} \\ \tabitem{Offer methods to manipulate the game state.} \\ \midrule \textbf{Proposed attributes} \\ \midrule \textbf{Proposed methods} \\ \tabitem{\textbf{getCurrentPlayer()}: Returns the player who should play next.} \\ \tabitem{\textbf{playMove()}: Play a move in the specified coordinates for the specified player.} \\ \tabitem{\textbf{playMoveForPlayer()}: Play a move in the specified coordinates for the player who should play next.} \\ \tabitem{\textbf{undo()}: Reset the state to how it was just before the last move was played.} \\ \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{GameBoard} \\ \midrule \textbf{Description} \\ Stores the state of a board position and handles its logic. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Store the vertices of a board position.} \\ \tabitem{Logic related to a board position.} \\ \midrule \textbf{Proposed attributes} \\ \tabitem{\textbf{board}: An array of the stones on the board.} \\ \midrule \textbf{Proposed methods} \\ \tabitem{\textbf{getGroupLiberties()}: Returns a set with the empty vertices adjacent to the group occupying a vertex.} \\ \tabitem{\textbf{getGroupLibertiesCount()}: Returns the number of liberties of the group occupying a vertex.} \\ \tabitem{\textbf{getGroupVertices()}: Returns a set with the vertices of the group occupying a vertex.} \\ \tabitem{\textbf{getGroupVerticesCount()}: Returns the number of stones of the group occupying a vertex.} \\ \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{GameMove} \\ \midrule \textbf{Description} \\ Stores information about a specific move and its relationships to the previous and next moves. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Store information about a move (board, player, coordinates\ldots).} \\ \midrule \textbf{Proposed attributes} \\ \tabitem{\textbf{board}: The board as of this move.} \\ \tabitem{\textbf{nextMoves}: The list of moves played after this one. Different moves represent different game variations.} \\ \tabitem{\textbf{previousMove}: The move before this one.} \\ \tabitem{\textbf{isPass}: True if the move is a pass and not a stone placement.} \\ \tabitem{\textbf{coords}: The coordinates of the board the move was played at. Has no meaning if \textbf{isPass} is true.} \\ \tabitem{\textbf{playerWhoPassed}: The player who made this move. Has no meaning if \textbf{isPass} is false, since the player can be obtained from the coordinates of the move when it is not a pass.} \\ \midrule \textbf{Proposed methods} \\ \tabitem{\textbf{getRow()}: Returns the row the move was played at.} \\ \tabitem{\textbf{getCol()}: Returns the col the move was played at.} \\ \tabitem{\textbf{getPlayer()}: Returns the player who played the move.} \\ \tabitem{\textbf{getNextPlayer()}: Returns the player who should play after this move.} \\ \tabitem{\textbf{getGameLength()}: Returns the number of moves the game has had.} \\ \tabitem{\textbf{getPlayableVertices()}: Returns the legal vertices for the next move.} \\ \tabitem{\textbf{addMove()}: Inserts a new children move for the given coordinates and for the player who should make the next move.} \\ \tabitem{\textbf{addMoveForPlayer()}: Inserts a new children move for the given coordinates and player.} \\ \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{GameBoard} \\ \midrule \textbf{Description} \\ Represents a board. Contains played stones and the amount of captures made by each player. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Store a specific layout of stones in the board.} \\ \midrule \textbf{Proposed attributes} \\ \tabitem{\textbf{board}: An array containing the stone layout.} \\ \tabitem{\textbf{capturesBlack}: The stones captured by black before the position.} \\ \tabitem{\textbf{capturesWhite}: The stones captured by white before the position.} \\ \midrule \textbf{Proposed methods} \\ \tabitem{\textbf{getBoardHeight()}: Returns the number of rows of the board.} \\ \tabitem{\textbf{getBoardWidth()}: Returns the number of columns of the board.} \\ \tabitem{\textbf{getGroupLiberties()}: Returns a list with the empty vertices adjacent to the group occupying a vertex.} \\ \tabitem{\textbf{getGroupVertices()}: Returns a list with the vertices occupied by the group occupying a vertex.} \\ \tabitem{\textbf{moveAndCapture()}: Makes a move and captures the corresponding stones if the move results in the capture of a group.} \\ \tabitem{\textbf{score()}: Gets the current score based on the already surrounded territory. This follows Japanese rules.} \\ \bottomrule \end{tabular} \vspace{\interclassSpace} \paragraph{Training System} \indent \\ \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{Trainer} \\ \midrule \textbf{Description} \\ Provides the neural networks with moves to train on. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Obtain moves from stored records of matches.} \\ \tabitem{Provide neural networks with moves to train on.} \\ \midrule \textbf{Proposed attributes} \\ %TODO: Explain why this is empty \midrule \textbf{Proposed methods} \\ \tabitem{\textbf{loadGameTree()}: Reads a file and generates a GameMove tree from its contents.} \\ \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{Parser} \\ \midrule \textbf{Description} \\ Reads SGF files and converts them to a tree of GameMove from the Game System. \\ \midrule \textbf{Responsibilities} \\ \tabitem{Read SGF files.} \\ \tabitem{Convert the content of the SGF files to a tree of GameMove.} \\ \midrule \textbf{Proposed attributes} \\ %TODO: Explain why this is empty \midrule \textbf{Proposed methods} \\ %TODO: Explain why this is empty \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{tabular}{p{0.9\linewidth}} \toprule \textbf{ASTNode} \\ \midrule \textbf{Description} \\ Makes up the tree resulting from the parsing of an SGF file.\\ \midrule \textbf{Responsibilities} \\ \tabitem{Obtain a GameMove tree from itself and its children.} \\ \midrule \textbf{Proposed attributes} \\ \tabitem{\textbf{children}: The nodes following from itself.} \\ \tabitem{\textbf{props}: The properties of the tree read from an SGF file.} \\ \midrule \textbf{Proposed methods} \\ \tabitem{\textbf{toGameTree()}: Returns a GameMove tree corresponding to the tree following from this node.} \\ \bottomrule \end{tabular} \vspace{\interclassSpace} \subsection{System Actors} There are various actors who will interact with the system, both human and non-human. \begin{itemize} \item The human player who interacts with the playing interface. \item The human user who interacts with the engine. \item A GUI software which uses the engine to generate moves. \end{itemize} \subsection{Use Cases} \begin{figure}[h] \begin{center} \includegraphics[width=\textwidth]{diagrams/useCases.png} \caption{Use cases.} \label{fig:useCases} \end{center} \end{figure} The different actors and use cases are represented on \fref{fig:useCases}. Each use case is explained next. \paragraph{Play a match} The game interface reads the moves presented by the player and shows their result on the board. \paragraph{Use as backend for machine player} The engine is used as the backend for generating moves for a machine player, this is, for automated play, either against a human who is using the GUI or against another machine player. \paragraph{Generate a move} The engine interface reads the input for generating a move as stated by the GTP protocol and outputs the coordinates of the board to play. \subsection{Use Case Analysis and Scenarios} \begin{figure}[h] \begin{center} \includegraphics[width=0.8\textwidth]{diagrams/useCase_playAMatch.png} \caption{Use case: Play a match.} \label{fig:useCase_playAMatch} \end{center} \end{figure} \begin{tabular}{lp{0.6\linewidth}} \toprule \multicolumn{2}{c}{\textbf{Play a match}} \\ \midrule \textbf{Preconditions} & The game interface has been started. \\ \midrule \textbf{Postconditions} & The program terminates after a match has been played. \\ \midrule \textbf{Actors} & Human player \\ \midrule \textbf{Description} & 1. The user enters the move to make.\newline 2. The result of playing that move is outputted by the program.\newline 3. Stop the program if the game has ended or go back to 1 if not. \\ \midrule \textbf{Secondary scenarios} & \textbf{The move is illegal}: An error message is shown. Go back to step 1 of main scenario. \\ \midrule \textbf{Exceptions} & \textbf{The input is wrong}: An error message is shown. Go back to step 1 of main scenario. \\ \midrule \textbf{Notes} & This scenario does not pretend to be a complete recreation of a go match. It will be playable, but its main purpose is to see the Game implementation in action.\newline A robustness diagram for this scenario is shown in \fref{fig:useCase_playAMatch}.\\ \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{figure}[h] \begin{center} \includegraphics[width=\textwidth]{diagrams/useCase_generateAMove.png} \caption{Use case: Generate a move.} \label{fig:useCase_generateAMove} \end{center} \end{figure} \begin{tabular}{lp{0.6\linewidth}} \toprule \multicolumn{2}{c}{\textbf{Generate a move}} \\ \midrule \textbf{Preconditions} & The game engine has been started. \newline Optionally, some moves have already been played. \\ \midrule \textbf{Postconditions} & A move is suggested via the engine output. \\ \midrule \textbf{Actors} & Human user and GUI program. \\ \midrule \textbf{Description} & 1. The user or program enters the player to generate the move for.\newline 2. The suggested move is outputted by the engine, either as coordinates or as an indication to pass. \\ \midrule \textbf{Secondary scenarios} & \textbf{The move is illegal}: An error message is shown. Go back to step 1 of main scenario. \\ \midrule \textbf{Exceptions} & \textbf{The input is wrong}: An error message is shown. Go back to step 1 of main scenario. \\ \midrule \textbf{Notes} & A robustness diagram for this scenario is shown in \fref{fig:useCase_generateAMove}.\\ \bottomrule \end{tabular} \vspace{\interclassSpace} \begin{figure}[h] \begin{center} \includegraphics[width=\textwidth]{diagrams/useCase_useAsBackend.png} \caption{Use case: Use as backend for machine player.} \label{fig:useCase_useAsBackend} \end{center} \end{figure} \begin{tabular}{lp{0.6\linewidth}} \toprule \multicolumn{2}{c}{\textbf{Use as backend for machine player}} \\ \midrule \textbf{Preconditions} & The game engine has been configured as engine for the software. \\ \midrule \textbf{Postconditions} & A match has been played against the engine. \\ \midrule \textbf{Actors} & GUI program. \\ \midrule \textbf{Description} & 1. The program gives commands to the engine to set up the game. The specific commands will vary from program to program.\newline 2. The program asks the engine for a move.\newline 3. The engine suggest a move to the program.\newline 4. The moves are shown by the program as if made by a player.\newline 5. The opponent gives a move to the program.\newline 6. Repeat from step 2 until the game ends. \\ \midrule \textbf{Secondary scenarios} & ---\\ \midrule \textbf{Exceptions} & ---\\ \midrule \textbf{Notes} & A robustness diagram for this scenario is shown in \fref{fig:useCase_useAsBackend}.\\ \bottomrule \end{tabular} \vspace{\interclassSpace} \subsection{Testing Plan Specification} \subsubsection{Unitary Testing} Tests for the python code are developed using the unittest\cite{python_unittest} testing framework. It has been chosen by virtue of being thoroughly documented and widely used. The coverage of unit testing is checked with Coverage.py\cite{python_coverage}, which can by itself run the unittest tests and generate coverage reports based on the results. % Maybe put an example report here?