diff options
Diffstat (limited to 'doc/tex/implementation.tex')
| -rw-r--r-- | doc/tex/implementation.tex | 386 |
1 files changed, 335 insertions, 51 deletions
diff --git a/doc/tex/implementation.tex b/doc/tex/implementation.tex index d46e805..d3cd0c3 100644 --- a/doc/tex/implementation.tex +++ b/doc/tex/implementation.tex @@ -1,8 +1,11 @@ \section{Implementation} +This section is about the specific tools and process used to implement +\program{}. + \subsection{Development Hardware} -The development and evaluation machine is a Satellite L50-C laptop. Its +The development and evaluation machine is a Toshiba Satellite L50-C laptop. Its specifications are shown as a list for readability. \begin{itemize} @@ -15,71 +18,71 @@ specifications are shown as a list for readability. \subsection{Development Software} The tools selected for the development of the project and the documentation are -listed and explained on this section. All the tools used are either -free\cite{fsf_free} or open source software. The development machine runs 64 -bits Arch Linux as its operating system. +listed and explained on this section. All the tools used are either free +\cite{fsf_free} or open source software. The development machine runs 64 bits +Arch Linux \cite{arch} as its operating system. \subsubsection{Language} -The programming language of choice is Python\cite{python}. The rationale behind +The programming language of choice is Python \cite{python}. The rationale behind this decision has been stated on Section \ref{sec:programmingLanguage}. It also allows easy use of the Keras library for implementing neural networks. -Various python libraries have been used to easy the development process or +Various Python libraries have been used to easy the development process or assist in the analysis of results. These are: -\paragraph{Keras/Tensorflow\cite{keras}} +\paragraph{Keras/TensorFlow} -Tensorflow is a platform for machine learning which provides a diverse range of -tools, one of which is a python library for machine learning. +TensorFlow \cite{tensorflow} is a platform for machine learning which provides a +diverse range of tools, one of which is a Python library for machine learning. -Keras is a high-level API for Tensorflow allowing for the easy definition of -neural networks. It permits easily testing and comparing different network -layouts. +Keras \cite{keras} is a high-level API for TensorFlow allowing for the easy +definition of neural networks. It permits easily testing and comparing different +network layouts. -\paragraph{NumPy\cite{numpy}} +\paragraph{NumPy} -A scientific package for python providing a lot of mathematical tools. The most -interesting for this project are its capabilities to create and transform -matrices. +A scientific package for Python providing a lot of mathematical tools +\cite{numpy}. The most interesting for this project are its capabilities to +create and transform matrices. -\paragraph{Matplotlib\cite{matplotlib}} +\paragraph{Matplotlib} -A python library for creating graphs and other visualizations. It is used to -show the likelihood of moves the neural networks of the project create from a -board configuration. +A Python library for creating graphs and other visualizations \cite{matplotlib}. +It is used to show the likelihood of moves the neural networks of the project +create from a board configuration. -\paragraph{PLY\cite{ply}} +\paragraph{PLY} -A tool for generating compilers in Python. It is an implementation of the lex -and yacc utilities, allowing to create lexers and parsers. It is used in the -project to create the SGF parser which transform SGF files to internal -representations of Go matches. +A tool for generating compilers in Python \cite{ply}. It is an implementation of +the lex and yacc utilities, allowing to create lexers and parsers. It is used in +the project to create the \acrshort{sgf} parser which transform \acrshort{sgf} +files to internal representations of Go matches. \paragraph{Other utility libraries} -These are some utility libraries commonly used for frequent programming tasks: +These are some utility libraries commonly used for frequent programming tasks. \begin{itemize} - \item \textbf{sys}: to stop the execution of the program or access system info such + \item \textbf{sys}: To stop the execution of the program or access system info such as primitives maximum values. - \item \textbf{os}: to interact with files. - \item \textbf{re}: to check strings with regular expressions. - \item \textbf{random}: to get random values, for example to obtain a random - item from a list. - \item \textbf{copy}: to obtain deep copies of multidimensional arrays. + \item \textbf{os}: To interact with files. + \item \textbf{re}: To check strings with regular expressions. + \item \textbf{random}: For randomness, for example to obtain a random item + from a list. + \item \textbf{copy}: To obtain deep copies of multidimensional arrays. \end{itemize} \subsubsection{Development Tools} -\paragraph{Neovim\cite{neovim}} +\paragraph{Neovim} -A text editor based on Vim\cite{vim}, providing its same functionality with -useful additions and defaults for modern computers and terminal emulators. With -some extensions and configuration it can become a powerful development -environment with a very fluid usage experience. That, and the fact that the -developer considers Vim's modal editing the best writing experience possible on -a computer, have made Neovim the editor of choice. +A text editor based on Vim \cite{vim}, providing its same functionality with +useful additions and defaults for modern computers and terminal emulators +\cite{neovim}. With some extensions and configuration it becomes a powerful +development environment with a very fluid usage experience. That, and the +preference of the developer of Vim's modal editing as the best writing +experience possible on a computer, have made Neovim the editor of choice. %TODO: Write about neovim extensions %\begin{itemize} @@ -88,21 +91,31 @@ a computer, have made Neovim the editor of choice. % %\end{itemize} +\paragraph{Git} + +A version control tool widely used in software development environments +\cite{git}. If the reader is not already familiar with it, it suffices to say it +allows to store and manage snapshots of a project, navigate through them and +diverge into different development branches, among other useful features. + +The source code of this document and of the rest of the project is publicly +available at \url{https://git.taamas.xyz/Taamas/imago}. + \subsubsection{Documentation Tools} -\paragraph{\LaTeX\cite{latex}} +\paragraph{\LaTeX} -A typesetting system widely used in the investigation field, among others. It -allows for documentation like this text to be written in plain text and then -compiled to PDF or other formats, which permits keeping the source files of the -documentation small and organized plus other benefits of plain text such as -being able to use version control. +A typesetting system widely used in the investigation field, among others +\cite{latex}. It allows for documentation like this text to be written in plain +text and then compiled to PDF or other formats, which permits keeping the source +files of the documentation small and organized plus other benefits of plain text +such as being able to be used in conjunction with version control tools. -\paragraph{PlantUML\cite{puml}} +\paragraph{PlantUML} -A program which creates diagrams from plain text files. PlantUML supports syntax -for many different sorts of diagrams, mainly but not only UML. It has been used -to generate the diagrams used in this text. +A program which creates diagrams from plain text files \cite{puml}. PlantUML +supports syntax for many different sorts of diagrams, mainly but not only UML. +It has been used to generate the diagrams used in this text. \paragraph{Make} @@ -113,9 +126,280 @@ updates them if they already exist but their source files are newer than them. It has been used to generate this text from \LaTeX{} and PlantUML source files. The contents of the Makefile with which this document has been compiled are -shown in \flist{code:makefile}. +shown in \lref{code:makefile}. -\begin{listing}[h] +\begin{listing}[p] \inputminted{make}{Makefile} \caption{Documentation Makefile}\label{code:makefile} \end{listing} + +\subsection{Execution of the Testing Plan} + +Part of the implementation process is to execute the testing plan. The results +of this execution are provided in this section. + +\subsubsection{Execution of the Unitary Testing} + +The script used to run the tests is shown on \lref{lst:test} and its output on +\lref{lst:testOutput}. + +\begin{listing}[p] + \inputminted{bash}{listings/test.sh} + \caption{Script to run the tests and list the result.} + \label{lst:test} +\end{listing} + +\begin{listing}[p] + \inputminted[fontsize=\footnotesize]{text}{listings/testOutput.txt} + \caption{Unitary testing output.} + \label{lst:testOutput} +\end{listing} + +\subsubsection{Execution of the Integration Testing} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.2\linewidth}p{0.3\linewidth}p{0.4\linewidth}} + \toprule + \multicolumn{3}{c}{\textbf{Engine and Game modules}} \\ + \midrule + \textbf{Test} & \textbf{Expected behaviour} & \textbf{Results} \\ + \midrule + The GTP interface of the engine is used to play a match & + The module handles the game and can show its state. & + The engine runs correctly and is capable of keeping track of and showing the + state of a match, which shows it is making good use of the Game module. \\ + \bottomrule +\end{tabular} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.2\linewidth}p{0.3\linewidth}p{0.4\linewidth}} + \toprule + \multicolumn{3}{c}{\textbf{Training and Engine module}} \\ + \midrule + \textbf{Test} & \textbf{Expected behaviour} & \textbf{Results} \\ + \midrule + The training process is started & + The training uses the network defined on the Engine module. & + The training output shows the network in training follows the design defined + on the Engine module. \\ + \bottomrule +\end{tabular} + +\subsubsection{Execution of the System Testing} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.2\linewidth}p{0.3\linewidth}p{0.4\linewidth}} + \toprule + \multicolumn{3}{c}{\textbf{Game interface}} \\ + \midrule + \textbf{Test} & \textbf{Expected behaviour} & \textbf{Results} \\ + \midrule + Play a game of Go with no engine & + The game can be played until the end. & + The interface continues asking for moves until both players pass + consecutively, which ends the game and the execution. \\ + \midrule + Provide a wrong move & + The interface shows it is wrong and the game continues without a change of + state. & + As expected, the interface shows a message signaling that the move is wrong + and showing an example of a move. \\ + \midrule + Close the game & + The interface closes. & + A message is shown signaling that the game is ending and the engine closes. + \\ + \bottomrule +\end{tabular} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.2\linewidth}p{0.3\linewidth}p{0.4\linewidth}} + \toprule + \multicolumn{3}{c}{\textbf{Engine interface}} \\ + \midrule + \textbf{Test} & \textbf{Expected behaviour} & \textbf{Results} \\ + \midrule + Ask for the available commands & + The interface outputs the available commands. & + The list of available commands is printed one per line. \\ + \midrule + Provide a move & + The state of the engine updates with the new move. & + No output is given, but after providing the \texttt{showboard} command it is + shown that the move has been registered. \\ + \midrule + Ask for a move & + The engine suggests a move without changing the state of the current game. & + After the necessary time for generating the move, it is printed. The + \texttt{showboard} is then used to check the state of the game has not + changed. \\ + \bottomrule +\end{tabular} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.2\linewidth}p{0.3\linewidth}p{0.4\linewidth}} + \toprule + \multicolumn{3}{c}{\textbf{Training interface}} \\ + \midrule + \textbf{Test} & \textbf{Expected behaviour} & \textbf{Results} \\ + \midrule + Provide some games to train on & + A neural network model is created. & + First, all the training files used are printed, and then, as expected, the + training process commences. A model is created or updated in the + \texttt{models} folder. \\ + \midrule + Start the training without providing games & + An error message is shown and the execution terminated. & + As expected, a message is shown asking for SGF files to be provided as + arguments and the execution is terminated.\\ + \bottomrule +\end{tabular} + +\subsubsection{Usability Testing} + +Two human users were asked to interact with the interfaces of \program{} and +presented with a questionary. + +The profile of the first user is of someone who has played some Go matches and +knows the fundamentals of the game but is a beginner, and who has little +experience with computers outside of their usage as office tools and internet +browsers. Here are their answers. + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}} + \toprule + \multicolumn{2}{c}{\textbf{Playing against a human}} \\ + \midrule + \textbf{Question} & \textbf{Answer} \\ + \midrule + Were you able to start the interface? & + Yes. \\ + \midrule + How hard was the interface of the game to understand? & + It was easy and intuitive because you just entered the command to start and + then you only had to tell it where you wanted to play the stones. It felt + easy to me.\\ + \bottomrule +\end{tabular} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}} + \toprule + \multicolumn{2}{c}{\textbf{Playing against the engine}} \\ + \midrule + \textbf{Question} & \textbf{Answer} \\ + \midrule + Were you able to start the interface? & + Yes. \\ + \midrule + How hard was the interface of the game to understand? & + It was easy to understand because it just was enering the commands and the + application did what it had to do, but it was more difficult than the + previous one because I'm not used to do these things with commands. It is + less intuitive than the previous one regarding playing against the machine + because you don't see each move as you write it, having to ask it to show + them instead.\\ + \midrule + How strong did you find the engine? & + It was not so aggresive as playing against a human who knows the game. It + doesn't play to harm its opponent.\\ + \bottomrule +\end{tabular} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}} + \toprule + \multicolumn{2}{c}{\textbf{Playing against the interface through a + third-party}} \\ + \midrule + \textbf{Question} & \textbf{Answer} \\ + \midrule + Were you able to start the interface? & + Yes. \\ + \midrule + Did you find any problems when setting up the engine? & + No, it was well explained step by step, although the images could be better + lined up with the text. Anyway, the explanations were clear and easy to + follow.\\ + \midrule + Do you think this tool has value for studying Go? & + Yes.\\ + \bottomrule +\end{tabular} + +The profile of the second user is of someone who has experience with computers +and works as a software developer, but who has just the bare minimum knowledge +of the game of Go. Here are their answers. + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}} + \toprule + \multicolumn{2}{c}{\textbf{Playing against a human}} \\ + \midrule + \textbf{Question} & \textbf{Answer} \\ + \midrule + Were you able to start the interface? & + Yes, I was able to. \\ + \midrule + How hard was the interface of the game to understand? & + I think six out of ten. Some people won't understand what a command is and + without a visual interface they could feel confused about it.\\ + \bottomrule +\end{tabular} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}} + \toprule + \multicolumn{2}{c}{\textbf{Playing against the engine}} \\ + \midrule + \textbf{Question} & \textbf{Answer} \\ + \midrule + Were you able to start the interface? & + Yes, I was. \\ + \midrule + How hard was the interface of the game to understand? & + I think seven out of ten. I was expecting to follow some steps so I could + execute it at the same time I was reading the manual, but in the end the + most practical thing was read everything before execute commands.\\ + \midrule + How strong did you find the engine? & + It followed good paths to win the game, I didn't feel random moves during + the game by its side.\\ + \bottomrule +\end{tabular} + +\vspace{\interclassSpace} + +\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}} + \toprule + \multicolumn{2}{c}{\textbf{Playing against the interface through a + third-party}} \\ + \midrule + \textbf{Question} & \textbf{Answer} \\ + \midrule + Were you able to start the interface? & + Yes, I was. \\ + \midrule + Did you find any problems when setting up the engine? & + No, I didn't.\\ + \midrule + Do you think this tool has value for studying Go? & + I think it is a good tool for a group of players, so they can practise and + even train the AI if they want to.\\ + \bottomrule +\end{tabular} + +The results of these usability tests were useful mostly to update the manual and +make it easier to understand and follow and also to address some problems with +the interfaces that raised up during the testing. |