path: root/doc/tex/implementation.tex

\section{Implementation}

This section is about the specific tools and process used to implement
\program{}.

\subsection{Development Hardware}

The development and evaluation machine is a Toshiba Satellite L50-C laptop. Its
specifications are shown as a list for readability.

\begin{itemize}
	\item \textbf{CPU}: Intel i5-6200U, 2.800GHz
	\item \textbf{Integrated GPU}: Intel Skylake GT2
	\item \textbf{Dedicated GPU}: Nvidia Geforce 930M
	\item \textbf{RAM}: 8 GiB
\end{itemize}

\subsection{Development Software}

The tools selected for the development of the project and the documentation are
listed and explained on this section. All the tools used are either free
\cite{fsf_free} or open source software. The development machine runs 64 bits
Arch Linux \cite{arch} as its operating system.

\subsubsection{Language}

The programming language of choice is Python \cite{python}. The rationale behind
this decision has been stated on Section \ref{sec:programmingLanguage}. It also
allows easy use of the Keras library for implementing neural networks.

Various Python libraries have been used to easy the development process or
assist in the analysis of results. These are:

\paragraph{Keras/TensorFlow}

TensorFlow \cite{tensorflow} is a platform for machine learning which provides a
diverse range of tools, one of which is a Python library for machine learning.

Keras \cite{keras} is a high-level API for TensorFlow allowing for the easy
definition of neural networks. It permits easily testing and comparing different
network layouts.

\paragraph{NumPy}

A scientific package for Python providing a lot of mathematical tools
\cite{numpy}. The most interesting for this project are its capabilities to
create and transform matrices.

\paragraph{Matplotlib}

A Python library for creating graphs and other visualizations \cite{matplotlib}.
It is used to show the likelihood of moves the neural networks of the project
create from a board configuration.

\paragraph{PLY}

A tool for generating compilers in Python \cite{ply}. It is an implementation of
the lex and yacc utilities, allowing to create lexers and parsers. It is used in
the project to create the \acrshort{sgf} parser which transform \acrshort{sgf}
files to internal representations of Go matches.

\paragraph{Other utility libraries}

These are some utility libraries commonly used for frequent programming tasks.

\begin{itemize}
	\item \textbf{sys}: To stop the execution of the program or access system info such
		as primitives maximum values.
	\item \textbf{os}: To interact with files.
	\item \textbf{re}: To check strings with regular expressions.
	\item \textbf{random}: For randomness, for example to obtain a random item
		from a list.
	\item \textbf{copy}: To obtain deep copies of multidimensional arrays.
\end{itemize}

\subsubsection{Development Tools}

\paragraph{Neovim}

A text editor based on Vim \cite{vim}, providing its same functionality with
useful additions and defaults for modern computers and terminal emulators
\cite{neovim}. With some extensions and configuration it becomes a powerful
development environment with a very fluid usage experience. That, and the
preference of the developer of Vim's modal editing as the best writing
experience possible on a computer, have made Neovim the editor of choice.

%TODO: Write about neovim extensions
%\begin{itemize}
%	\item FZF
%	\item Extensions
%
%\end{itemize}

\paragraph{Git}

A version control tool widely used in software development environments
\cite{git}. If the reader is not already familiar with it, it suffices to say it
allows to store and manage snapshots of a project, navigate through them and
diverge into different development branches, among other useful features.

The source code of this document and of the rest of the project is publicly
available at \url{https://git.taamas.xyz/Taamas/imago}.

\subsubsection{Documentation Tools}

\paragraph{\LaTeX}

A typesetting system widely used in the investigation field, among others
\cite{latex}. It allows for documentation like this text to be written in plain
text and then compiled to PDF or other formats, which permits keeping the source
files of the documentation small and organized plus other benefits of plain text
such as being able to be used in conjunction with version control tools.

\paragraph{PlantUML}

A program which creates diagrams from plain text files \cite{puml}. PlantUML
supports syntax for many different sorts of diagrams, mainly but not only UML.
It has been used to generate the diagrams used in this text.

\paragraph{Make}

A tool for specifying and handling dependencies on a build system. It reads a
file, typically named ``Makefile'', containing which files are needed and on
which other files each of them depends, and then generates those files or
updates them if they already exist but their source files are newer than them.

It has been used to generate this text from \LaTeX{} and PlantUML source files.
The contents of the Makefile with which this document has been compiled are
shown in \lref{code:makefile}.

\begin{listing}[p]
	\inputminted{make}{Makefile}
	\caption{Documentation Makefile}\label{code:makefile}
\end{listing}

\subsection{Execution of the Testing Plan}

Part of the implementation process is to execute the testing plan. The results
of this execution are provided in this section.

\subsubsection{Execution of the Unitary Testing}

The script used to run the tests is shown on \lref{lst:test} and its output on
\lref{lst:testOutput}.

\begin{listing}[p]
	\inputminted{bash}{listings/test.sh}
	\caption{Script to run the tests and list the result.}
	\label{lst:test}
\end{listing}

\begin{listing}[p]
	\inputminted[fontsize=\footnotesize]{text}{listings/testOutput.txt}
	\caption{Unitary testing output.}
	\label{lst:testOutput}
\end{listing}

\subsubsection{Execution of the Integration Testing}

\vspace{\interclassSpace}

\begin{tabular}{p{0.2\linewidth}p{0.3\linewidth}p{0.4\linewidth}}
	\toprule
	\multicolumn{3}{c}{\textbf{Engine and Game modules}} \\
	\midrule
	\textbf{Test} & \textbf{Expected behaviour} & \textbf{Results} \\
	\midrule
	The GTP interface of the engine is used to play a match &
	The module handles the game and can show its state. &
	The engine runs correctly and is capable of keeping track of and showing the
	state of a match, which shows it is making good use of the Game module. \\
	\bottomrule
\end{tabular}

\vspace{\interclassSpace}

\begin{tabular}{p{0.2\linewidth}p{0.3\linewidth}p{0.4\linewidth}}
	\toprule
	\multicolumn{3}{c}{\textbf{Training and Engine module}} \\
	\midrule
	\textbf{Test} & \textbf{Expected behaviour} & \textbf{Results} \\
	\midrule
	The training process is started &
	The training uses the network defined on the Engine module. &
	The training output shows the network in training follows the design defined
	on the Engine module. \\
	\bottomrule
\end{tabular}

\subsubsection{Execution of the System Testing}

\vspace{\interclassSpace}

\begin{tabular}{p{0.2\linewidth}p{0.3\linewidth}p{0.4\linewidth}}
	\toprule
	\multicolumn{3}{c}{\textbf{Game interface}} \\
	\midrule
	\textbf{Test} & \textbf{Expected behaviour} & \textbf{Results} \\
	\midrule
	Play a game of Go with no engine &
	The game can be played until the end. &
	The interface continues asking for moves until both players pass
	consecutively, which ends the game and the execution. \\
	\midrule
	Provide a wrong move &
	The interface shows it is wrong and the game continues without a change of
	state. &
	As expected, the interface shows a message signaling that the move is wrong
	and showing an example of a move. \\
	\midrule
	Close the game &
	The interface closes. &
	A message is shown signaling that the game is ending and the engine closes.
	\\
	\bottomrule
\end{tabular}

\vspace{\interclassSpace}

\begin{tabular}{p{0.2\linewidth}p{0.3\linewidth}p{0.4\linewidth}}
	\toprule
	\multicolumn{3}{c}{\textbf{Engine interface}} \\
	\midrule
	\textbf{Test} & \textbf{Expected behaviour} & \textbf{Results} \\
	\midrule
	Ask for the available commands &
	The interface outputs the available commands. &
	The list of available commands is printed one per line. \\
	\midrule
	Provide a move &
	The state of the engine updates with the new move. &
	No output is given, but after providing the \texttt{showboard} command it is
	shown that the move has been registered. \\
	\midrule
	Ask for a move &
	The engine suggests a move without changing the state of the current game. &
	After the necessary time for generating the move, it is printed. The
	\texttt{showboard} is then used to check the state of the game has not
	changed. \\
	\bottomrule
\end{tabular}

\vspace{\interclassSpace}

\begin{tabular}{p{0.2\linewidth}p{0.3\linewidth}p{0.4\linewidth}}
	\toprule
	\multicolumn{3}{c}{\textbf{Training interface}} \\
	\midrule
	\textbf{Test} & \textbf{Expected behaviour} & \textbf{Results} \\
	\midrule
	Provide some games to train on &
	A neural network model is created. &
	First, all the training files used are printed, and then, as expected, the
	training process commences. A model is created or updated in the
	\texttt{models} folder. \\
	\midrule
	Start the training without providing games &
	An error message is shown and the execution terminated. &
	As expected, a message is shown asking for SGF files to be provided as
	arguments and the execution is terminated.\\
	\bottomrule
\end{tabular}

\subsubsection{Usability Testing}

Two human users were asked to interact with the interfaces of \program{} and
presented with a questionary.

The profile of the first user is of someone who has played some Go matches and
knows the fundamentals of the game but is a beginner, and who has little
experience with computers outside of their usage as office tools and internet
browsers. Here are their answers.

\vspace{\interclassSpace}

\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}}
	\toprule
	\multicolumn{2}{c}{\textbf{Playing against a human}} \\
	\midrule
	\textbf{Question} & \textbf{Answer} \\
	\midrule
	Were you able to start the interface? &
	Yes. \\
	\midrule
	How hard was the interface of the game to understand? &
	It was easy and intuitive because you just entered the command to start and
	then you only had to tell it where you wanted to play the stones. It felt
	easy to me.\\
	\bottomrule
\end{tabular}

\vspace{\interclassSpace}

\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}}
	\toprule
	\multicolumn{2}{c}{\textbf{Playing against the engine}} \\
	\midrule
	\textbf{Question} & \textbf{Answer} \\
	\midrule
	Were you able to start the interface? &
	Yes. \\
	\midrule
	How hard was the interface of the game to understand? &
	It was easy to understand because it just was enering the commands and the
	application did what it had to do, but it was more difficult than the
	previous one because I'm not used to do these things with commands. It is
	less intuitive than the previous one regarding playing against the machine
	because you don't see each move as you write it, having to ask it to show
	them instead.\\
	\midrule
	How strong did you find the engine? &
	It was not so aggresive as playing against a human who knows the game. It
	doesn't play to harm its opponent.\\
	\bottomrule
\end{tabular}

\vspace{\interclassSpace}

\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}}
	\toprule
	\multicolumn{2}{c}{\textbf{Playing against the interface through a
	third-party}} \\
	\midrule
	\textbf{Question} & \textbf{Answer} \\
	\midrule
	Were you able to start the interface? &
	Yes. \\
	\midrule
	Did you find any problems when setting up the engine? &
	No, it was well explained step by step, although the images could be better
	lined up with the text. Anyway, the explanations were clear and easy to
	follow.\\
	\midrule
	Do you think this tool has value for studying Go? &
	Yes.\\
	\bottomrule
\end{tabular}

The profile of the second user is of someone who has experience with computers
and works as a software developer, but who has just the bare minimum knowledge
of the game of Go. Here are their answers.

\vspace{\interclassSpace}

\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}}
	\toprule
	\multicolumn{2}{c}{\textbf{Playing against a human}} \\
	\midrule
	\textbf{Question} & \textbf{Answer} \\
	\midrule
	Were you able to start the interface? &
	Yes, I was able to. \\
	\midrule
	How hard was the interface of the game to understand? &
	I think six out of ten. Some people won't understand what a command is and
	without a visual interface they could feel confused about it.\\
	\bottomrule
\end{tabular}

\vspace{\interclassSpace}

\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}}
	\toprule
	\multicolumn{2}{c}{\textbf{Playing against the engine}} \\
	\midrule
	\textbf{Question} & \textbf{Answer} \\
	\midrule
	Were you able to start the interface? &
	Yes, I was. \\
	\midrule
	How hard was the interface of the game to understand? &
	I think seven out of ten. I was expecting to follow some steps so I could
	execute it at the same time I was reading the manual, but in the end the
	most practical thing was read everything before execute commands.\\
	\midrule
	How strong did you find the engine? &
	It followed good paths to win the game, I didn't feel random moves during
	the game by its side.\\
	\bottomrule
\end{tabular}

\vspace{\interclassSpace}

\begin{tabular}{p{0.4\linewidth}p{0.6\linewidth}}
	\toprule
	\multicolumn{2}{c}{\textbf{Playing against the interface through a
	third-party}} \\
	\midrule
	\textbf{Question} & \textbf{Answer} \\
	\midrule
	Were you able to start the interface? &
	Yes, I was. \\
	\midrule
	Did you find any problems when setting up the engine? &
	No, I didn't.\\
	\midrule
	Do you think this tool has value for studying Go? &
	I think it is a good tool for a group of players, so they can practise and
	even train the AI if they want to.\\
	\bottomrule
\end{tabular}

The results of these usability tests were useful mostly to update the manual and
make it easier to understand and follow and also to address some problems with
the interfaces that raised up during the testing.