2.4 Nature of Software Development

Architecture is defined as "the process and product of planning, designing and constructing buildings." Buildings here can be substituted for software without losing much in intend.

There are only two hard things in Computer Science: Cache Invalidation and Naming Things. - Phil Karlton

Software systems are complex because of the fact that there are a lot of entities and myriads of relationships among them. Inorder to have a control in creating systems as they grow complex, it was essential that recurrent patterns were spotted and given a name to, so that it helped to manipulate and reason about these concepts. As the size of the systems in the current project started growing one of the biggest challenges came in the form of having to name things. It was one of the increasingly difficult concepts to minimize the name clashes as and keep them separated apart so that there was no confusion.

Fred Brooke's in his book argues that coherent systems are the one's where the whole system can be kept in ones' mind as noted by Fred Brooks. As he notes, it became increasingly difficult when it came to the naming aspect to keep all of this in my mind.

Inorder to construct an architecture, a language haS to be developed to talk about it. This made the architecture tangible and provides an unambiguous description of architectural building blocks as well as concrete system while abstracted away from implementation details.

The architecture required definition of the building blocks (types of things) from which the actual system will be built. It can be thought of as a bespoke language constructed in order to talk about the system. Though I couldn't take it to a fuller refinement, I have provided systems diagrams [provide crosslink] to explain the construction of the software created for the ecosystem.

Controlling complexity is the essence of computer programming. — Brian Kernighan

Software development is an enterprise that is deeply enmeshed with controlling complexity has been one of the pivotal concerns in this project. As Djikstra points out "computing scientist’s main challenge is not to get confused by the complexities of his own making", this rang quite true throughout this project.

Almost all of this project was completed in a programming language known as Clojure. It's a Lisp that runs on the JVM. As of the writing it has emerged as the most popular lisp ever written. But popularity can't be held as an indicator of quality. What I feel is a good indicator is the ease with which a tool enables creation and in this regard Clojure proved to be a solid tool to build software systems. I am convinced that the nature of minimal syntax of Lisp and composability of mathematical functions are great concepts to manage the emergent complexity when developing software. The great tooling around the core language and an active and smart community around the language have been pivotal towards this achievement. The lanugage being functional in nature also helped me learn a lot about programming in general since my education at the university was heavily focused towards object oriented and imperative languages. Simply put, it has been transformative in more ways than one. Simple core abstractions, immutability, referential transparency and a whole lot of other great concepts that Clojure entails have greatly shaped the way I program and I hope will continue to influence my future. Implications of developing in a functional paradigm is mentioned in the methods section.

A note that that I have to add about Clojure is that it simply doesn't make the software development process easy for you. It is simply not so for an incoming newbie. You have to be well versed on a large pool of concepts to help it to see the programming domain in the way it sees but once you are beyond that plateau, leveraging Clojure leads to a simplified view of creating processes and procedures.

Describe the steps taken in order to making the systems legible.

There is a concept known as dogfooding which means to test the software by yourself. Since I am one among the target audience for which I'm constructing this system for, it was a natural fit. But instead of focusing on how these helped inform my academic decisions which I'm relegating to the evaluation section I will briefly mention how I got a good taste of this when I used my docs-gen [NVDocsGen 2015] [link] to generate the documentation for describing the apps in the ecosystem. It was a good experience to create software that describe the docs generator. This was a good benchmark to test the soundness of the concept as you continuously used the software yourself while also presenting an alternative view, I made sure that I shared my creations with my colleagues to see the sticking points or potentially confusing interactions that might arise.

Systems had to be constructed with the idea that specifications would always change. And as the saying goes change being the only thing that is constant, it was high priority to build this ecosystem in a manner that it accommodated change. This has been achieved by keeping the systems modular.

The nature of lisp lends itself to create components which can be weaved together to form a whole app. I could only fare as far as creating structures of web components as outlined in the development section, but some more work is to be done in order to make it truly componentized. This means that apps can be composed as a series of higher order functions.

One of the central precepts of this dissertation is that of modularization and all the apps can be considered as an module that consists of further more modules inside them. Due to my limited time and experience I had with lisp this has not been realized to a fuller extend in this project but to give a brief account of how this can be enabled, here's some very minimal Clojure code that uses the compose function in Clojure.

; Comp(ose)ing the components together

(def app ((comp) header main-area footer))

; Passing in the data (app data)

This would help enable the componentization and hence reuse of different modules developed for each app among the different apps

Better abstractions enable structuring your programs better. Throughout the process I was able to see how a thorough abstraction of a certain model of process enabled me to be more flexible at designing my functions. In order to give a simple illustration:

The code

(def )

whereas if I employ the following code:

(def )

would result in much more flexibility in terms of the actual product. This was proven time and again throughout the time I did my dissertation.

One way is to start from scratch and supply as much care and control over everything by building them from the ground up but the scope of this dissertation as it had to be produced in a limited time didn't allow for such levels of explorations. I have to mention that the language adopted for building this ecosystem played a key part in keeping the complexity to a manageable level. Also, this nature of software being always in flux is what lead me to create the current architecture of centralizing it with the pervasive model of Internet technology - the client server model.

It was essential to follow certain models and probably I have tried to explore different sets of architectural styles for each apps. For example the ExamVault website follows a typical client server model with most of the visual rendering predetermined on the server side whereas in NCLProgress the majority is done on the client side, similarly for IvoryTower which forms the backbone of apps by providing them data uses a RESTful architecture in delivering the information. I will be detailing about these styles later in fuller detail later in the thesis but it is to be noted that modularizing the apps and creating a domain for a coherent set of functionality enables the exploration of creating bespoke architectures that are well fitted to each domain. This could be a major win from a polyglot perspective as I have mentioned this enables for integrating newer systems into the ecosystem with ease.

Lack of my knowledge with respect to software architectures in general and inability to visualize the future of these apps have rendered to this to be sub-optimal but the good thing about software is that everything can be rebuilt from scratch once you have explored the domain space thoroughly.

A few structures that I have in mind to architecture the ecosystem was Supervisor-Workers model from Erlang OTP documentation. Attention had to be paid to the behaviour of this software. I have decided that if further work is to be carried out if I wants to create a completely resilient system. The implications of adopting this model is further explained in the [future] section.

Fred Brookes' tar pit. Silos and Snowclones.