Couple things come to mind when thinking about metrics and comparison of algorithms. As we all know, the efficiency of an algorithm can be deduced from the resource usage, thus comparing the scripts quantitatively as oppose to qualitatively, would probably be an easier task. And that’s natural only because quantitatively we can look at numerous data points that can provide insights into quantities of computation resources consumed during execution of an algorithm. In this article, I’ll concentrate mainly on the Quantitative Metrics

When comparing two algorithms by using quantitative metrics to find which algorithms is more efficient, it often depends on which measure of efficiency is considered the most significant. There is a number of quantitative metrics that we can measure. Following two are some of the most used when benchmarking the efficiency:

Algorithm’s execution time & CPU usage - The measurement of algorithm’s execution time is very often one of the most important quantitative metrics when comparing two algorithms. This allows us to compare the algorithms directly and immediately observe which of the compared scripts executes faster. The CPU resources taken during execution of an algorithm are very easy to observe, and they serve as one of the most reliable indicators of the algorithm’s efficiency. The CPU usage comparison would typically be measured as the amount of time for which a central processing unit (CPU) was used for processing instructions of a computer program. “The CPU time is measured in clock ticks or seconds. Often, it is useful to measure CPU time as a percentage of the CPU's capacity, which is called the CPU usage.” (CPU time, 2016). High execution speed is one of the most desirable features of the program.

Memory footprint – Memory footprint refers to the amount of main memory that a program uses or references while running. “In computing, the memory footprint of an executable program indicates its runtime memory requirements, while the program executes. This includes all sorts of active memory regions like code segment containing (mostly) program instructions (and occasionally constants), data segment (both initialized and uninitialized), heap memory, call stack, plus memory required to hold any additional data structures, such as symbol tables, debugging data structures, open files, shared libraries mapped to the current process, etc., that the program ever needs while executing and will be loaded at least once during the entire run.” (Memory footprint, 2015). As memory is one of the most limited resources in traditional computing systems, a script that uses minimum RAM resources would under normal circumstances be considered as more efficient regarding resource usage.

For computers that are battery powered or for lengthy calculations in mainframes, other measures of interest are: “Direct power consumption: power needed directly to operate the computer; and Indirect power consumption: power needed for cooling, lighting, etc.” (Algorithmic efficiency, 2016)

Some of the less conventional measures, but nevertheless important are:

Transmission size: This reflects on the ability of an algorithm to serve the data in the reduced amount. Typically, this refers to usage of bandwidth. A good example is the usage of text instead of an image when algorithm needs to work over the internet.

External space: This metric doesn’t refer to RAM usage, but instead to space used on an external storage device which is utilized by the algorithm. In this case, less storage space indicates a better-optimized program.

Response time: In real time application it’s one of the most critical measurements, especially if algorithm needs to provide responses to external events immediately.

The total cost of ownership:  When all the direct and indirect costs are summed up, the algorithm’s cost can be quantified as a full cost of ownership. This is one of the most important metrics for business to consider. This measure basically quantifies the cost of an algorithm in terms of various other resources needed for it to perform, such as algorithm’s hardware and software requirements, network-server-workstation hardware and software, Installation and integration of hardware and software, warranties and licenses, migration expenses, susceptibility to vulnerabilities, availability of upgrades, patches and future licensing policies, etc.

References

Algorithmic efficiency (2016) in Wikipedia. Available at: https://en.wikipedia.org/wiki/Algorithmic_efficiency (Accessed: 21 September 2016).

CPU time (2016) in Wikipedia. Available at: https://en.wikipedia.org/wiki/CPU_time (Accessed: 21 September 2016).

Memory footprint (2015) in Wikipedia. Available at: https://en.wikipedia.org/wiki/Memory_footprint (Accessed: 21 September 2016).

Total cost of ownership (2016) in Wikipedia. Available at: https://en.wikipedia.org/wiki/Total_cost_of_ownership (Accessed: 21 September 2016).