--Randy Hyde, Jan 2007.

Motivation

• To improve OS independence (to make the library easier to port)
• To use a more consistent naming scheme for library routines
• To make it easier for users of other assembly languages to call standard library routines
• To improve the design of certain routines, even if the semantics and syntax change for those routines
• To use the standard library source code as an educational resource
• To improve the performance of commonly-called functions
• To provide a formal test suite to improve the quality of the library
• To automate the build process for the library
• To make it possible for anyone to work on the library code

Of course, one of the primary reasons for working on a new version of the HLA stdlib is because it is part of the HLA v2.0 project.

OS independence

The HLA Standard Library v1.x was originally written for Windows and hacked up to support Linux. There was never any plan to port HLA v1.x to any operating system other than Windows and the Standard Library's organization reflected this naive design philosophy. While the code was easily enough hacked to work with Linux, when the port to FreeBSD began, it became clear that the library source code was not well-organized to make it easy to port the code to different operating systems.

The new library has been organized so that the OS-dependent functions are all collected together into a set of special directories. This makes it easier to find all the OS-dependent code (and ignore all the OS-independent code) when porting the library to a new operating system. Also, many of the previously OS-dependent functions were rewritten in an OS-independent fashion to reduce the number of routines that need to be modified when porting to a new operating system.

Consistent Function Naming

Some functions (in particular, hexadecimal conversion, input, and output functions) use names that are inconsistent with other library functions. For example, functions that output integers, unsigned integers, and real values use names like stdout.puti32, stdout.putu32, and stdout.putr32. The hexadecimal output functions use names like stdout.putb, stdout.putw, and stdout.putd. In v3.x and later of the standard library the goal is to use a more consistent naming scheme. For example, the hexadecimal output routines are now called names like stdout.puth8, stdout,puth16, and stdout.puth32.

Improved Design

After using the standard library for several years, it has become obvious that certain routines could have benefitted from a better interface or slightly different semantics. For example, many routines now pass parameters in registers that previously passed those parameters on the stack. If a routine expects a parameter, then the high-level syntax for the function's call allows you to specify the parameter (particularly if that parameter is passed in a register). Another big improvement in the design of the library is that many "holes" have been filled in in the functional matrix. For example, in HLA stdlib v1.x there were several functions that produced a string output but did not have an "a_*" variant of the function. Certain functions (particularly involving hexadecimal conversions) did not support all possible HLA data types. These oversights are being corrected in HLA stdlib v3.x and later.

Use With Other Assemblers

Although users of other assemblers (that support static linking) could always call HLA standard library routines, the new HLA stdlib is being explicitly designed with this in mind. Furthermore, all of the stdlib code is being written in mostly "pure" assembly language so that "purists" will find calling stdlib code less objectional.

Using the Stdlib as an Educational Resource

One legitimate complaint about the HLA v1.x's source code is that it was offered as an education resource – source code that people could study – yet it was almost all written using high-level assembly (i.e., taking advantage of high-level-like control structures in assembly language) which doesn't do a great job of teaching low-level assembly langauge programming. To address those complaints, as well as produce slightly more efficient library code, the new HLA stdlib is written in "mostly pure" assembly language. Procedure calls to other library routines will continue to use high-level calling syntax (because the advantages of type and parameter checking and improved readability are just too great to pass up) unless there is a strong efficiency reason to use the low-level calling sequence.

Improved Performance

Many of the standard library routines have been rewritten using different algorithms to improve their performance (note that in the speed vs. space matrix, speed has been chosen over space in almost every case).

Automated Test Suite

Perhaps the largest addition to the standard library is the availability of an automated regression test for each module in the system. Beyond making the code more reliable (by catching defects in the system before releasing each library module), this automated test suite makes it possible for anyone to make changes to the library and then be able to test their changes so they don't have to worry about breaking the library.

Automated Build Process

The first version of the standard library was built using hand-written make files. Maintaining these make files was a lot of work and was very error-prone. Often, certain library functions wouldn't get rebuilt at all and it was common for the make dependencies to be incorrect. Starting with v3.x of the standard library, there is an automated makefile generation program that goes through all the standard library functions and automatically builds their makefile entries, ensuring a correct build every time.

Improved Community Support

The HLA Standard Library v3.x and later has been designed to better allow contributions from the HLA programming community. In addition to the automated test suite and the automated build process, several other changes have been made to encourage the HLA programming community to contribute to this project. First, and most important, the documentation for the HLA stdlib v3.x and later has been written with Microsoft Word and saved in an RTF format (rather than Adobe Framemaker that was used for the documentation for v1.x of the library). Now anyone who modifies a library routine or adds a routine to the library can check out the RTF file and update the documentation. Furthermore, the current reorganization was explicitly done to support a SourceForge repository for the standard library.

HLA v2.0

The HLA Standard Library, v3.x and later, is an integral part of the HLA v2.0 project. HLA is being rewritten in HLA itself. A large part of the motivation for rewriting the standard library has been to support the HLA v2.0 project. The HLA v2.0 system will not be complete until a new standard library was written for it, so it makes good logistical sense to do the library first, so it can be used while writing HLA v2.0 (and so major sections of HLA v2.0 won't have to be rewritten down the road, when the library changes significantly).

HLA Stdlib v3.x

The HLA Standard Library, v3.0 is the initial completion of the HLA stdlib v2.x project. All the the code was rewritten and all the documentation was completed and the version number was changed from 2.5 to 3.0. It also turns out that the port to FreeBSD also coincided with the release of HLA stdlib v3.0.

Version History:

Stdlib v3.1

v3.1 fixed the strings module documentation (which was never really updated from v1.x), added lots of new string functions, added the new os-independent filesys code, and incorporated many new string functions.

Stdlib v4.0

v4.0 introduces the port to Mac OS X, along with a large number of bug fixes and a few new functions.

Current Status

Current Version: v4.0 – 5/2/2008

Includes Mac OSX, FreeBSD, Linux, and Win32 versions of the library

Using Stdlib v4.x

Note: The HLA stdlib v4.x requires HLA v1.102 or later in order to compile the library code and compile and run the test suite.

Note that several stdlib v1.x fileio functions have been moved from the fileio module to the filesys module in stdlib v4.x. See the filesys.hhf header file for the list of routines that have moved.

If you have any old HLA code that makes use of the HLA stdlib v1.x, HLAStdlib_rtf.zip HLAStdlib_html.zip you should port it to use the new standard library as quickly as possible. The HLA project will no longer support the old v1.x standard library. During the porting process, you should create a set of extra directories in your HLA directory: include1, include4, hlalib1, and hlalib4. When working with older code that requires HLA stdlib v1.x, copy the include1 and hlalib1 directories to the include and hlalib directories inside HLA. When working with new code, copy the include4 and hlalib4 directories to the include and hlalib directories. By doing this, you can avoid having to mess around with environment variables all the time. Of course, you should move your source code base to the HLA stdlib v4.x or later as soon as possible and avoid using the older library code.

Note that it only took a day to convert all the HLA example programs and both sets of Art of Assembly sample programs (for the printed edition and for the electronic edition) to HLA stdlib v4.x. Of course, this was done by someone (Randall Hyde) who was intimately familiar with both sets of library semantics, but the bottom line is that the conversion process isn't that difficult and most offending library calls are syntactically incorrect (so you get compiler errors if there is a problem). If you have questions about how to convert any functions to stdlib v4.x form, you should post a message to the AoA/HLA programming group on Yahoo (http://tech.groups.yahoo.com/group/aoaprogramming/)

HLA Stdlib v4.x Downloads

The following links are the zipped documentation files for the HLA standard library v3.x and later. Download the RTF files if you intend to edit the documentation, download the HTML files if you simply plan to read the documentation. You can read the on-line documentation (HTML format) directly from this page with the links below.

Download Documentation (RTF files, zipped)

Download Documentation (HTML files, zipped)

On-Line Documentation

• Passing Parameters to HLA stdlib functions (read this first)
• Args Module
• Arrays Module
• Bits Module
• Chars Module
• Console Module
• Conversions Module
• Coroutines Module
• Cset Module
• Date Module
• Environment Strings/Variables Module
• Exceptions Module
• File Class Module
• Fileio Module
• Filesys Module
• HLA Constants and Equates Module
• HLL Module
• Linux Module
• Lists Module
• Math Module
• Memory-mapped Files Module
• Memory Module
• Patterns Module
• Random Number Generator Module
• Stderr module
• Stdin module
• Stdout module
• Strings Module
• Tables Module
• Time Module
• Timer Module
• ZStrings Module

Building the Library

Building mkMake

To build mkMake, you'll need a copy of Borland's make.exe program under Windows or the GNU gmake program under Linux or FreeBSD (which generally ships with all Linux/FreeBSD systems). If you're a Windows user and you don't have Borland's make.exe installed, you can get a free copy from their website (last time I looked, it was found at http://cc.codegear.com/free/turbo; their make program is part of their Turbo C++ package).

To build mkMake, CD into the mkMake subdirectory. Execute the "make" command on the supplied Makefile to compile mkMake.hla to mkMake.exe (under Windows) or mkMake (under Linux and FreeBSD). Once this is done, execute mkMake from within the mkMake directory. Note that mkMake will compile under either the HLA stdlib v1.x or the HLA stdlib v3.x (or later). However, support for HLA v1.x was created only so that the first make files could be generated for the HLA stdlib v3.x. At this point, because the binary library code is available, there is no reason for building mkMake using the older library code.

Running mkMake

Note: you must have set the hlainc and hlalib environment variables to point at the HLA stdlib v3.x include directory and the HLA stdlib v3.x hlalib.lib file (in the hlalib subdirectory) before running mkMake. The mkMake program, itself, reads these environment variables in order to determine where the include and library files can be found. If they are not set correctly, you'll probably get a bunch of error messages when you try to run mkMake.

You must run mkMake from within the mkMake subdirectory (which, itself, must be in the hlalibsrc subdirectory). The mkMake program will move up to the parent directory and look for a set of subdirectories with known names (like args, stdin, math, etc.) and will change into those directories and scan for all HLA source files found in those directories. It will process each file, looking for #include and #includeonce statements, and build the associated make files with appropriate dependency lists. In each directory that it scans into, the mkMake program will create a file called <dirname>.mak (e.g., "args.make" and "stdin.mak"). Finally, in the main directory (the parent of the mkMake directory), it will create a "Makefile" file that will invoke each of the other make files it has generated.

mkMake has a lot of work to do, so it may take a couple of minutes to run. Be patient. Do watch the screen, however, to ensure that you're not getting error messages because the environment variables weren't set up properly.

Currently the mkmake program generates a set of makefiles compatible with Borland's make program under Windows and the FSF "gmake" program under Linux and FreeBSD (and just about everywhere else). The mkMake program (which is included with the HLA library source code) can be easily modified to generate make files for Microsoft's nmake program (basically, you'll need to do a search and replace to change all occurrences of "make" to "nmake"). You can obtain a copy of Borland's make from their website as part of the free compiler tools they distribution (see the link earlier, or scan the Borland/CodeGear website). I have not personally run gmake (GNU make) under Windows, but there is no reason that gmake shouldn't be able to process the generic make file that mkMake produces. Note, however, that there are some differences between the make files produced for Windows and the make files produced for Linux and FreeBSD. Therefore, you cannot copy the makefile produced under one OS to the other and expect it to run properly. You must run the mkmake program on the OS where you wish to build the stdlib.

You must also *manually* build a couple of utility programs that the system uses (delete and hlacmp). My apologies for not automating these, but it wasn't easy to automate the builds of these programs given the current state of the HLA stdlib v3.x. Note that binary versions of these utilities are include with the HLA stdlib source code, so unless you delete them, you should be able to use them as-is.

There are two methods for building the HLA stdlib v3.x:

1. Building the HLA stdlib v3.x from scratch on a system with HLA stdlib v1.x installed.
2. Building the HLA stdlib v3.x from an HLA system that already has HLA stdlib v3.x installed.

Method 1 is a lot more work and really shouldn't be necessary anymore as you can download HLA with HLA stdlib v3.x already installed (HLA v1.100 and later). Therefore, this documentation will skip that discussion.

Method 2 is a whole lot easier.

Here are the steps you should go through to build the library under Linux with HLA stdlib v3.x already installed:

1. Under Windows, I always build the library with HLA using MASM as the back-end to HLA. This allows me to compile in debugging information for use by OllyDbg and other debugging programs. I usually set the following environment variables to ensure that HLA uses MASM in this manner (this is optional): "set hla=mhla", "set hlaasmopt=-Zi -Zd -Zf", "set hlalink=mslink", and "set hlalinkopt=-DEBUG". This step is fully optional; you may compile the stdlib using the built-in FASM back end without any problems, though you will not get debugging information incorporated into the object files. Under Linux and FreeBSD, you should opt for Gas by setting the environment variables "hlaasmopt=-cg -gstabs" to obtain debugging information under those OSes. Again, this is an optional step.
2. Assuming the mkMake executable isn't already available in the mkMake directory, CD into the mkMake directory and execute the "make" command. This will build the mkmake program that you'll use to create the makefiles that will build the HLA stdlib v3.x.
3. If the binaries aren't available in the main hlalibsrc directory, compile the "delete" and "hlacmp" applications. Just use the commands "hla delete" and "hla hlacmp" to accomplish this. Like mkmake, these two applications were designed to be compilable under either HLA stdlib v1.x or v3.x. In general, you should already have the binaries (or can easily download new copies if you've lost them), so this step shouldn't be necessary.
4. Now, run the mkMake program by typing "mkMake" on the command line. You must still be in the mkMake directory and the mkMake directory must still be in the hlalibsrc directory (along with the directories of all the other modules). The mkMake program moves up one level in the directory hierarchy and processes all the directories and files in that outer directory, so make sure you run mkMake executable from inside the mkMake directory. If you start getting messages about how mkMake can't find certain include files, that probably means you've not set the hlainc environment variable or that you've run mkMake executable from an improper location.
5. Once mkMake successfully completes, you want to CD back into the main hlalibsrc directory (e.g., by using the command "cd ..").
6. Okay, now you're ready to build the library. Just type "make" at the command line and several minutes later, you should have a completed library. Note that the build process leaves "hlalib.lib" (Windows) or "hlalib.a" (Linux and FreeBSD) sitting in the hlalibsrc directory. If you actually want to link against this code, you will need to move the file to whatever file hlalib is referencing (typically something like "/usr/hla/hlalib/hlalib.a"). It would be a wise idea to save the original hlalib.lib/hlalib.a file on your system in case there is some sort of disaster and the new library you've built doesn't work.

Once you've successfully built the HLA standard library v3.x, it's a good idea to run the regression test suite on it to make sure everything is cool and operating as expected. The test suite is fully automated. To run the test suite, do the following:

Under Windows:

1. Make sure you're in the hlalibsrc directory (or whatever name you've used for the directory that holds all the stdlib v3.x modules).
2. Make sure the hlainc and hlalib environment variables point at the HLA stdlib v3.x include files and at the hlalib.lib/hlalib.a file you built earlier.
3. Make sure you've got the "delete" and "hlacmp" applications built before running the test suite.
4. If you're using MASM as the back end to HLA, make sure you reset the hlaasmopt environment variable before attempting to run the test suite. Unfortunately, the PE/COFF object file format's support for debugging information does not allow source files longer than 65,535 lines. Alas, many of the test suite source files emit MASM sources files well in excess of 65,535 lines of code. If you do not turn off the debugging information (the "-Zi -Zd -Zf" option listed earlier), the test suite will bomb out in the middle and you'll get to start it all over again :-(. To turn off the debug options, just type "set hlaasmopt=" from the command line. I recommend building the stdlib in a separate command window from the one you use to test it, so you're not constantly having to turn this option on and off during development. (Note that this is only an issue under Windows, Linux/FreeBSD users can ignore this.)
5. The test suite will take a while to complete (depends on the speed of your machine, but it will probably take between 10 and 50 minutes for most machines still in active use). Go have some coffee. Or do something else. Unless you have a multi-core machine, don't try to continue using your machine -- the CPU usage will be pegged (well, you can always lower the process priority if you must continue to use the machine).
6. The Win32 version of the test suite should automatically stop if it detects any errors. So you can ignore the machine for a while until the test suite finishes.
7. The Linux/FreeBSD version of the test suite ought to automatically stop on an error, but there are many cases where it does not (a bug I've got to fix someday). If you've made some changes to the library, I'd recommend testing that module individually and paying attention to the test, then running the whole test suite once you've checked an individual module. To test an individual module, you need to CD into the "unit_tests" directory of the module's directory (e.g., for the patterns module, you'd CD into "patterns/unit_tests"). Just type "make" from the command line once you're in the unit_test directory and the system will do the rest.

Note that some of the tests may take several minutes to run all by themselves. Be patient. I don't believe there are any infinite loops and exception handling code appears liberally in most of the test suite functions. If a function seems like it's taking a very long time, the reason is probably because that function takes a very long time.

Cross development between Windows, FreeBSD, and Linux

If you're doing work on the HLA stdlib v3.x for your own personal reasons and you don't intend to share your code with anyone, you can limit your changes to whatever OS you happen to be most happy with. However, if you want to use your code under multiple OSes or you want to share your code with users who might need to support "the other OS", you'll need to set up your workflow to do cross-OS development.

I, personally, do all my development under Windows and switch over to Linux and FreeBSD only for compiles and tests. Personally, I find the tools under Windows more to my liking. No doubt, some of you might prefer working under Linux and switch to Windows only for compiles and tests. It really doesn't matter. What's important is that you make sure that you're working from a single, consistent, code base for both Windows and Linux. I have used SAMBA to map my Windows development disk to a directory on my Linux box. This allows me to edit my files under Windows and with the press of a (KVM) switch, switch over to Linux and compile the code there (indeed, I'm getting ready to purchase a new Mac Pro box that will allow me to run Windows, Linux, and Mac OS simultaneously in separate windows on the same screen, to make this process even easier). There is no reason you cannot do the same thing in reverse -- edit your files under Linux, press the KVM switch, and compile them under Windows.

The only catch is that pesky line-ending issue that exists between Windows and all the other OSes. I would love to say that the HLA compiler ignores the type of line endings (mostly, it does) and that most tools you'll use ignore line endings. Alas, this isn't true. The make program under Linux, for example, doesn't do line continuations very well if a "\" character is followed by a <CR> rather than a <LF>. Also, many of the output verifications that take place in the test suite will report differences in output if the test file contains one type of line ending and the standardized data file contains the other type of line ending.

The bottom line is that if you're going to do cross-platform develoment, you need to convert the line endings.

I have written a Win32 application, textto (name taken from QNX, if you're wondering) that translates line endings between the various OSes (it will even work with old Mac OS9 [and earlier] <CR> terminated line endings). When I do cross development, I make a copy of my hlalibsrc3 and include3 directories to my mapped drive. I then execute the following commands on these copies of those directories:

cd include3
textto -u -r *.hhf

cd hlalibsrc3
textto -u -r *.hhf
textto -u -r *.hla
textto -u -r *.data
textto -u -r Makefile

Of course, you should make a shell script to do all this for you automatically, unless you really like typing everytime you start some cross-development work.

My (Windows) editor of choice is Code Wright, and it automatically figures out whether a file should have Unix line endings or Windows line endings, so I can edit the files in the Linux directory copies without fear of corrupting them.

Of course, after several edits and compiles, the Linux working directory becomes inconsistent with respect to the original Windows directory. Though I try to copy the files back and forth as I work on them, like most humans I make mistakes and fail to keep the two directories in sync. That's where a good directory diff program, like "Beyond Compare" or "Araxis Merge" come in real handy (no doubt, there are free equivalents of these programs, but I happen to like these two commercial programs). They let me easily see if the two directories have gotten out of sync and let me bring them back into sync.

Though not strictly necessary for cross platform development, I cannot imagine anyone doing serious work on a project the size of the HLA stdlib without using some form of source code control. I personally maintain the HLA stdlib using Subversion and the Tortoise GUI interface to Subversion. This lets me easily back out of changes I've made that wound up breaking the code. I highly recommend Subversion (or some other source code control system) to anyone working on the library code.