Build Utility Documentation

Contents:

• Chapter 1: Introduction
• • 1.1. Invocation
  • • 1.1.1. Command Line Arguments
    • 1.1.2. Command Line Flags
• Chapter 2: Project Directory Handling
• • 2.1. Special Directories
  • • 2.1.1. End User's Documentation Directory
    • 2.1.2. Programmer's Documentation Directory
    • 2.1.3. Specifications Directory
    • 2.1.4. The Output Directory
  • 2.2. Special Files
• Chapter 3: Special Services
• • 3.1. HTML Directory Cleaning
  • 3.2. Backups
  • • 3.2.1. Backup Files
    • 3.2.2. Backup Directory Maintenance
• Chapter 4: File Types
• • 4.1. Association Files
  • • 4.1.1. Association File Header
  • 4.2. Document Files
  • • 4.2.1. Document File Header
  • 4.3. Specification Files
  • • 4.3.1. Specification File Header
  • 4.4. Assembly Language Source Files
  • • 4.4.1. Assembly Language Source File Header
    • 4.4.2. Assembly Command Escape Sequences
  • 4.5. C Source Files
  • • 4.5.1. C Source File Header
    • 4.5.2. C Command Escape Sequences
  • 4.6. Script Files
  • • 4.6.1. Script File Header
    • 4.6.2. Script Command Escape Sequences
    • 4.6.3. Script File Execution
• Chapter 5: Markup
• • 5.1. Simple Markup
  • 5.2. Complex Markup
  • • 5.2.1. Paragraphs
    • 5.2.2. Lists
    • • 5.2.2.1. Plain Lists
      • 5.2.2.2. Numbered Lists
      • 5.2.2.3. Indented Lists
      • 5.2.2.4. Nested Lists
      • 5.2.2.5. Mixed List Types
    • 5.2.3. Section Titles
    • 5.2.4. Listings
    • 5.2.5. Tables
    • 5.2.6. Figures
    • • 5.2.6.1. Figure Styles 0 and 1
      • 5.2.6.2. Figure Styles 2 and 3

Tables:

• Table 1-1. Command Line Flags
• Table 4-1. Assembly Command Escape Sequences
• Table 4-2. C Command Escape Sequences
• Table 4-3. Script Command Escape Sequences
• Table 5-1. Simple Markup
• Table 5-2. Table Style 0 Example
• Table 5-3. Table Style 1 Example
• Table 5-4. Table Style 2 Example
• Table 5-5. Table Style 3 Example

Listings:

• Listing 4-1. Association File Header Example
• Listing 4-2. Assembly File Header Example 1
• Listing 4-3. Assembly File Header Example 2
• Listing 4-4. C File Header Example 1
• Listing 4-5. C File Header Example 1
• Listing 4-6. Script File Header Example
• Listing 4-7. Script Example 1
• Listing 4-8. Script Example 2
• Listing 5-1. Plain List Example
• Listing 5-2. Numbered List Example
• Listing 5-3. Indented List Example
• Listing 5-4. Nested List Example
• Listing 5-5. Mixed List Example
• Listing 5-6. Inappropriately Mixed List Example
• Listing 5-7. Listing Example
• Listing 5-8. Listing Style 0 Example
• Listing 5-9. Listing Style 1 Example
• Listing 5-10. Listing Style 2 Example
• Listing 5-11. Listing Style 3 Example
• Listing 5-12. Listing Style 4 Example
• Listing 5-13. Listing Style 5 Example
• Listing 5-14. Table Example
• Listing 5-15. Source For Figure Style 0 and 1 Examples
• Listing 5-16. Source For Figure Style 2 Example and Figure Style 3 Example

Figures:

• Figure 2-1. Example Project Directory Structure
• Figure 5-1. Figure Style 0 Example
• Figure 5-2. Figure Style 1 Example
• Figure 5-3. Figure Style 2 Example
• Figure 5-4. Figure Style 3 Example

Chapter 1: Introduction

The Build Utility is a utility that has been custom designed to aid in the management and maintenance of the BCOS project, including:

• Automatically removing obsolete (deleted or renamed) files from the web site
• Automatically detecting if/when files have been modified
• Automatically tracking dependancies between individual pieces of the project
• Automatically assembling, compiling and executing scripts to build individual pieces of the project (if necessary, when dependancies are satisfied)
• Automatically creating backups of the project

1.1. Invocation

WARNING: When started, the Build Utility assumes that the current working directory is the project directory. If this is not the case then expect it to create unwanted directories and unwanted files, and (potentially) delete wanted files and wanted directories.

The Build Utility has 1 command line arguments, and several command line flags.

1.1.1. Command Line Arguments

The only (optional) command line argument is the name of a backup directory. If no backup directory is supplied then no backup is done. See Section 3.2. Backups for more information on how backups are performed.

1.1.2. Command Line Flags

The command line flags are described in Table 1-1. Command Line Flags.

Chapter 2: Project Directory Handling

The Build Utility processes an entire directory tree (including subdirectories) that make up the project, and creates 2 output directory trees (one for the end user documentation alone, and one for the web site and developers) containing the generated HTML files. Certain subdirectories within the project's directory have special meaning.

For an example, the directory structure for a project might look like this:

2.1. Special Directories

2.1.1. End User's Documentation Directory

The end user's documentation directory ("udoc") contains all text files for end user documentation. These files are actually converted to HTML twice (once for the "stand-alone" end user's documentation, and again for the main HTML directory). The Build Utility ensures that there are no links from the end user's documentation to other HTML files.

2.1.2. Programmer's Documentation Directory

The programmer's documentation directory ("pdoc") contains all text files for programmer's documentation (things like tutorials, examples, etc).

2.1.3. Specifications Directory

The specification directory ("spec") contains all text files for specifications.

2.1.4. The Output Directory

The output directory ("_out") contains 2 subdirectories. The first ("_out/www") contains the project's main web including specifications, programmer's documentation and end user documentation. The second ("_out/udoc") contains the stand-alone end user's documentation.

2.2. Special Files

Within the output directory, the Build Utility creates 2 web pages containing site maps ("_out/www/_map.html" and "_out/udoc/_map.html"). These are just to assist people navigating through the HTML.

Within the main project directory, the files "log.txt", "todo.txt" and "features.txt" get special handling. They are the only documents that are allowed outside the "udoc" and "pdoc" subdirectories, and are intended to contain the Project's log, a list of reminders (things that still need to be done) and a list of the project's features.

Chapter 3: Special Services

3.1. HTML Directory Cleaning

When source files and directories are renamed, moved or deleted; the corresponding HTML versions of those files become "orphaned". A previous version of the Build Utility didn't do anything about this, which meant that developers had to manually remove any "orphaned" HTML files and directories; and because it was easier for developers to simply delete the entire HTML directory and let the Build Utility regenerate all HTML files, this led to 2 minor problems:

• there's a small amount of time where the project didn't exist on the web site (after the HTML files had been deleted but before the build utilty has regenerated them)
• it made the time stamps at the bottom of HTML pages inaccurate (the time stamp would often reflect when the developer/s had wiped out the HTML directory last, rather than reflecting when the corresponding source file was modified last)

To solve these problems and make things even easier for developers; this version of the Build Utility will search for and remove any orphaned files within the "_out" directory by deleting any normal files that have a ".html" extension that couldn't have been created from the current set of source files. In a similar way, the Build Utility will remove any empty subdirectories within the "_out" directory tree.

The Build Utility will not follow symbolic links to files and directories, and won't remove files that don't have a ".html" extension, as neither of these things could have been created by the Build Utility. It is possible (and sometimes advantageous) to take advantage of this. For example, it's possible to have an "ftp" or a "pictures" directory somewhere, with symbolic links to these directories.

3.2. Backups

3.2.1. Backup Files

If a backup directory is specified as a command line argument (see Section 1.1. Invocation) the Build Utility will create a backup of the project every time it is run. The backup consists of a "tar" of all files in the project directory, excluding files that are directly or indirectly generated by the Build Utility (HTML files directly generated by the Build Utility, and any files that are indirectly generated when the Build Utility executes assemblers, compilers and scripts).

The new backup will be created with a name that reflects when the backup was made (e.g. the file name will have the form "yyyy-mm-dd_hh-mm-ss.tar").

3.2.2. Backup Directory Maintenance

If the Build Utility created a backup every time it runs but did nothing else, it would quickly add up to a large number of (mostly redundant) files. To save disk space, the Build Utility will clean the backup directory and remove (some) older backups. The rules for the backup directory maintenance are as follows.

First, files that have the form "yyyy-mm-dd_hh-mm-ss.tar" are compressed. This includes the backup that was just created; and means that if anything goes wrong with compressing it then it'll just be done next time. It also means that if someone decompresses a backup (perhaps to recover a lost file or something) then the Build Utility will automatically compress it again later.

Second, files that have the form "yyyy-mm-dd_hh-mm-ss.tar.gz" are examined, and the 2 most recent days are determined (these days may not be sequential - e.g. if backups are created on Monday, Tuesday and Thursday but not Wednesday, then the 2 most recent days may be Tuesday and Thursday). For backups created before the 2 most recent days, the Build Utility determines the oldest backup for each day and renames it into the form "yyyy-mm-dd.tar.gz" and deletes all other backups from that day.

Third, files that have the form "yyyy-mm-dd.tar.gz" are examined, and the 2 most recent months are determined (these months may not be sequential). For backups created before the 2 most recent months, the Build Utility determines the oldest backup for each month and renames it into the form "yyyy-mm.tar.gz" and deletes all other backups from that month.

The end result of all of this is that you end up with any number of backups from the last few days, up to 61 backups from previous days, plus one backup per month for previous months.

Chapter 4: File Types

4.1. Association Files

Association files are mostly intended for navigating the web site. If you think of the web site as a hierarchical tree, the association files are the branches (while the leaves are documentation and specifications). In addition to site navigation, association files are useful for describing the purpose of areas of the web site.

4.1.1. Association File Header

An association file begins with the file's title on the first line. This is (optionally) followed by any subtitle lines (lines starting with a '+' character). For an example, see Listing 4-1. Association File Header Example. The header is followed by the page's contents (simple and complex markup).

4.2. Document Files

Document files are intended to contain FAQs, usage instructions, tutorials, examples, etc.

4.2.1. Document File Header

The document file header is the same as the header for association files (see Subsection 4.1.1. Association File Header).

4.3. Specification Files

Specification files are intended to contain formal specifications for file formats, protocols, APIs, etc.

4.3.1. Specification File Header

The specification file header is the same as the header for association files (see Subsection 4.1.1. Association File Header).

4.4. Assembly Language Source Files

For binary files generated from assembly language source code, the assembly language source code must have a special file named 0index.asm. This special file includes the assembly language source file header, then the assembly language source code itself.

Multiple files are included by using the assembler's %include directive in the special 0index.asm file, and other binaries can be included by using the assembler's incbin directive in special 0index.asm file. The build utiity will automatically find these directives to detect all of the dependancies.

There must not be any %include directives or incbin directives in other files (e.g. in files included by the 0index.asm file, as the build utility will not find these and will not know there are additional dependancies. This practice has the added benefit of making source code maintenance easier - for example, all things that effect the structure of the output binary (e.g. the ORG or section definitions, the file header, and any included source and binary files) all in one place in 0index.asm, plus normal source files that only contain code and data (and don't effect the structure of the output binary).

For any "*.inc" files that aren't in the directory; the build utility will try to find the in a "inc" directory of the parent directory, then the "inc" directory of the parent's paren, and so on (up until the project's root directory). For example, if the file src/80x86/hello/0index.asm contains the line %include "foo/bar.inc", then the build utility will search for src/80x86/hello/foo/bar.inc, then src/80x86/inc/foo/bar.inc, then src/inc/foo/bar.inc, then inc/foo/bar.inc (stopping when it is found). This allows you to have a system of include directories where where more specific files take precedence over less specific files (e.g. the architecture specific src/80x86/inc/foo/bar.inc file would take precedence over the less specific src//inc/foo/bar.inc file).

4.4.1. Assembly Language Source File Header

The first line of the 0index.asm file must contain a list of output binaries (separated by whitespace). The next line must contain a (possible empty) list of additional input files. The next lines (starting with a '+' character) contain commands to convert the source code into each of the binaries listed on the first line. For an example, see Listing 4-2. Assembly File Header Example 1.

4.4.2. Assembly Command Escape Sequences

Assembly commands can have special escape sequences, where the escape sequence is expanded before the command is executed. These escape sequences are listed in Table 4-1. Assembly Command Escape Sequences.

Using these escape sequences, the assembly file header shown in Listing 4-3. Assembly File Header Example 2 is equivelent to the assembly file header shown in Listing 4-2. Assembly File Header Example 1.

4.5. C Source Files

For binary files generated from C source code, the C source code must have a special file named 0index.c. This special file includes the C source file header, then the C source code itself.

Multiple files are included by using the compiler's #include directive in the special 0index.c file, and the build utiity will automatically find these directives to detect all of the dependancies. There must not be any #include directives or in other files (e.g. in files included by the 0index.c file, as the build utility will not find these and will not know there are additional dependancies.

The build utility does not support compiling individual C source files into object files and then linking these object files. C source code is only intended to be used for small utilities, where it is expected the entire source code being compiled and the compiler itself will all fit in memory at the same time (and therefore separate object files aren't needed).

4.5.1. C Source File Header

The first line of the 0index.c file must contain a list of output binaries (separated by whitespace). The next line must contain a (possible empty) list of additional input files. The next lines (starting with a '+' character) contain commands to convert the source code into each of the binaries listed on the first line. For an example, see Listing 4-4. C File Header Example 1.

4.5.2. C Command Escape Sequences

C commands can have special escape sequences, where the escape sequence is expanded before the command is executed. These escape sequences are listed in Table 4-2. C Command Escape Sequences.

Using these escape sequences, the C file header shown in Listing 4-5. C File Header Example 1 is equivelent to the C file header shown in Listing 4-4. C File Header Example 1.

4.6. Script Files

Any file ending with the extension .sh is treated as a script. A script includes the script header, then the contents of the script. The contents of the script includes empty lines, comments (linest beginning with a '#' character) and the commands to be executed.

4.6.1. Script File Header

The first line of the script file must contain a list of output binaries (separated by whitespace). The next line must contain a list of input file names. For an example, see Listing 4-6. Script File Header Example.

For the list of input files, the name html is special. If this name is present then it is not treated as a file name, but is used to indicate that the script depends on any of the files in the project's _out directory. This makes it possible to (for example) have a script that copies the HTML files for end user documentation onto an OS installation CD, or have a script that creates a "tar.gzip" of the web site for people to download.

4.6.2. Script Command Escape Sequences

Script commands can have special escape sequences, where the escape sequence is expanded before the command is executed. These escape sequences are listed in Table 4-3. Script Command Escape Sequences.

Using these escape sequences, the script in Listing 4-8. Script Example 2 is equivelent to the script in Listing 4-7. Script Example 1.

4.6.3. Script File Execution

Commands in scripts are executed one at a time in the order listed in the script file. Execution stops as soon as any of the commands fail.

Chapter 5: Markup

5.1. Simple Markup

Simple markup involves sequences of characters that always begin with a '[' character and end with a ']' character. Typically the characters inside the square brackets are a keyword, a colon, and some text. The easiest way to describe all available simple markup is to display them.

5.2. Complex Markup

5.2.1. Paragraphs

A new paragraph begins after any other complex markup, or after a blank line. More text can be added to the same paragraph on subsequent lines.

5.2.2. Lists

There are 3 types of lists supported by the Build Utility - plain lists, numbered lists and indented lists.

5.2.2.1. Plain Lists

A line of text that begins with a '-' character will be treated as a line in a plain list. A plain list looks like this:

• First line in list
• Second line in list
• Third line in list

5.2.2.2. Numbered Lists

A line of text that begins with a '%' character will be treated as a line in a numbered list. A numbered list looks like this:

1. First line in list
2. Second line in list
3. Third line in list

5.2.2.3. Indented Lists

A line of text that begins with a '_' character will be treated as a line in an indented list. An indented list looks like this:

• First line in list
• Second line in list
• Third line in list

5.2.2.4. Nested Lists

Lists can be nested up to the the maximum nesting depth. If you reach the maximum nesting depth you'll get an error (but this is unlikely, as you'll probably run out of screen width before you reach the maximum nesting depth). An example:

1. First line in list
2. Second line in list
   1. Third line in list
   2. Fourth line in list
      1. Fifth line in list
3. Sixth line in list
4. Seventh line in list

5.2.2.5. Mixed List Types

List types can be mixed in any combination of ways. Here's an example:

1. First line in list
2. Second line in list
   • Third line in list
   • Fourth line in list
     1. Fifth line in list
        • Sixth line in list
        • Seventh line in list
3. Eighth line in list
4. Ninth line in list

List types can also be mixed inappropriately, where different list types are used at the same nesting depth. If the type of list at any nesting depth changes from one type to another, then the Build Utility just behaves as if a new list has been started at that nesting depth. For example:

1. First line in list
2. Second line in list
   • Third line in list
   1. Fourth line in list
   2. Fifth line in list
   • Sixth line in list
   1. Seventh line in list
   • Eighth line in list
3. Ninth line in list

5.2.3. Section Titles

A section title begins with the characters "*s" at the start of the line, followed by a digit from 0 to 4 that determines the type of section (4 = part, 3 = chapter, 2 = section, 1 = subsection, 0 = subsubsection), a comma, an optional reference name (that can by used to reference the section with "[d:<reference_name>]" or "[s:<reference_name>]"), another comma, then the section's title text.

Note: If four or more section titles are used, or if a table of listings, a table of tables or a table of figures is generated, then a table of contents will be automatically generated from the section titles.

5.2.4. Listings

A listing consists of a listing header and listing lines. A listing header begins at the start of the line with the characters "*%" followed by a digit from 0 to 3 that determines the style of the listing, a comma, an optional reference name (that can by used to reference the listing with "[%:<reference_name>]"), another comma, then the listing's title.

Listing lines begin with the character '+' followed by the listing line's text.

Note: If four or more listings are used then a table of listings will be automatically generated.

All of the following listings use the same source text shown in Listing 5-7. Listing Example, except for different style characters.

5.2.5. Tables

A table consists of a table header, table rows, and table fields. A table header begins at the start of the line with the characters "*#" followed by a digit from 0 to 3 that determines the style of the table, a comma, an optional reference name (that can by used to reference the table with "[#:<reference_name>]"), another comma, then the table's title. A table row begins with the '+' character and consists of table fields seperated by a '|' character, where table fields contain nothing or normal text (potentially with simple markup). Within table fields the simple markup "[|]" can be used to avoid seperating table fields if a '|' character is needed.

Note: If four or more tables are used then a table of tables will be automatically generated.

All of the following tables use the same source text shown in Listing 5-14. Table Example, except for different style characters.

5.2.6. Figures

A figure consists of a figure header and figure lines. A figure header begins at the start of the line with the characters "*f" followed by a digit from 0 to 3 that determines the style of the figure, a comma, an optional reference name (that can by used to reference the figure with "[f:<reference_name>]"), another comma, then the figure's title.

!Figure lines begin with the character '+' followed by the figure line's text. The format and of the figure line's text depends on the figure style.

Note: If four or more figures are used then a table of figures will be automatically generated.

5.2.6.1. Figure Styles 0 and 1

Figure styles 0 and 1 work the same as a listing, where text is displayed "as is". These types are intended for simple figures where a graphics image isn't necessary.

5.2.6.2. Figure Styles 2 and 3

Figure styles 2 and 3 are used to display lists of ranges.

A line that begins with '+' character followed by a (decimal) number defines a range. The number is the size of the range (which must be between 0 and 9); where any text after the range size but before an '|' character is the range's description (which maybe be empty). Any text after the '|' character is a "bottom of range" marker. The "bottom of range" marker and the '|' character may be ommitted if no marker is desired. Also, "[|]" may be used within the range description if a '|' character is needed.

Because the top of one range is usually the bottom of the previous range "top of range" markers aren't needed. However, for the top of the first range there is no previous range and a "top of range" marker may be needed. A "top of range" marker is a line that has no range size, where the first non-whitespace character is a '|' character.

A line that begins with '+' and has no range size, where the first non-whitespace character is not a '|' character is a note. A line that begins with '+' that has no range size and no non-whitespace characters is a seperator. Notes, seperators and "top of range" markers can be inserted between ranges to create multiple lists of ranges within the same figure.