Brendan's Developer Diary
Contents:
1 September 2008 - The Build Utility and Other Tools Revisited
1 September 2008 - Floppy Image Utility Done
31 July 2008 - Redundant Floppies?
22 July 2008 - "DOC_GEN" Utility Done!
7 July 2008 - Technical Documentation
3 July 2008 - Website Generator
1 September 2008 - The Build Utility and Other Tools Revisited
With everything else out of the way it became time for me to write the fourth and final utility for generating web pages. The idea was to have one utility that generates the web site's main pages (the home page, this page, etc), one utility to generate specifications, one utility to generate web pages corresponding to assembly source code (and to build/assemble the source code, disassembly the resulting binary, do hex dumps, etc), and one utility to generate "glue" pages to make it easier for people to navigate through the other pages.
With this in mind I started the utility to generate the "glue" pages. I didn't get far.
The main goal of these utilities is to make it easy to maintain the web site by automating most of the tedious work (like cross-linking web pages, making sure the formatting is consistent, etc). The problem is that over time I ended up with slightly different markup for different utilities, and having 4 different utilities to maintain would have been a pain in the neck. The solution? A large utility that does all of it.
Well, not quite all of it. The utility that generates the web site's main pages is considered part of the web site and not part of any version of the OS. From the home page the "Current Project" link sends your browser through a symbolic link to the current project's web pages, so that (for e.g.) I can copy the current project into a new directory and update the symbolic link, and maybe one day have links to previous/older versions of the project (or maybe a "stable" and a "development" branch).
Anyway, I've been working on a system build utility that incorporates everything (except the web site's main pages). This should mean faster build times (because more can be done in parallel), no need to manually track dependencies (e.g. if one binary is included by another the build utility can figure that out and wait for the first binary to be assembled before assembling the second binary), different files can easily reference information in other files using identifiers rather than HTML links (which end up as dead links when files are moved). As a bonus, it's already generating a full "project map" and navigation links in each web page, and automatically creating directories if they don't exist (so I can add a new source code directory without also needing to add a new directory for the resulting web pages). It's also doing recursive directory searches too, which mostly means I'll never need to create or modify any scripts when new pieces are added to the project.
Basically, I'll be able to just write the source files, and I'll never need to worry about dead links, formatting, scripts, dependencies, makefiles, etc.
If you're curious, this is all "live" already - you'll be able watch my progress by viewing the resulting web pages on-line while I write the utility. Just click on the "Current Project" link at the top of this page...
1 September 2008 - Floppy Image Utility Done
Well, I've been slack - this utility was actually completed a few weeks ago...
Now I can tell the utility what format the floppy is (in KiB, from "2880" to "160") and where to find the relevant files, and it'll create a suitable floppy image. This includes automatically deciding if a redundant floppy image should be used, setting relevant fields in the BPB according to the floppy format, setting a "redundant" flag in the boot sector, calculating CRC values for any of the files that don't have one yet and doing any padding (so that different files can grow without needing later files to be relocated). The floppy boot code supports all of this too.
The end result is that now I can easily generate floppy images for lots of different floppy formats, ranging from 2880 KiB floppies down to (very obsolete) 160 KiB single sided floppies, including the standard 1440 KiB floppies and the "not quite standard" 1680 KiB floppies.
31 July 2008 - Redundant Floppies?
I thought it was a good idea, so I implemented it. The idea is that if everything the OS needs is small enough, the boot loader can use the second side of a floppy as a redundant copy of the first side (for e.g. to increase the chance of a successful boot when the floppy has bad sectors). It should work well, it's just that I can't actually test it!
The problem is that the "redundant floppy" format is a little unusual, and isn't supported by any tools. If I could get the floppy image in the right format it'd be easy to write the image onto a standard floppy, it's just finding a good way to get the floppy image into the right format (e.g. "track1, track1, track2, track2, track3, track3, etc" rather than "track1, track2, track3, etc").
The best solution is to write a utility that does all the work - you tell it where to find the binaries (boot loader, faulty RAM list, common PC-BIOS module, boot script and boot image) and it'll create a floppy image containing those binaries, and automatically generate a redundant floppy if there's enough space (or not, if it won't fit). As an extra bonus, it should be simple to generate images for less common floppy formats too, like "1200 KiB 5.25 inch" floppies, "1680 KB 3.5 inch" floppies, "720 KB 3.5 inch" floppies, etc.
It sounds good to me, but to be honest I've been working on boot code to avoid doing utilities. It's funny really - I've already got more code for various utilities than I've got code for the OS, and I need at least 2 more utilities now (one for making floppy images and another for generating "glue" web pages). Oh well, I guess a utility to create floppy images won't take long...
22 July 2008 - "DOC_GEN" Utility Done!
Yes, the new utility to generate technical documentation (in HTML form) from plain text files is done. It supports everything I currently need (and a little more), including tables, figures, listings, contents, cross-referencing, etc. Unfortunately, I haven't really written much technical documentation for the utility to "generate", so you'll need to wait a while before you can see the results...
7 July 2008 - Technical Documentation
I just spent 4 days playing with LaTeX, hoping I could write specifications and user guides in a relatively simple format and convert them into HTML and PDF format. I don't think I've ever used an uglier bunch of tools in my life.
It's several layers of wrappers, scripts, etc that won't behave like a sane utility (only displaying errors and warning) and instead fills the screen with pages of dribble for the simplest of tasks; and insists on plastering the source directory with a plethora of obscure "temporary" files (foo.toc, foo.4ct, foo.ct4, foo.toc, foo.log, foo.lg, foo.wtf, etc). To fix these issues I had to write my own BASH script (yet another wrapper!) - copy the source file into a temporary directory, send it's terminal output to /dev/null, move the file/s you want back where you want them, then delete all the trash it left in the temporary directory. My script worked...
Of course none of the tools check if the input files are newer than the output files, so I also needed to use "make" to keep the OS's build fast. Yes, another layer on top of the "wrapper and script" pile.
For HTML output, first I tried "latex2html". It worked for what I tried, but the result was so ugly that I couldn't use it. I never did find a way to get rid of the "about" page (a blatant advertisement in my opinion), which is a bit scary when you're thinking of doing 20 or more documents (with 20 or more "about" pages, and 20 or more style sheets). Then I found "TeX4ht" which generated much better looking HTML output (still with style sheets so you get 2 files when you need one, but I could've lived with that).
Unfortunately, the set of wrappers and assorted crud known as "Tex4ht" doesn't work once you get past "Hello World". The latest problem is tables - the parser works fine for PDF output, but for exactly the same input file the HTML files are truncated in the middle (where the table is). Is there a sane error message or warning? No. No message sent to the screen, no message sent to the log file, and no message sent anywhere else. It's not the first problem I had either - I learnt you need to constantly check the HTML output and the PDF output just in case the "contents" goes missing or something.
It's my own fault I guess - I should've known that a utility that needs to be executed twice just so it doesn't stuff up the table of contents isn't worth the time it takes to download.
To be fair, I also took a quick look at Docbook (or XML, or SGML, or whatever they feel like calling it today). It's another set of scripts, wrappers and miscellaneous crud! Apparently to convert Docbook files into PDF format it gets converted into Tex format first (the same borked mess that lives under all the LaTeX wrappers), and the HTML output from "docbook2html" reminds me of "latex2html" - not too pretty.
Basically I've decided I won't be using any of these tools - I'll write my own utility to generate HTML from simple text. Sadly this will mean there won't be specifications, user guides, etc in PDF format, but that's the least of my problems.
Now you're probably wondering why I don't just use HTML to begin with. The reason is that for HTML I need to do the formatting (and linking) manually, which is a pain in the neck when you're looking keeping many files cross-linked and formatted consistently. For LaTeX and Docbook (and my "website generator" utility) you don't need to do this - the tools do the formatting for you, which (in theory) makes it easier to write the document. Of course this isn't the case with LaTeX or Docbook - the syntax you need to use is more verbose and error-prone than HTML.
Anyway, tomorrow I start the new "document generator" utility...
Generated on Mon Sep 1 13:39:23 2008
(UTC)