NuLib2 Manual
Home ] NuLib Downloads ] NuLib Library ] [ NuLib2 Manual ] NufxLib API ] Bugs & Features ]


 
NuLib2 v2.1.1 Manual - By Andy McFadden - Last revised 2006/02/18

Table of Contents

 

Introduction

NuLib2 is a command-line archive utility, along the lines of "PKZIP".  It allows you to perform many common operations on NuFX archives, such as those created by the Apple II "ShrinkIt" utility, as well as Binary II archives.  Files with extensions "SHK", "SDK", "BXY", "BSE", "SEA", "BNY", and "BQY" are handled.

You can add, delete, extract, test, and list files in a NuFX archive.  Compressed disk images can be extracted to files, and vice-versa, making it a handy utility to have when using an Apple II emulator.

NuLib2 is the successor to NuLib, which did many of the same things.  NuLib2 is not meant to set a new standard or fight for supremacy on the PC desktop; rather, it is intended to help people working with Apple IIs and Apple II emulators.  All compression algorithms specified in the NuFX specification are fully supported.

NuLib2 has a number of features not found in NuLib:

  • Supports filetype preservation (on systems with "long" filenames).
  • Supports resource forks.
  • Archive integrity test actually does something useful.
  • ShrinkIt-style table of contents listing formats filenames correctly.
  • Support for adding and viewing comments.
  • Allows streaming input (i.e. read from stdin) for most "read-only" operations.
  • Can operate on large archives while using very little memory (with streaming mode).
  • Transparently handles .BXY, .SEA, and .BSE wrappers.
  • Automatically recognizes and handles Binary II archives.
  • EOL conversions (e.g. CRLF to CR) are more automatic.
  • Deals with some Y2K issues.
  • Comes with an auto-configuration script for UNIX-like platforms.
  • Creates "version 3" records, and uses LZW/2.  Most of GS/ShrinkIt's quirks are emulated, so the archives created by NuLib2 are nearly identical to those created by GSHK.
  • Optionally supports gzip-style "deflate" compression and bzip2 BWT (as an extension to the NuFX standard).
  • Source code is cleaner and command-line options are simpler.

All features of the original NuLib are supported, except for a couple of really obscure ones.  NuLib2 is built on top of NufxLib, a NuFX archive manipulation library.

 

Command Overview

Commands are specified like this:

nulib2 -command[modifiers] archive [filename-list]

There are seven commands: add files, list files (two variations), extract files (two variations), delete files, and verify archive integrity.  There are ten modifiers, discussed in later sections.

If you run nulib2 without any arguments, you will be presented with an identification banner and a command summary.  The identification banner describes the current version of the NuLib2 application, and the version of NufxLib upon which it was built.  The latest version of NuLib2 can always be found at http://www.nulib.com/.  If you want a more detailed command summary, use:

nulib2 -h

If you wish to specify modifiers, you may lump them together with the command, like this:

nulib2 -xel archive.shk

or specify each independently, like this:

nulib2 -x -e -l archive.shk

If you want to give your archive a name that starts with a hyphen, you will have to specify it as a full or partial path, e.g. "./-archive.shk".  Otherwise, NuLib2 will think you are trying to specify more modifiers.  One exception to this is that you can specify stdin as the archive for list and extract operations:

nulib2 -v - < archive.shk

Some commands require a list of filenames.  These must be listed after the archive name, e.g.:

nulib2 -x archive.shk foo ack:splat bar

The names given will be compared with records in a case-insensitive fashion, so asking NuLib2 to extract "foo" or "FOO" would match on "foo", "FOO", and "fOo".

The commands are explained next.

 

Listing Archive Contents (-v, -t)

The contents of NuFX archives are listed in a format similar to what ProDOS 8 ShrinkIt 2.x displays.  If you use the command:

nulib2 v archive.shk

you will see output similar to this:

 archive.shk     Created:22-Aug-90 15:33   Mod:22-Aug-90 15:33     Recs:    4

 Name                        Type Auxtyp Archived         Fmat Size Un-Length
-----------------------------------------------------------------------------
 BUG.REPORTS                 TXT  $0003  22-Aug-90 15:33  lz2   32%      3089
 FINDER.DATA                 FND  $0000  22-Aug-90 15:33  unc  100%       458
+HELLO                       BAS  $0801  22-Aug-90 15:33  lz2   86%       605
 GIF.SYS16                   S16+ $0000  22-Aug-90 15:33  lz2   31%     39165
-----------------------------------------------------------------------------
 Uncomp: 4152  Comp: 1988  %of orig: 47%
For each file, the display includes the filename, ProDOS file type, ProDOS "aux" type, the date and time when the file was added to the archive, the compression format, a compression ratio percentage, and the length of the file before it was compressed.  The first line has some information about the archive, and the last line has a summary of file sizes and compression performance.

A "+" in the leftmost column, in front of the filename, indicates that the file is "locked".  NuLib2 considers a file to be locked when it has the ProDOS write, rename, and delete permissions disabled, but still has read permission enabled.  All other files are considered to be unlocked.

A "+" is shown next to the file type if the file is "extended", meaning it has a resource fork.  A "-" next to the file type indicates that it's an empty file stored improperly due to a bug in GSHK.

The listing for a disk image is similar:

 prodisk.shk     Created:21-Apr-90 15:11   Mod:21-Apr-90 15:11     Recs:    1

 Name                        Type Auxtyp Archived         Fmat Size Un-Length
-----------------------------------------------------------------------------
 TEST                        Disk 140k   21-Apr-90 15:11  lz1   37%    143360
-----------------------------------------------------------------------------
 Uncomp: 143360  Comp: 53051  %of orig: 37%
The file type is displayed as "Disk", the "aux" type is the size of the disk image in KBytes, and -- if the image is for a ProDOS disk -- the filename is the disk volume name.

The "Fmat" column will be one of "lz1", "lz2", or "unc", indicating ShrinkIt LZW/1, LZW/2, or uncompressed.  Files added to archives by NuLib2 always use LZW/2 or, if compression fails, are stored uncompressed.  LZW/1 is used by P8 ShrinkIt and the original NuLib.

The filename field will show the full name, including all subdirectories leading up to the file.  If there isn't enough room to display the entire filename, it will be truncated on the left, replaced with two dots.  The following example has five files from the "gno:user:man:man2" directory, one of which was too long to fit in the display:

 Name                        Type Auxtyp Archived         Fmat Size Un-Length
-----------------------------------------------------------------------------
 gno:usr:man:man2:sigblock.2 GWP  $8010  01-Dec-97 00:07  lz2   58%      2667
 gno:usr:man:man2:signal.2   GWP  $8010  01-Dec-97 00:07  lz2   56%      7036
 gno:usr:man:man2:sigpause.2 GWP  $8010  01-Dec-97 00:07  lz2   61%      2358
 ..usr:man:man2:sigsetmask.2 GWP  $8010  01-Dec-97 00:07  lz2   56%      3046
 gno:usr:man:man2:wait.2     GWP  $8010  01-Dec-97 00:07  lz2   51%      6577
-----------------------------------------------------------------------------
You can also list the contents of an archive like this:

nulib2 t archive.shk

The 't' command generates a simple list of filenames:

BUG.REPORTS
FINDER.DATA
HELLO
GIF.SYS16
The filenames are never truncated or embellished, which makes this command useful when you're searching for a specific file.

 

Creating Archives and Adding Files (-a)

You can add files to a new or existing archive with the "-a" command.  If the archive you specify does not exist, a new one will be created.

All files will be added to the end of the archive.  If the name of the file being added matches the name of a file already present in the archive, you will be allowed to replace the existing file or skip adding the new file:

Replace BASIC.System? [y]es, [n]o, [A]ll, [N]one:

If you respond with "y", the existing file will be deleted when the archive is updated.  If you say "n", the new file will be skipped.  Entering "A" or "N" will cause NuLib2 to automatically enter "y" or "n" respectively on any future conflicts.  (You may also hit 'q' here to abort the operation and quit.)

Files in subdirectories will be added with whatever path separator is appropriate for the current system.  On a UNIX-like system that would be '/', while under Win32 it's '\'.  GS/OS follows the Mac OS convention and uses ':'.

There are a number of modifiers that can be used with this command.

-u
Update files.  If a matching file is found in the archive, it will only be replaced if the file on disk is newer.  Files that don't already exist in the archive will be added.  The "Replace...?" dialog will not appear.
-f
Freshen files.  If a matching file is found in the archive, it will only be replaced if the file on disk is newer.  Files that don't already exist in the archive will not be added.  The "Replace...?" dialog will not appear.
-r
Recursively descend into subdirectories.  The standard behavior for NuLib2 is to ignore directories.  When this flag is set, it will add all of the files in a specified subdirectory.  (Note: empty directories will not be added.)
-j
Junk paths.  If you add a file called "foo/bar/myfile", when this flag is set it will be stored simply as "myfile".
-0
Don't compress.  Files will be stored without compression.  (Note that's "dash zero", not the letter 'O'.)
-z
Use "deflate" compression, equivalent to "gzip -9".  Both NufxLib and NuLib2 must be built with zlib support, or the flag will be disabled.  Note that "deflated" files can only be opened by applications based on NufxLib, so don't use this for files that will be opened on an Apple II.  If you specify "-zz", compression equivalent to "bzip2 -8" will be used instead.  (Support for bzip2 compression is disabled by default, so it may not be available.)
-c
Add one-line comments.  NuLib2 will prompt you for a comment on every new file.  Hitting return ends the comment.  (Hint: on most UNIX systems, you can use Ctrl-V Ctrl-M to insert a carriage return.  NuLib2 automatically converts carriage returns in comments to whatever is appropriate for the current system, so you can "sneak" multi-line comments in this way.)  Maximum length is 200 characters.
-k
Store files as disk images.  All files will be processed as ProDOS-ordered disk images.  See Disk Images for more information.
-e
Preserve ProDOS file types.  Files with preservation information will be added with their file type and aux type intact, and resource forks and disk images will be processed correctly.  For this to work, the files must have been extracted with the "-e" flag set.  If you specify "-ee", NuLib2 will attempt to guess the file types of common files by their extensions (e.g. "file.txt" would be stored as type $04 (TXT)).  See the ProDOS Attribute Preservation document for more information.

While adding files, NuLib2 displays the name of the file as it will appear in the archive, along with a completion percentage:

DONE compressing  BASIC.System
DONE storing      BOOT.GSOS
DONE compressing  BOOTU3
DONE storing      COMM/FINDER.DATA
DONE compressing  COMM/ProTERM/PT3.SYSTEM
DONE compressing  COMM/ProTERM/PT3
DONE compressing  COMM/ProTERM/PT3.GLOBAL
DONE storing      COMM/ProTERM/PT3.WELCOME
 15% compressing  COMM/ProTERM/PT3.CODE0

Files stored without compression will say "stored", while files compressed with LZW/2 will say "compressing".  NuLib2 tries to mimic GS/ShrinkIt as closely as possible, so files shorter than 512 bytes are never compressed, and files that don't get smaller are stored uncompressed.

 

Extracting Files (-x, -p)

Files can be extracted with the "-x" and "-p" commands.  You may specify a list of files to extract, or specify none and extract all files.

A simple example that extracts all files from an archive:

nulib2 x archive.shk

If you wanted to extract a file called "fubar" and a file called "ack:splat", you would use:

nulib2 x archive.shk fubar ack:splat

The directory hierarchy is always preserved unless "-j" is set.  In the above example, the files would be extracted as "fubar" in the current directory and "splat" in a subdirectory called "ack" (assuming that the file was archived on a system where ':' separates directory names).

Using the "-r" flag (described below), you could extract all of the files in the "ack" subdirectory, like this:

nulib2 xr archive.shk ack:

The "-r" flag does a prefix string match, meaning it just compares the first part of the name in the archive against the name you specify.  So if you had entered:

nulib2 xr archive.shk ack

above, it would have extracted "acknowledge" and "acksent" as well as "ack:foo" and "ack:bar".  Specifying the filesystem separation character (usually '/' or ':') allows you to grab just the directory you want.

If you try to extract a file with the same name as an existing file, you will be prompted for instructions:

Replace BASIC.System? [y]es, [n]o, [A]ll, [N]one, [r]ename:

You can choose (from left to right) to overwrite the existing file, skip the extraction of this file, overwrite all existing files, never overwrite an existing file, or rename the current file.  If you elect to rename, you will be prompted for a new name for the file.  (You may also hit 'q' here to quit immediately.)

The modifiers usable with the "-x" command are:

-u
Update files.  If a matching file is found on disk, it will only be replaced if the file in the archive is newer.  Files that don't already exist on disk will be extracted.  The "Replace...?" dialog will not appear.
-f
Freshen files.  If a matching file is found on disk, it will only be replaced if the file in the archive is newer.  Files that don't already exist on disk will not be extracted.  The "Replace...?" dialog will not appear.
-r
Recursively describe the set of files to extract.  This allows you to extract an entire subdirectory tree from the archive.  (Perhaps someday NuLib2 will support extraction by regular expression, and this modifier can go away.)
-j
Junk paths.  The pathname is stripped off of the file.  If you extract "foo" and "ack:splat", you will end up with a file called "foo" and a file called "splat" in the current directory.
-l
Convert EOL.  When set, NuLib2 tries to identify which files in the archive are text files and which have binary data.  It then converts the text file end-of-line markers to whatever is appropriate on the current system.  If you specify "-ll", NuLib2 will convert all files except disk images.  High-ASCII files, such as DOS text files and Merlin 8 source files, are automatically stripped.
-c
Display comments while extracting.  If a comment was stored with the record, it will be displayed on the screen.  End-of-line markers (e.g. CR, LF, or CRLF) used in the comment are automatically converted.
-s
Stomp existing files.  When set, NuLib2 will overwrite existing files without prompting you for confirmation.
-e
Preserve ProDOS file types.  Extracted files will have the ProDOS file type and aux type appended to the name (e.g. "foo.txt#040000").  Resource forks and disk images will be labeled.  If you specify "-ee", NuLib2 will append the file's extension to the end of the file, so that systems like Win32 that rely on filename extensions can recognize the file type (e.g. "foo.txt#040000.txt").  See the ProDOS Attribute Preservation document for more information. 

While extracting files, NuLib2 displays progress information:

DONE expanding   BASIC.System
DONE extracting  BOOT.GSOS
DONE expanding   BOOTU3
DONE extracting  COMM/FINDER.DATA
DONE expanding   COMM/ProTERM/PT3.SYSTEM
DONE expanding   COMM/ProTERM/PT3
DONE expanding   COMM/ProTERM/PT3.GLOBAL
DONE extracting  COMM/ProTERM/PT3.WELCOME
 90% expanding   COMM/ProTERM/PT3.CODE0
The filenames shown are the names as they are being written to disk. For example, if you extracted files with "-e" (preserve types) and "-l" (convert end-of-line character) set: 

nulib2 -xle archive.shk

You would see something like:

DONE expanding   BASIC.System#ff2000
DONE extracting  BOOT.GSOS#fc0801
DONE expanding   BOOTU3#060800
DONE extracting  COMM/FINDER.DATA#c90000
DONE expanding   COMM/ProTERM/PT3.SYSTEM#ff2000
DONE expanding   COMM/ProTERM/PT3#ff2000
DONE expanding + COMM/ProTERM/PT3.GLOBAL#040000
DONE extracting+ COMM/ProTERM/PT3.WELCOME#040000
 90% expanding   COMM/ProTERM/PT3.CODE0#060000
The "+" sign indicates that an EOL conversion was performed on the file.

 

The "-p" command extracts the files to a pipe.  The contents of the extracted files are written to stdout.  The normal status messages are suppressed.

Only the "-r" and "-l" modifiers can be used with "-p".

This command is useful for examining the contents of individual files.  For example, the command:

nulib2 -pl archive.shk document.txt | more

will dump the contents of "document.txt", with text automatically converted, and pipe the results into "more".  If you specify more than one file, they will be sent to the output one after the other.

 

Verifying Archive Integrity (-i)

If you want to verify that an archive is undamaged, use this command.  NuLib2 does the same processing that it does when extracting files from the archive, but no output is produced.  Every CRC (Cyclic Redundancy Check - a checksum computed when the data was added to the archive) is tested.

The progress information shown when testing an archive looks like this:

DONE verifying  BASIC.System
DONE verifying  BOOT.GSOS
DONE verifying  BOOTU3
DONE verifying  COMM/FINDER.DATA
DONE verifying  COMM/ProTERM/PT3.SYSTEM
DONE verifying  COMM/ProTERM/PT3
DONE verifying  COMM/ProTERM/PT3.GLOBAL
DONE verifying  COMM/ProTERM/PT3.WELCOME
 90% verifying  COMM/ProTERM/PT3.CODE0

If a problem is found, you will see a message like this one:

  Found a bad CRC in BASIC.System
  Archive may be damaged, continue anyway?  [y]es, [n]o:

If you choose 'y', NuLib2 will pick up where it left off, and continue processing the archive.  If you choose 'n', NuLib2 stops immediately.

If you don't specify a list of files:

nulib2 -i archive.shk

then NuLib2 will test every file in the archive.  If you use a list:

nulib2 -i archive.shk foo ack:splat

it will only test the files you told it to.

You may use the "-r" modifier when specifying files to test, in the same fashion as when extracting files.

 

Deleting Files (-d)

This command deletes entire records from an archive.  A list of filenames must be specified. If you manage to delete all of the files, the archive itself will be removed.

An example of this command:

nulib2 -d archive.shk foo ack:splat

A progress report will be displayed during processing, e.g.

Deleting foo
Deleting bar
Deleting ack/splat

The "-r" modifier may be used to select entire subdirectories.  See the notes in the section on extraction, above.

 

Disk Images

NuLib2 doesn't try to write floppy disk images onto a floppy the way ShrinkIt does.  Instead, it extracts disk images as ProDOS-ordered files, suitable for use with your favorite Apple II emulator.

When adding disk images, NuLib2 assumes that the file is in ProDOS block order.  If you want the archive to work correctly with ShrinkIt on an Apple II, don't add DOS-ordered disk image files.

The NufxLib library comes with a sample program called "imgconv" that can convert between 2IMG (".2mg") images and NuFX (".shk") archives.  It is able to convert DOS-ordered .2mg files to ProDOS order before adding them to the NuFX archive.

NuLib2 will only accept files that are a multiple of 512 bytes.  If you insist on trying to add odd-sized files as disk images, NuLib2 will print a warning and add it as an ordinary file.

 

Binary II Archives

NuLib2 supports Binary II (.BNY, .BQY) archives in a more or less transparent fashion.  For all operations except adding and deleting files you can specify a .BNY file in place of a .SHK and get more or less the same results.

A few modifier flags aren't currently supported for Binary II archives.  You can't specify EOL conversion with "-l", freshen or update with "-f" or "-u", or extract comments (there are no such things) with "-c".  You can, however, specify filenames using "-r" for partial matches, use "-e" and "-ee" to extend filenames, and use "-j" to truncate paths.

The default behavior is to refuse to overwrite existing files.  NuLib2 currently just stops if it encounters a file that already exists.  You can use the "-s" flag to tell NuLib2 to overwrite any existing files.  It will not, however, overwrite directories with files or files with directories.

Files that were compressed with "Squeeze" compression (via BLU v2.27 or SQ3) will be automatically un-squeezed.  If the filenames end in ".QQ", the extension will be stripped.  Programs like ShrinkIt and BLU use the filename embedded within the SQ format as the output name when extracting, but NuLib2 ignores that because it tends to leave you with a filename you weren't expecting.

In some cases you will need to explicitly tell NuLib2 that a file is a Binary II archive with the "-b" flag.  If you want to strip the Binary II header off of a .BXY file, you will need to use "nulib2 -xb file.BXY".  Otherwise, NuLib2 defaults to extracting from the NuFX archive embedded in the BXY file.  You will also need to use the "-b" modifier for a streaming archive (e.g. "nulib2 -xb - < file.BQY"), because the stream can't be rewound after the test for NuFX fails.

 

Miscellaneous

Files that were somehow archived without a filename will be referred to as "UNKNOWN".  Rumor has it some older versions of ShrinkIt omitted the file name for DOS 3.3 disk images.

On Win32 and UNIX-like systems, the ProDOS access permissions are not preserved precisely.  ProDOS has read, write, rename, and delete permission, as well as "backup needed" and "invisible" flags.  If NuLib2 believes the file is locked, it will disable write permission on the file.  When adding files, "locked" files will have read and "backup needed" enabled and the other flags disabled, while "unlocked" flags will have all the flags except "invisible" enabled.  In other words, the basic concepts of "locked" and "unlocked" are preserved, but the full set of flags is not.

The NuFX specification forbids filenames that start with the filename separator character, i.e. you can't put "/foo/bar" in an archive.  If you specify a full pathname, the leading '/' will be dropped.

On Win32 and UNIX-like systems, a leading "./" in the filename is redundant and will be stripped.  If ".." is used as a path component, the pathname will be reduced to just the filename.

When making changes to an archive, a new archive is constructed in a temporary file ("nulibtmpXXXXX") in the current directory.  When everything completes successfully, the temp file is renamed over the original archive.  The exception to this is that newly-created archives are written in place.  If NuLib2 crashes or is killed with signal while modifying an archive, the temp file may be left lying around.

Filenames with invalid characters will be converted to something palatable for the current system (e.g. "/" is set to "_" on UNIX-like systems).  If file type preservation is enabled, the character will be preserved exactly (e.g. '/' becomes "%2f").

NuFX archives store three dates with every file: creation, modification, and when it was archived.  On systems that don't have creation dates, the modification date will be substituted.

There are certain filenames you can't use on a Windows "FAT" filesystem, such as "AUX" and "PRN".  Neither Win98 nor Linux's vfat driver will allow it.  Standard utilities like WinZip fail with a mysterious error message.  As a workaround, the Win32 version of NuLib2 will consistently prefix all MS-DOS device entries with '_', so "AUX" and "aux.foo.txt" will be extracted as "_AUX" and "_aux.foo.txt".  If filetype preservation is enabled, the name used will be "%00AUX" (the %00 is removed when the file is added with -e, thus preserving the original name).  This handling does not apply to versions built for other systems, so attempting to extract a file named "AUX" onto a FAT filesystem under Linux will cause NuLib2 to fail.

No attempt is made to extract and preserve comments, even in "preserve" mode.  This is probably a bug.

Archive files that start with junk -- such as a vestigal MacBinary header or HTTP headers -- appear occasionally on FTP sites.  NuLib2 will search through the first 1024 bytes of the file to find the actual archive start.

Silly benchmark of the day: creating a 14MB archive containing the contents of my hard drive took about 40 minutes on an accelerated IIgs.  NuLib2 accomplished the same feat in about six seconds on a 500MHz Pentium-III running Linux.

 

Acknowledgments

NuLib2 would not have been possible without all the lessons learned from NuLib.  Andy Nicholas, Kent Dickey, Frank Petroski, Robert B. Hess, Bruce Kahn, and Devin Reade contributed code to NuLib's development.

My original plan for preserving ProDOS file types was, in retrospect, mighty screwed up.  My thanks to Bill North for setting me straight.

Eric Shepherd spent a bunch of time messing around with early versions of NuLib2 on BeOS, and helped me get all the configuration stuff in order.

Devin Reade built it on several different platforms, and made a repository for binary distributions.


This document is Copyright 2000-2006 by Andy McFadden.  All Rights Reserved.

The latest version can be found on the NuLib web site at http://www.nulib.com/.