Freesite Insertion Wizard
=========================

This program is GPLed. See GPL.txt for details.

System requirements:
--------------------

a Java runtime environment 1.3.1 or higher.
A running Freenet node.

Start fiw.jar by double clicking it or by CHANGING INTO ITS DIRECTORY
and running

java -jar fiw.jar

in the console. You need write access to the FIW (==your current)
directory (for saving config and log files).

for the commandline help, type

java -jar fiw.jar help 

(==fiw help)

Contact me
----------

you can contact me at <schierlm@users.sourceforge.net>, or on IIP. My nick
there is mihi and I am in channel #fiw.
You can also write to the fiwiz-devl mailing list. For more
information visit the FIW project page at http://fiwiz.sf.net/

Building
--------

Extract src.zip into the same directory where the rest is (including
pathnames) and run "ant" from there. You can get ant from 
<http://ant.apache.org/>. If you run "ant dist", you will get a "dist"
subdirectory that looks like what you have in fiw.zip.

Updating from 0.05 or earlier
-----------------------------

these notes have been moved to the end of the file.


Changelog
---------

++ 2004-01-17 Version 0.08 released ++

- allow giving time (not only date) in MetaDataBrowser for DBR slot
  selection 
- show FCP host and port in report files and load status window.
- exclude thumbs.db, desktop.ini, CVS etc. from inserts
- internationalization
  * Spanish version by "Alejandro R. Mosteo Chagoyen" <public@mosteo.com>
  * German version by me
- rewrite of FCP layer
  * test case for the FCP layer
  * clean handling of multiple inserts into multiple Fred instances
    from one FIW instance.
- added metadata servlet to contrib
- bugfixes
  * set all dates in container .zip files to unix epoch
  * allow slashes in keydir (without crashing the logger)
  * restored captions of slot selectors (insert panel).
  * Advertize advanced settings in Cli mode properly
  * InsertJobs have been unregistered twice sometimes
  * InsertJobs have not been unregistered at all sometimes
  * increase SocketTimeout (to infinity) for FEC encoding
  * fixed stupid bug (NullPointerException) in HTML parser
  * do not parse files when retrying an insert
- Code cleanup
  * remove dead code
  * Refactored code out of GUI classes
  * new event interface for the Insert GUI (to allow better GUIs).
  * javadoc improvements

++ 2003-08-22 Version 0.07 released ++

- added support for "SSK entropy" (with build 5017 and above)
- fixed a bug in the metadata browser when viewing really long
  metadata
- added support for S.C.U.M.'s page tags (see TFEE for details)
- you can "lock" the insert date used by the parser (for reinsertion).
- added an interactive command line interface - this is quite stupid,
  but at least it works and it looks similar to the FIW process (if
  you are on linux: FIW gui is "make menuconfig/xconfig", FIW cli is
  "make config").
- retry insert on unexpected errors instead of stopping it
- added a configurable SocketTimeout (default 5 minutes)
- invalid config file settings are printed to stdout at startup
- a new button to "clone" a project
- new InsertEngine:
  * Insert is split into "equal" jobs first which are inserted
    afterwards. This makes the new engine more flexible.
  * Uses thread pooling and dynamic reordering of jobs now
  * You can give priorities for files
  * File inserts after mapfile insert (for FEC chunks, too!)
  * Compares data property when verifying or checking for key
    collisions
  * Container support
  * Fixed bug with NIM auto-advance with small NIMs (it did advance
    even if the NIM was identical)
  * Verify now shows both insert count and verify count
  * NIM checks use the new job system as well
  * FIW does no longer crash when inserting "completely empty" keys
  * URI errors (created by SSK entropy or SVK keys) are detected
    properly now.
  * Supports to add arbitrary metadata to CHK@ "file" keys - used for
    NIMs and for containers.
  * Parallel insert of metadata and file for NIM inserts.
- other code cleanup:
  * Moved the command line insert code from Main to CliInsert
  * Factored out project and tuning options to fiw.core.option 
    classes - used by both GUI and CLI clients.
  * reordered the wizard steps a bit and fixed going back from select
    panel (it might need to go to tuning panel)
  * javadoc improvements

++ 2003-05-27 FIW CVS hosted at sourceforge.net ++

- split FIW source into packages and made ant build script
- fixed a bug in handling of multisegment split files
- fixed a GUI bug with setting "next free nim" text box uneditable

++ 2003-03-23 Version 0.06 released ++

- FIW remains open when an insert is started, so you can start another
  one while the first one is running - or after it has finished.
- On abort of an insert it is tried to be shut down cleanly (may take
  a few seconds).
- fixed dozens (yes, really!) of small bugs and rewrote some "ugly"
  parts of the code.
- addded an option for one-shot sites for all the users that can't
  read descriptive text.
- does not use any __ files any more by default.
- added/removed/resorted advanced options and changed some defaults
- no private SSKs are stored in logfiles any more
- added a new parser for HTML sites:
  * is controlled completely by in-band-signalling (so you do not need
    to edit your source code to take advantage of it, any HTML program
    does)
  * can parse multiple files - the results will be placed in
    .fiw/parsed
  * has NIM forms and other templates available
  * can include files, dates and URL content
  * you can make custom vars and insert them (incremented or
    decremented as well) into your site
  * support for charsets other than ISO-8859-1 (specifies the charset
    in the metadata as well)
- NIM support:
  * fields in the project settings to specify nim prefix and number
  * parser can insert NIM forms
  * nim checker that tells you which new NIMs you have got
  * optionally: next free nim number is adjusted at each insert to
    match the number that failed in the nim checker.
  * NIM/file inserter can use next slot if a slot is used by another
    NIM.
  * NIM/file inserter inserts small files/NIMs as one key now.
- added a Metadata browser and a Freenet Load Status window (with
  graph) as extra modules (and added batch files for them for the
  windoze users)
- improved insert:
  * if you have them, you can load balance inserts to more than one
    Freenet node
  * character set support for HTML files (charset is given in mapfile
    so that it is not filtered by the anonymity filter)
  * FEC blocks are not generated/inserted with test inserts by
    default.
  * you can stop insert after building FEC blocks (as you do not need
    to be online for that)
  * you can clear the progress file before inserting (when reinserting
    really old projects or for test inserts after you wiped your
    datastore)
  * now detects for non CHK keys if they collide with different
    content or with the same (different content creates an error)
  * test insert mapfiles that collided are inserted again as CHK@.
  * Checks Unreachable/Restarted/Rejected params of RouteNotFound and
    stops insert faster if requests do not make it off your node.
  * improved both the report contents and the file names (they contain
    the key dir now)
  * retries DNFs once now instead of giving up after the first failure
- options regarding FEC:
  * FEC minimal size is 1 MiB now (since Freenet 0.5.1 will cache 1
    MiB files regardless of store size)
  * TXT and HTML files are not FEC encoded by default
  * you can use "usefec.filename=true" or "usefec.filename=false" in
    .fiw/fec.ini to override file size and extension for selecting
    FECed files.
  * FEC encoder is now treated as singleton (only one file encoded at
    the same time) regardless how many projects are running.

++ 2003-02-05 Version 0.05 released ++

- now uses the last "." of a filename to determine extension
- fixed some small bugs and the numbering of mapfiles in the report
- made the error message that occurs when mimetpe.conf is not found
  more intuitive.
- added a button to start another upload while one is running (creates
  a new instance of FIW)
- filenames starting with a dot are ignored now.
- FIW caches which files it already inserted and inserts only the
  other ones - unless verify fails on them.
- added icon
- added a hack to parse more than just index.html with the old parser
  (undocumented, see the source of OldHTMLUtils.java#doHeaderAdd)
- gj changed FEC spec, so i hacked my fec inserter (after flaming
  Matthew - see devil archive)

++ 2003-01-03: Version 0.04b released ++

- fixed a small bug when you rename a file that ir already  FEC metadata
  calculated for 
- fixed a severe bug when Ignore local datastore is off.
- fixed a more severe bug when creating splitfiles for large files.
- added a workaround for some OS/JRE combinations that have a bug in
  their TextArea class

++ 2002-12-29: Version 0.04 released ++

- added rudimentary commandline support
- added FEC support
- added support for indirect mapfiles
- separated GUI and core code better
- improved the GUI of the insert window
- improved the layout of the logfile
- General checks are now written to report file
- solved problem with GenerateSSKPair and improved its logging
  features
- now you can insert the mapfile(s) before the files if you really
  want to (not when there are FEC files!)
- now you can decide if you want to ignore the local datastore (as in
  Version 0.02 and 0.03) or not (as in 0.01)

++ 2002-12-02: Version 0.03 released ++

- some minor changes
- insertion report is saved as a file
- improved insertion process:
  * inserting dbr slots and retrieving is now threaded.
  * retrieving is done with a higher HTL than inserting. (customizable
    increment)  
  * both insert and verify show both CHK@ and filename in report file
  * Files are verified directy after inserting
  * Error in inserting or DNF on verify repeats insert (up to 5 times) 
    (customizable) 
  * RNF on verify repeats retrieve (up to 5 times) (customizable)
  * All files are verified after successful insert; failed retrieves
    cause the insert to restart from the beginning
- new step for setting up project independent settings (e. g. # of
  threads, retry counts)
- subdirectory support (files/directories starting with __ are
  excluded from inserts now) 
- support for inserting NIMs/single files
- unknown mime types are inserted as "application/octet-stream"
- made it easier to use the same SSK for more than 1 project
- you can delete projects - or move them up and down
- negative edition references (e. g. references to last edition)
  possible now 

+++ 2002-11-12: Version 0.02 released +++

- some minor changes
- reduced numer of ActiveLinks in default __header.html to 3
- new way for Edition header generation
- added support for DBR sites
- default insert HTL for new projects is now 15.
- longer MIME type list inside the jar, you can customize it outside
- added check feature
- "update existing project" now default (as you need it most of the time)
- using .next link to next edition
- failed mapfile inserts for test inserts insert the mapfile as chk now
  (so that you can test an edition more often than once)
- you can use KSKs now for Edition freesites (although it is nonsense)
- you can create one shot freesites by specifying Edition number zero,
  even into KSK and CHK keys.
- files are numbered from 1 on instead from 0 on in the log.
- now closes files directly after inserting them.
- error messages in last step remain and do not disappear after a few ms.
- subdirs in homepage dir create a warning (no error any more) and are 
  ignored.
- added a comment to top of each java file (as all want to read the code...)
- added support for external files in your map file (__mapfile
  support)
- Local node's data is ignored when inserting with HTL>0
- you can fetch every key after inserting it - a useful test
- changed metadata modifications so that they fit to latest metadata
  specification.

+++ 2002-11-03: Version 0.01 released +++


Updating from 0.05 or earlier
-----------------------------

FIW 0.06 is not 100% backward compatible to FIW 0.05 - so you should
make sure that everything still works (do test inserts after you
checked these points):

- DO NOT CLOSE FIW MAIN WINDOW WHILE INSERTS ARE RUNNING - in that
  case you will stop them all.
- FEC minimal size is 1 MiB now; and HTML and TXT files are no longer
  FEC encoded - so it might cause problems when refreshing one shot
  sites with files between 0.5 and 1 MiB. 
- insert mapfile before the files has been removed due to problems
  with it. So you cannot use that option any more - however, progress
  files should help you inserting larger sites without this option.
  If you *really* need it, tell me - perhaps I'll include it again in
  0.07 then.
- the new parser does not use __index.html any more. to "convert" your
  old projects, do an insert with both "Test insert" and "Stop after
  creating FEC blocks" checked (ignore the warning) and delete __index.html
  afterwards. The new "base file" is index.html then. 
- go through the settings (especially the new "Parser settings" tab)
  and set up the parser as you like it. Then, check it if it really
  does what the old one did. 
- now files are no longer stored in the __fiw folder, but in a .fiw
  folder by default. For old projects (that already had a __fiw
  folder) the old folder is still used. If you do not want to go back,
  you can rename it.
- if you had a __mapfile file, move it to .fiw/mapfile.ini.
- if you use a DBR site whose DBR has been created by another program,
  you might get an error that the DBR settings differ from an earlier
  insert. You can disable that error by putting the line
     tuning.nodbrmismatch=true
  into your fiw.conf.
- check advanced options - some params were added and some default
  values changed.


NOTES:
- FIW checks now if colliding mapfiles are identical. So when
  refreshing a one shot site fails, you might get an error. You did
  not get this error in 0.05, even if the refreshe failed there as
  well. So when you get this error, reinsert the site with the tool
  you used originally to insert it, or insert it to a new URI.