L04 web-based helpers

Note that there is a newer interface, currently still in development, at www.let.rug.nl/p04/.
These helpers are here to make life simpler for semi-expert users of basic command-line L04.

Introduction

L04 is a package that allows analysis on varying dialects, and is written by Peter Kleiweg. It is a collection of command line utilities that takes some reading up to run. This interface - and there is in the works that comes with L04, though its approach is diferent - is meant to simplify using L04, and perhaps clarify it in the process.

These helpers interface works a little differently than the standalone version. It mostly just takes the data and runs the L04 executables, making the work around thit a little simpler. A diagram of the processing currently possible:

Since most paths are fairly simple, I am working on making a few forms that handle making an end product in one step; currently the above is available only in separate steps.

Note that this is currently being developed, so things may break as I work on them. Also, some of L04's programs are missing, and some features of programs that are here. Examples are the G.I.W. based analysis, xstokens' data preparation, difference matrix alteration and mdsplot, but also a few clustering and mapping options. For details, see the overview link below and the one to the manual.

Processing

Note: These scripts are disabled. I have to port them, but am in no hurry yet. If you're interested or were using these, drop me some mail.

Preparation:

...is your job.
The table extractor expects a table in which columns are dialects and each row is a data point (word) for each dialect, but can take some other formats.
This format, as well as the the lists the list analysis uses, are expected to be tab-delimited data, which should be literally what is exported from SPSS, Excel and other spreadsheet-like programs when you tell them to `save as' `tab-delimited text'. (Watch for alterations, though. I noticed Excel likes to add "quotes" around cell text when it sees a space in that cell)

Preprocessing:

Table extraction converts tab-delimited tables to a collection of L04 dialect files (a.k.a. location files) and a label file. This can include translation from other character encodings (Currently a makeshift implementation for IPALA. Other useful transformations may be implemented on request.
tab-delimited data → Archive containing dialect files, label file,

Analysis:

Feature-based levenshtein distance calculation:
- ...for two words showing a lot of details, for insight into the process and models.
- ...for word lists.
  tab-delimited list → tab delimited list with distances
- ...for dialect files (location files).
  Archive with dialect files, label file &rarr difference matrix
(Hierarchical) Clustering on the location distance data.
difference matrix → clusterfile (OR difference matrix)
MultiDimensional Scaling analysis
difference matrix → vector file

Visualisation:

Dendrograms are based on cluster data.
clusterfile → dendrogram image
Mapping of difference matrix, cluster or vector data.
diffmatrix/cluster/3Dvector file + coordinates (+ clipping) (+ map) → map image

Also provided:

Conversion of output forms while not interpreting it can be useful. Other conversions (tree formats?) may be added on request.
difference matrix → more readable HTML table,
difference matrix → graph,
2D MDS data → scatterplot,
PostScript → PNG,
PostScript → PDF
Location collector interface (Gazetteer + Google Maps).
some work → coordinate file

Notes

On the helpers above

The ability to use multiple dialect files (locations) with different weights is not yet implemented - that is, the weights are not added to the dialect files by the extractor. You could alter these yourself, but I need to test just how much change to the archive the code will take before it trips over it because of non-unix newlines.

Also, there are now versions of leven and features that do not use the 0-65536 integer intermediate distances but the floats they represent instead, making some talk about that moot. This interface does not use those versions internally yet, though.

on L04 itself

L04's documentation is technical and leaves a number of things implied, so I've detailed the findings relevant for my personal work. You can skip most of this