Edit File: CRAWL.announce
The TRAVERSAL code from old versions of Lynx has been upgraded by David Mathog (mathog@seqaxp.bio.caltech.edu) so that it works again, can be implemented via a command line switch (-traversal) instead of via a compilation symbol for creating a separate Lynx executable as in those previous versions, and can be used in conjunction with a -crawl switch to make Lynx a front end for a Web Crawler. Usage: lynx [-traversal] [-realm] [-crawl] ["startpage"] Added switches are: -traversal Follow all http links derived from startpage that are on the same server as startpage. If startpage isn't specified then the traversal begins with the default startfile or WWW_HOME. -realm Further restrict http links to ones in the same realm (having a matching base URI) as the startpage (e.g., http://host/~user/ will restrict the traversal to that user's public html tree). -crawl With [-traversal] outputs each unique hypertext page as an lnk###########.dat file in the format specified below. With [-dump] outputs only the startpage, in the same format, to stdout. Note on startpage: If a startpage is specified and contains any uppercase characters, on VMS it should be enclosed in double-quotes. The code that extracts the access and host fields from startpage for comparsions with links to ensure they are not on another server, and the comparisons with already traversed links, are case sensitive, and the startpage will go to all lowercase on VMS if no double-quotes are supplied, such that it might be treated as a new link if encounted with uppercase letters. Files created and/or used with the -traversal switch, based on definitions in userdefs.h: TRAVERSE_FILE (traverse.dat): Contains a list of all URLs that were traversed. Note that if a URL appears in this file it will not be traversed again (important if runs are started and stopped). Placing an entry in this file BEFORE the run will block traversal of that URL. Unlike reject.dat a final * has no effect (see below). Note that Lynx internal client-side image MAP URLs will be included in this file (e.g., LYNXIMGMAP:http://server/foo.html#map1), in addition to the "real" (external) http URLs. TRAVERSE_FOUND_FILE (traverse2.dat): Contains a list of all URLs that were traversed, in the order encountered or re-encountered (but not re-travered) during a traversal run, and the TITLEs of the documents (separated from the URLs by TABs) A URL and TITLE may be present in this list many times. To simplify the list, on VMS use: sort/nodups traverse2.dat;1 ;2 Note that the URLs and TITLEs of the Lynx internal client-side image MAP pseudo-documents will not be included in this file, though "traversed", but only the http URLs and TITLEs derived from the MAP's AREA tag HREFs that were traversed. TRAVERSE_REJECT_FILE (reject.dat): Contains a list of URLs that have been rejected from the traversal. Once a URL has been entered in this list, it will not be traversed. URLs that end in a * will cause rejection of all URLs that match up to the character before the *. So for instance, to reject all htbin references on a site put this line in the reject.dat file BEFORE starting the run: http://www.site.wherever:8000/htbin* TRAVERSE_ERRORS (traverse.errors): A list of links that could not be accessed or had an unknown status returned by the http server. If the owner of the document containing the link is know via a LINK REV="made" HREF="mailto:foo" in it and the MAIL_SYSTEM_ERROR_LOGGING was set true in userdefs.h or lynx.cfg (not recommended!!!), a message about the problem will be mailed to the owner as well. Files created during traversals if the -crawl switch is included with the -traversal switch: lnk########.dat Numbered output files containing the contents of traversed hypertext documents in text format. All hypertext links within the document have been stripped, and the URL and TITLE of the document are recorded as the first two lines, e.g., for the seqaxp.bio.caltech.edu home page the first two lines will be: THE_URL:http://seqaxp.bio.caltech.edu:8000/ THE_TITLE:SAF Web server home page The VMSIndex software is being adapted to use this information to extract the corresponding URL and TITLE for use in indexing the lnk########.dat files, e.g.: $ build_index - /url=(text="THE_URL:") - /topic=(text="THE_TITLE:",EXCLUDE) - /output=INDEX_NAME - lnk*.dat A clever person should be able to figure out a way to index the lnk########.dat files on Unix as well. If you want the hypertext links in the document to be numbered, include the -number_links switch. By default, this will cause the list of References (URLs for the numbered links) to be appended as well. If you want numbered links but not the References list, include the -nolist switch as well. Note that any client-side image MAP pseudo documents that were "traversed" will not have lnk########.dat output files created for them, but output files will be created for "real" documents that were traversed based on the HREFs of the MAP's AREA tags. This functionality is still under development. Feedback and suggestions are welcome.