bigQuestions.com
http://no-org.net/data/reference/art/bigquestions.php
"bigQuestions.com" is a data-mining project which mines the web and approximately 1000 world-wide library catalogues for data pertaining to some of the "big questions" of philosophy, religion and (recently) science. The result is a searchable database. I am interested in how we contextualize these concepts, both culturally and linguistically. The most interesting (and often amusing way) to proceed is not to search for the "big questions" but for some other terms of interest. This narrowing of focus gives us a kind of slice of life and reveals the ways in which those things which interest us are embedded in the universe of "big questions".
In addition to allowing for searches, bigQuestions.com also has a real time display, where the visitor can look into a small window of time during the data mining process. The real time display also allows the visitor to step through the lines of the data-mining program and to watch how the data is collected.
I have developed all of the software for this project, both the scripts which mine the web and the scripts which search library catalogues. Library searches are done using the Z39.50 information protocol. I am myself a contributor to the perl Net::Z3950 project and have developed a module which does asynchronous queries to large numbers of libraries; it's the module I use when mining library databases. On the browser side, I use DHTML and communicate with the server by means of perl scripts.