Telling computer science from oncology by its vocabulary

The problem

German universities publish doctoral and master's theses openly, but each repository exposes them differently and none of them is organised around the question “what computer science came out of here”.

The decision

Everything comes from interfaces the repositories publish for machines rather than from scraping rendered pages, which is both more polite and more stable. Abstracts missing from those records are backfilled from a second source, and full texts are pulled where they are served.

The part that took thought

Repositories mix every discipline, and the vocabulary does not separate cleanly. Algorithm, network and model appear about as often in medicine as in computer science.

So terms only count when corroborated by other computing context. A single keyword hit is not a match. That is what keeps an oncology thesis about network models out of a computer science dataset, and it was worth more to the final quality than any amount of tuning the fetch layer.

The result

11,675 files across three batches of universities, with metadata and abstracts for everything and full text wherever it was available.