Does somebody just need to buy a lot of hard drives and data tapes, and program a bunch of raspberry pi to download everything it can find?
Edit: What I’m specifically asking about is the feature reddit had to search the site itself. Obviously for reddit this process is much simpler, since they’re just searching their own database.
Is a search engine what is needed? I have some thoughts on how to make an effective one but it won’t be cheap. Essentially drunk from the activity pub firehouse and index everything