April 25, 2003
Log harvesting
Via Big Pink Cookie
The data from the logs can most certainly be analyed to "tell" what's going on. Combine it with a bit of indexing and spidering of other sites and you can most certainly tell who's reading what. There don't appear to be many people doing this... yet.
You post an article, that's one link. Pings are made, making new links. Timestamps exist for them so we've got a temporal fix as well. Do some geo-ip lookups and you can tell location. Now, spider the referral links your site provides and you can see who else linked into it. Ditto on comments and trackbacks. Extend that out to the linked sites. A very big picture starts to emerge.
Now, what does that picture mean? It would depend on the answer you want. On one level some idiots will try to take it out of context to support some harebrained perspective they espouse. RSS/XML has seen one particular vendor do this time and again.
So the question becomes will people refrain from exposing data because of what the idiots will do with it? I sure hope not. Because once the data is out there it becomes possible to build a /true/ big picture. Right now we're at the mercy of a things taking only short-sighted approaches based on short-term data. As more stuff comes online it becomes possible to build bigger pictures. As a result the short-sighted perspectives are exposed for the junk they espouse. But unless the good data gets shared the bad data will outnumber it.
Yes, this is a terrifyingly risky process. To share ones use of the net and links to data poses all sorts of exposure risks. As more people expose data the one's that don't will become obvious. Turning things against them will be a lot easier.
It's sort of like income tax records. They're all public. I can find out what you made last year simply by following the law and asking for it. We do this to politicians as a way to expose hidden agendas. Those that don't expose this info are made to look suspicious; what're they hiding?
This will certainly get worse before it gets better.
