Hey...just wanted to say thanks to Rattle and Decius. I personally know that they basically spent *ALL* day on Sunday (in to the wee hours of the morning) coding to fix this problem. I am very impressed at their determination to get this problem resolved before the monday morning "oh god I am at work, guess I will surf the web instead" rush hour. And thanks to Starbucks and Venti sized coffees, which I also saw had a hand in this tour-de-force.

Decius wrote:
This morning a few links popped up on MemeStreams that don't work if you click on them. You get an error message which says "I could not find the proper URL based on the OID. You may have reached this page due to an error."

We realize that this is a serious problem. Unfortunately, it is not a simple to fix and it is going to take time.

When we first started working on this project back in 2001 we were completely green at relational database design, and we made a poor decision, which was to use a feature of Postgresql called "OID" which at the time was defined as globally unique for an entire database. Unfortunately, the Postgres team later decided that this was impossible, and change the definition of OID. In newer versions of the database, which we are running due to numerous performance enhancements, OIDs aren't globally unique, or even unique to a table.

At this point, unfortunately, we had a hell of a lot of code invested in OIDs. We figured it would be a long, long time before OIDs would end up repeating in our database, and so we could wait to change all of that code. Unfortunately, today, OIDs appear to have started repeating in the blog table far in advance of when we thought this was likely, as this table only has 44,000 entries. There are about 500 references to OIDs in our code, and in order to fix this problem, we'll likely have to change them all.

This is going to require time from us, and hopefully patience from you.

RE: The OID Problem

For each week of 2004, we provide a graph of that week's social network activity, the most popular discussions, and the most popular users. Every link between people in the graphs represents at least one blog entry that was read and re-recommended; hence propagated further through the network. The arrows point to sources; they represent the flow of reputation. The users in each graph are those whose memes were re-recommended in the preceding two weeks. Users are colored depending on their popularity that week, with the highest scores getting the brightest colors. Links are only mapped for memes posted on a user's MemeStream. Replies in threads are not graphed.

fiine...see...I memed it...its cool

MemeStreams - Year in Graphs 2004

Rattle's comments:

For several months now, work has been underway building the next version of MemeStreams. It has been necessary to recode most of the site from scratch, so its taking awhile. At any given time the trials of life, lack of funding, bad timing, hardware failure, and general bad luck is screwing up the works. However, we _are_ making progress.

Just before the new year, I hit the point in the development process where new capabilities of mining and graphing social network data were becoming available. I decided to go off on a little tangent, embrace the milestone, and do something to show our technology's progress. The result is the Year in Graphs 2003. Over the course of putting this together I've wound up fixing all kinds of problems with our database conversion code and wrote much of what will become the new graphing engine. I even had a really good "eureka" moment in relation to some of our network theory.. Its been time usefully spent.

That being said, this is all very kludgy. I did not spend that much time crossing I's and dotting T's.. The fonts in the nodes are hard to read, some of the graphs look "squished", its missing the "Show Links" feature the current Social Network portion of the sites has, etc.. There is much room for improvement with our graphing. I look forward to additional feedback.. :)

While these graphs may be fun to look at, the data they are built with is what's really exciting.. The same thing that allowed me to make these graphs is what's going to lead to improvements in the capabilities of the Reputation Agent.

Anyway, I hope you all enjoy browsing through this review of the past year!

MemeStreams - The Year in Graphs 2003

Please read and rerecommend. I want this to get as much coverage on the site as possible, and I want feedback.

Currently the privacy policy says that your reputation tables are private. This reflects the fact that I think what you read ought to be your own business.

However, what you recommend is not exactly the same as what you read, and this is reflected in the reputation data. When you recommend something you are telling the site that you like it. Telling the site implies that you don't mind the site knowing. In fact, you want the site to know. If I do a little digging I can see who you got an article from, directly. This is a bit of an oversite. Something that can be "matured" out of the code. Thats one direction to go in from here. Keep the reputation data as private as possible.

However, if I monitor the site, and I see a certain person recommend an article, and then you rerecommend it, I'll know where you got it from. I do this often enough, in an automated fashion, for long enough, and I'll get a pretty good understanding of who you are reading. The site can't protect you from this. By recommending articles you are making them public. By making them public, you are giving up some of your privacy.

The only way to truely protect the fact that you are reading someone's recommendations is to never recommend their recommendations. They will still show up in the agent, but this information, I think, is private and ought to stay that way.

However, and as I've hopefully illustrated, the recommendations are public in a very real way. What I want to ask the site is if the recommendation DATA ought to be public too.

The reason I want to ask you this is because Rattle has assembled another visualization. This visualization is interactive. You can see a graph of the people who are highly connected to you, click on their names, and see a graph of the people who are highly connected to them, and so on.

Right now this data is amusing but not all that rich. There are about 15 people who regularily post to the site, and they hold all the reputation capital. Everyone's graphs look pretty similar, because we are all really recommending content from the same 15 people, even those of us who don't often recommend content and who aren't often read.

This is going to change.

As this site scales, clusters are going to form. I think the one that currently exists will always exist, but there will be others. People will begin to have different perspectives on the data, based on their interests. Thats what this site is designed to do.

And as those different communities of interest begin to form, the maps of the reputation data rattle is developing will become richer, and you will be able to surf through MemeStreams via the reputation system, hopping from person to person in search of interesting ideas.

I think this is a very compelling feature set, and I want to enable it. But, I've promised to keep your reputation data private. All of it. This promise may not have been very well thought out, but I did make it. So I can only break it with your permission, and that is what I am asking for.

I want to publish your recommendation related reputation data. Your clickthrough related reputation data will stay private. What is published is the number of times you have rerecommended articles from another person on the site, in the form of a graph.

Its similar to the graph on the "Visualizing Memestreams" page, but it is labelled, and interactive.

An important privacy question