Decius wrote:
] Sort of, but not exactly. Their bookmark attempts to solve the
] same problem, but it doesn't support threading. I think
] threading is very important. Furthermore, I think the
] reputation system will factor in heavily here.

That's roughly what I meant by MemeStreams being more discussion oriented. They are just tracking links.

We have yet to really even tap the reputation end of things with the thread code..

] This presents an interesting design question. When we pull in
] feeds, ought we to thread them? This may be computationally
] too intensive for us to do.

There are a ton of issues I have not made any conclusions about in regard to the feed stuff.. I'm still in the process of just getting done the "nuts and bolts" code to manage the feeds, pull down and import the content, etc.. I'm also working on a rewrite of the code that generates user blogs & bios at the same time, since it has similar functions taking place.. And for that matter, Public Circles are going to require much of the same framework.. All reasons why I said in that last email that it felt way premature to mention this on the site.. Its all ways off. There is much work to be done before it really starts to get close to hitting the site.. The changes to user blogs will likely make it to the site before the feeds. The ability to have pictures in your bio, and attached to memes, will happen before feeds..

I'm moving along in the direction of treating the feeds in exactly the same mannor as users.. It all really comes down to what the next version of threads and the agent do.. And to another degree, how we wind up handling mutiple links in memes.. The things I'm working on right now are the pretty straight forward parts.. My rough plan was to get the basic feed handling stuff done, and while fine tuning things like the daemon that retrieves feeds, we could work out the strategy for the changes in the next version of the agent, threads, circles, and recommend that incorporate it.. Also, there is a fair amount of stats munging issues that I want to address.. Need time for that too. Also next revision of the social net stuff, which I have roughly planned out.. And the XML-RPC API.. Durring this whole process, we will probably make radical changes to the database structure..

As far as my thoughts on how we should handle the threading goes.. I have many... Most not very developed yet. Read all this as a brain dump.. Feels premature to be discussing it pubicly, but what the hell, it can't hurt. There is much back end stuff to work out before we really get into the interface/usage issues, and all the parts of it that really effect users in the end, are interface and usage issues.

At the top of the thread pages, we can have stats for the URL/articlememe the thread is for.. Stats about what search terms hit the thread hierarchy where it was being discussed.. All kinds of neat popularity/time graph type stuff. Being thats the one place where we are focused on a particular URL, thats the place to really go nuts on any stats we can compile for that URL. Something like when you first click into the thread, you get a nice expanded stats view, links to related threads, controls below it to drill into the thread, then any recent memes in the thread that come from users whom have high rep with you, followed by the most popular memes.. Mostly options and paths to drill in.. Where you came from could effect what you were shown initially.. When you drill in, the stats at the top collapse and/or become specific to how you filtered down.. What options would we want for collapsing/expanding, filtering, and sorting memes? Timeframe, reputation, popularity, etc.. Show feeds, feeds & users, or users.. etc..

This also touches on something else I've been thinking about lately.. When a search engine sends a user to a page, you get two types of info.. The path the user trying to get, and the search terms that brough the user there. I think the mannor in which the search engine indexes the thread content can be engineered to make the most out of these two things.. The goal being to get the specific meme the search result brought up, and what the search terms were.. When the user retrieves the page, we know what search terms they were searching for so we can show the user what they came for (what the path references), along with other memes in the thread that hit the search term, links to related threads that hit the search term, etc.. Give the user, to the best of our ability, not only what they came in search of, but also links to everything else we may have that is what they are looking for. Most web sites ignore the feedback the search engine gives them for anything other then their web stats.. On the website end, very little is every done to help the user find what they are looking for, even though there is information you can use to do that.. I think there is something that can be done there.. Another example of how this idea can be applied.. When a search engine sends a user to a page that is updated regularly for a given search term, and the info related to that search term has rolled off, they are screwed.. Or at least they have to dick with the search engine or go looking around the site to find it.. Its usually not trivial.. When that user comes in, its possible for the page to see what they came in for, and display a link at the top like "show past memes for search term 'whateverterm'", that will pull up memes that were in that topic category, or user blog, that fit the bill.. Non-evil search engine manipulation.. On active sites, search engines usually lag behind the content a fair amount, they should be helped out.. :)

Back to threads.. Since a given entry in a feed is going to have a URL that points to where that entry is at its source weblog, whenever that gets recommended, thats the URL that is going to lead that thread.. Any links contained within, would be in the description of the feed (or non-existant, and the user would have to click thru to get links, or the target _is_ the link). We will be able to parse those, and when they are displayed, insert a (Thread[]) link behind it if its a URL thats in the system somewhere.. And do the same with user blogs.. If a URL isn't in the system, its gets no Thread link, but it still passes thru redirect, so if you get a link from a description, the source user/feed gets rep points if you recommend it.. If someone recommends it, or a feed links it, then it will get a (Thread[]) link.. The Thread link can convey more information.. Like (Thread[#r/#l/#d]) Where r = recommends, l = links within description, d = discussion (replies).. Something like that. [shrug]

Most of the statistic stuff is computationally expensive, mainly on the database.. But its not the type of thing thats expensive every time a user hits a page, thats the stuff that hurts. Doing those queries over and over hurt.. Most of the hard work can take place in the background, much like the social network graphs or the generation of the top level pages.. Those are not trivial to generate, but the daemons just run in the background, updating the stats all the time, and its easy to manage, can be moved to another machine, split between several machines, etc.. Noone cares if the stats lag behind real-time a little bit.. As far as the user specific stuff goes, most of it can be mitigated by some properly applied caching.. I'm not too worried about that.. We can work around those problems.

I'm also excited about the XML-RPC stuff.. Thats the point where we can make some leet blogging tools appear.. I want a recommend editor with inline spel check. An editor where I can highlight a word or phrase, and make it a link by pulling down a menu with my recent links and selecting one. An editor that has inlne spell check.. An editor that would allow you to paste text into a quote.. Oh yeah, and inline spell chek too!

Ok.. Enough brain dumping.. I still have a shitload of work to complete before most of what I rambled about here is really relevant..

] No. Their current events page shows bloggers who have blogged
] stories from major media outlets. Our main page is more like
] blogdex then this.

True.. Our system sees all links as equal. The Man gets no special treatment.

] The key difference between our site and these systems in both
] cases is that MemeStreams has a topic system that allows us to
] organize the content.

Hrm.. While on the topic of topics, "Society/Media" and maybe also "Society/Media/Blogging" might be in order..

RE: Technorati: Current Events, with context