It's useful to filter out \"conversational\" tweets and look for tweets with links to another page or picture, etc. <\/p>\n
We create a view that only map tweets with link entities.<\/p>\n
\r\nimport couchdb\r\nfrom couchdb.design import ViewDefinition\r\nimport sys\r\n\r\ndef url_tweets_by_created_at(doc):\r\n if doc.get('created_at'):\r\n _date = doc['created_at']\r\n else:\r\n _date = 0 # Jan 1 1970\r\n\r\n if doc.get('entities') and doc['entities'].get('urls') \r\n and len(doc['entities']['urls']):\r\n if doc.get('user'):\r\n yield (_date, doc)\r\n\r\nview = ViewDefinition('index', 'daily_url_tweets', \r\n url_tweets_by_created_at, language='python')\r\nview.sync(db)\r\n<\/pre>\nNext we create an app that reads from this view and displays the results.<\/p>\n
\r\nimport couchdb\r\nfrom datetime import datetime\r\n\r\ndef run(db, date, limit=10):\r\n """Query a couchdb view for tweets. Sort in memory by follower count.\r\n Return the top 10 tweeters and their tweets"""\r\n print "Finding top %d tweeters"%limit\r\n\r\n dt = datetime.strptime(date,"%Y-%m-%d")\r\n stime=int(time.mktime(dt.timetuple()))\r\n etime=stime+86400-1\r\n tweeters = {}\r\n tweets = {}\r\n # get screen_name, follower_counts and tweet ids for looking up later\r\n for row in db.view('index\/daily_url_tweets', startkey=stime, endkey=etime):\r\n status = row.value\r\n screen_name = status['user']['screen_name']\r\n followers_count = status['user']['followers_count']\r\n tweeters[screen_name] = int(followers_count)\r\n if not tweets.has_key(screen_name):\r\n tweets[screen_name] = []\r\n tweets[screen_name].append(status['id_str'])\r\n\r\n # sort\r\n print len(tweeters.keys())\r\n di = tweeters.items()\r\n di.sort(key=lambda x: x[1], reverse=True)\r\n out = {}\r\n for i in range(limit):\r\n screen_name = di[i][0]\r\n followers_count = di[i][1]\r\n out[screen_name] = {}\r\n out[screen_name]['follower_count'] = followers_count\r\n out[screen_name]['tweets'] = {}\r\n # print i,screen_name,followers_count\r\n for tweetid in tweets[screen_name]:\r\n status = db[tweetid]\r\n text = status['orig_text']\r\n # print tweetid,orig_text\r\n urls = status['entities']['urls']\r\n #name = status['user']['name']\r\n for url in urls:\r\n text = text.replace(url['url'],url['expanded_url'])\r\n out[screen_name]['tweets'][tweetid] = text\r\n\r\n return out\r\n\r\nserver = couchdb.Server('http:\/\/localhost:5984')\r\ndb = server[dbname]\r\ndate = '2012-03-05'\r\noutput = run(db, date)\r\n<\/pre>\nFind the complete codebase on github at: https:\/\/github.com\/telvis07\/twitter_mining<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"It’s useful to filter out “conversational” tweets and look for tweets with links to another page or picture, etc. We create a view that only map tweets with link entities. import couchdb from couchdb.design import ViewDefinition import sys def url_tweets_by_created_at(doc): if doc.get(‘created_at’): _date = doc[’created_at’] else: _date = 0 # Jan 1 1970 if doc.get(‘entities’) … Continue reading twitter mining: top tweets with links<\/span>