It has been a profound few weeks in the Arab world to witness, especially as an Egyptian-American. A couple of us (namely myself and code developer Christopher Morton) have been working relentlessly to get R-Shief's Twitter Analysis onto a stable network environment so we can continue to data mine and aggregate all Twitter posts with the hashtags #Jan25, #Tunisia, #Wikileaks -- among others -- and make them available for future research. Our next steps include (a) to implement semantic content analysis software, and (b) to begin a series of visualizations that succeed at conveying the influence Twitter has had over the recent historical events in the Arab world. I argue that Twitter and its surprising political usages are important. Most recently in Tunisia, Egypt, and other parts of the Arab world, they have created an interplay between the application of structure and resistance that has been transformative. Data Visualizations:
http://twitterminer.r-shief.net (need to add the tweets from aug 2010- feb 2011.)
http://www-958.ibm.com/v/97934 (#syria)
http://www-958.ibm.com/v/94740 (#iwd - international women's day)
http://www-958.ibm.com/v/90653 (#jan25)
http://www-958.ibm.com/v/97946 (#libya)
This digital humanities project began last summer when I was trying to design an information visualization (info vis) of tweets made at the Arab Techie Women's conference in Beirut in May 2010. During the workshop, all 30 participants had agreed upon using the hashtag #ATWomen to be compiled into a visualization of the data. A few weeks after the conference, we tried to capture the Twitter feeds. However, since we were not storing the tweets soon enough, we were unable to retrieve past tweets. (Interestingly, I was in regular email contact about this with Slim Amamou, Tunisian Arab Techie, who was recently was appointed as Secretary of State for Sport and Youth in the new government in Tunisia). Frustrated by the loss of the documentation of our Arab Techie Women's conference, I started researching various ways to capture data from Twitter. I was lucky to be introduced to Christopher Morton over the summer, who kindly developed the code to mine the data you see now: http://rim.r-shief.org. It has been in development since. In this initial interface, our data is searchable by hashtag, language, and range of dates, and outputs a word cloud, language comparison graph, links within tweets, top users/contributors, and all tweets in the query all on one page. We started capturing data on 08-26-2010 and have been pulling new data every 15 minutes. (Twitter allows you to only capture 1,500 tweets at a time).
Last Thursday, Jan 27, R-Shief's internet service provider (ISP) pulled the switch on us because the sudden spike in data from #jan25 was more than their shared server could handle. Bad timing! Luckily, a friend on the East coast who runs his own hosting company, MENA Media man, came to our rescue and kindly offered us server space. We are back online, but again, #jan25 alone is bringing in about 100,000 tweets a day and growing! And Twitter only allows you to pull 1,500 at a time...it fails a lot. Alas, I thought to turn to my department; so finally, the Institute for Multimedia Literacy (IML) at USC is configuring a dedicated server for us right now. One suggestion I was given is to create more than one database, so that the website itself doesn't query an entire 2.5G database with each and every click. Conceptually, I find that interesting as well -- like mirror sites, multiple databases, it makes total sense. One database could even be in xml and just hold the data of any specific query. As DJ Craig D reminded me, it's all about timing when functions are executed, and making sure the data "weight" is distributed.
The week before the protests began in Egypt, I was approached by the Annenberg Innovation Lab here at USC about this project. IBM is donating hardware and their semantic search analytics package —Big Sheets, IBM Content Analytics and Hadoop— to the Lab and they offered for me to be trained on it and use it for the R-Shief Twitter Analysis, which means we'll be able to draw all sorts of new, nuanced meaning. It's experimental. (I would love to hear you thoughts about where to begin conceptually designing this via semantic content. Email.) We begin next week, and I will migrate the data as soon as possible. In the meanwhile, history is being made and we urgently have to set up the site on the dedicated server so the system can handle it, which will be Christopher's labor at this point. These things take a moment to do properly.
One thing I have learned is that it has taken us many months just to create a stable and scalable database with limited resources. Though we are not there yet, it is just a matter of days now. Once that is done, I can begin experimenting with creating various renderings of the data. Per my original design concept, these tweets will ultimately be understood through a series of informational and artistic visualizations -- creative tools (using Processing and/or Flash) that capture the special something that makes Twitter (and other social media sites) so feared that a government would shut down Internet to an entire nation during civil uprising and protest! Many media scholars and journalists write about the vacuousness of Twitter. They make arguments that social media is not effecting change. Even if I do not agree, I can understand how they might have arrived to their analyses. One of the challenges to rigorous analysis of Twitter is that this system of 140 characters at a time operates within, not one, but a nexus of paradigms that foster entirely different epistemologies around what is social (and pre-social), agents/actions, space/time, to name a few. What I mean by that, briefly, is that one could argue that Twitter operates based on principles of uncertainty, where there are no groups, only formations of groups, and where non-linear time and space still create narratives and meaning, and where objects (such as Twitter) have agency in a social network. I think this it quite apt to examine the current situation of civil uprisings throughout the Arab world via a theoretical model of uncertainty -- it is subversive, open-ended, and offers vast potentialities.
As I tweeted yesterday, if you had told me 2 years ago that I'd one day watch governments fall 140 characters at a time, I'd have called you crazy. Is that really what happened? Or is Twitter just a part of a larger paradigmatic shift, taking hold in younger generations. I do not question its truth claim: it does not matter. What I think is significant is that the rhetoric has been that the protestors are the youth of the Arab world, the young people. Indeed, over 50% of Egypt's population is under the age of 40. Are we seeing an epistemological conflict in this collective manifestation of protest and civil uprisings all over Tunisia, Egypt, Yemen, Jordan, around many of its embassies throughout the world, in virtual communities on Facebook and other media platforms? I think it is important to stay clear from essentializing. Social media in the Arab world is unique -- both in terms of how the social is operating, tight-knightly woven social fabric; and also the history behind media in the Arab world that was born in print form as an apparatus of the State under the Ottoman empire. Where media acts almost like the 4th branch of government in the US, and a fundamental right to ensure the power of the government remains under checks and balances, media in the Middle East functions quite differently. And so when, in Egypt, media became actively dependent on the social fabric, rather than institutional sources of information and analysis, that opened up an uncertain bag of worms for the nation. The Egyptian Revolution -- which side are you on -- that of security or of uncertainty?
I have only begun to think through how to use Twitter as a research tool, specifically for the field of Middle Eastern Studies, but I do have a couple ideas percolating that involve providing interactive projections available online to be used during lectures and events. More details will follow. First thing first, though, I must get this project on a server where we can finally let it run automatically. Thank you.
--