Tweet Dreams: A four-week survey of Twitter dreams

Twitter is a very simple human noise and data storage cloud updated in real time from around the globe. As such it is possible to scrape this data cloud for trending topics including those topics people report in their dreams. By filtering Twitter's data cloud I performed a four-week informal survey of dream trends to see if there were any significant patterns worth following. While there are some obvious problems with the reliability of the data cloud and the survey methodology, the initial data indicates that trending patterns do exist, and that human dream content is susceptible to popular trends like any other form of social media. Results which indicate that dreaming is a social collective exercise may be pertinent to ongoing dream and psi research. Also, the viability of dream and psi research conducted through the monitoring of social media content should be further explored.

Methodology

The Twitter data cloud can be scraped in real-time in many ways. The methods I used for the first phase of this survey included third-party tools such as TweetDeck and monitter.com to watch live results for people on the compound search term "had a dream" OR dreamt OR dreamed. This filter generates spontaneous results on the order of 2-3 Tweets Per Minute (TPM) or dozens to hundreds of Tweets Per Hour (TPH) at peak usage. These results were scanned over a period of two-hours each day for salient material and trending topics were recorded daily. This methodology was carried out for the first two weeks.

When the growing unreliability of the Twitter service became a problem for real-time monitoring, the second phase of the survey was done through scraping search filters at search.twitter.com as well as through third party sites like Twitscoop and Twist. Trends noticed were recorded on a daily basis.

The third and final data analysis phase was set up an automated service to scrape the twitter RSS feed for my search string every few minutes and dump the results into my web-based news aggregator at Supermassive.com. With this third phase I could get round-the-clock updates and then create a tag cloud of dream topics by day or week, instantly ranking dream trends with trends in current news. While this methodology is not optimal is does create the initial framework for a more sophisticated a system of monitoring trending dream content across a wide population in near real-time.

Findings: Phase 1

The initial results of my human eyeball survey were more amusing than surprising. The first trend that stuck out at me was how many people dreamt about their iPhones, or dreamt of wanting iPhones, or dreamt about dropping or cracking or losing their iPhones. This dream-gadget fetishism was probably the first trend I noticed, and is no doubt indicative of the technophile demographic that uses Twitter on a daily basis. The second trend I noticed was how many people report dreaming about tweeting or being on Twitter. Although this trend was considered of secondary interest, it is probably the most popular dream trend on Twitter. In fact, after a week of visually following dreams on Twitter I found myself beginning to dream in scrolling snippets of text. This trend is obviously an artifact of adapting to communication through the Twitter service. Dreaming of twittering and then waking up and tweeting about it seems to be a consistent trend not susceptible to cycles.

The second thing I found that I did not expect was how many people were dreaming about zombies, or a zombie apocalypse, or being chased by zombies. For a few days this trend was so significant I started re-tweeting the more amusing ones at a rate of three or more a day. Then mention of zombie dreams died out for a few days, and then they started up again. I found that the zombie trend is cyclical and that it waxes and wanes over time. Trends which revolve around TV shows also follow this cycle, like dreaming of American Idol or Lost the night after it airs. Media driven trends are cyclical and easy to spot, like seeing a spike in dreams about Seth Rogen or Paul Rudd during the weeks that their movies opened across the country.

In addition to trends like iPhones, zombies, and dreams about celebrities, I also noticed consistent patterns of classic dream scenarios, like having teeth fall out, dreaming of waking up but still being asleep and dreaming, or dreaming of being in an argument with a friend or family member. Other trends that I noticed were dreams of eating or being tempted by junk food. I did not expect this particular trend, but feel that it is very telling about the modern psyche. Other consistent dream trends include dreaming of friends in romantic ways, dreaming of friends in non-romantic ways, dreaming of celebrities, dreaming of Obama, dreaming of snakes, dreaming of spiders, and so on.

Beyond topical trends I also noticed recurring psi-related dreams that caught my eye. The classic psi dream is dreaming about someone you haven't heard from in many years, and then the next morning you wake up and find they have called or e-mailed or contacted you on Facebook. This psi dream is reported on Twitter with the frequency of roughly one to two times a week. Those who study psi may be interested in following up on this trend to see how often people dream of each other before subsequently seeking to connect through social media. If this trend is significant it could prove that dreaming is a direct link between social networking and neural networking, and is sometimes presented in the form of psi-like precognition or simultaneous cognition that leads two spatially removed individuals to re-connect in virtual space.

The other psi-related phenomenon I noticed was the number of people who reported dreaming about earthquakes on March 30th, 2009, the day of the small earthquake that hit the San Francisco area. Earthquakes are a low-frequency recurring disaster dream, but reporting of them spiked when they coincided with an actual earthquake at the epicenter of the tech community. Precognition of natural disasters is a commonly reported psi event, further examination of the Twitter cloud and other social data sources may yield similar findings.

Findings: Phase 2

After two weeks of casual eyeball survey it became clear that dream trends did exist, but following the data in real-time was becoming more difficult due to personal time constraints and the slowness and unreliability of the Twitter service. During this phase I would cherry pick search tags that people were dreaming about and pop them into search.twitter.com or similar service to see if the dream topic was consistent, cyclical, rare, or unique. For instance, I found that people on Twitter dream of iPhones with higher frequency than zombies, and both compete with the Jonas Brothers on the same levels of trendiness. But @donniewahlberg is still the trendiest celebrity dream guest star I could find.

Findings: Phase 3

The growing unreliability of the Twitter service soon made even casual search scanning problematic, so a more robust solution was employed for the final stage of data gathering. Using a modified MagpieRSS aggregator at Supermassive News I was able to poll the Twitter cloud via RSS every eight minutes for updates on dream related content and stick the results under a Dream heading for easy browsing. Once the RSS data had been parsed into the Supermassive database it was then tagged by the Yahoo API news tag generator and displayed as a sortable Dream Tag Cloud via a modified TagSoup PHP script.

The results of the dream cloud are spotty for a variety of reasons which will be discussed, but despite the problems trend tags can still be found and analyzed. The first problem with the dream cloud is that trend keywords are sorted by weight against all the other news headlines in the Supermassive database, which means if Miley Cyrus is in the headlines and one person dreams about her, that trend will be instantly larger than all other dream trends. This can be remedied in a later phase. The second problem with the dream cloud is that it can spot keywords like "Zac Efron" but misses twit-speak words like "zefron" or "@zachefron". If the Twitter cloud is to be analyzed for data then a more robust tagging tool must be employed to allow for esoteric syntax. The third problem with the dream cloud is that it cannot filter out noise created by non-dream chit-chat and re-tweets about the same dream, thus artificially inflating some terms. The fourth problem is that some dreams will get dropped due to lag or poor response from the Twitter service, something that can be remedied by more robust scraping. The last problem is that the cloud cannot sort dreams into categories like a human reader would, putting all dreams about pregnancy and babies and giving birth under one tag as opposed to splitting all dreams of pregnancy and birth and babies into separate syntactical tags.

Despite the problems with the dream cloud I could still see after two days of operation that the initial trends I spotted in my earlier casual surveys were indeed actual trends. The dream cloud picks up an average of 700 dreams a day, and after two days of operation the tags iPhone and Twitter and Zombie(s) were found prominently in the cloud. But then, after the third day, these topics dropped in prominence while ambiguous topics like cheese rose in rank, demonstrating that there are cyclical trends in the number of people dreaming of and reporting specific cultural topics. The data in this cloud recycles after seven days, meaning that the cloud will max out at around 5000 dreams per week. This is a small pool of data to analyze and there is still a significant noise factor, but for the short term this dream cloud can function as a limited prototype for a more sophisticated real-time dream trend monitoring tool.

Analysis

Although the Twitter cloud has the advantage of being a direct source of easily searchable material pertaining to dreams, it also has some disadvantages. The first disadvantage is that the data is skewed to the demographic of the average Twitter user, hence the emergence of trends like Twitter and iPhones and Blackberries and videogames in people's dreams. Also, since Twitter is a technophile social networking site it is probable that people are more likely to tweet about dreams that include Twitter or iPhones or videogames or other Twitter users because the content is more interesting to that social group. It is impossible to say how or why a Twitter user edits his or herself while sharing dream content, but it is reasonable to assume that most people who tweet their dreams do so because they are particularly memorable, salient, or potentially interesting to others in their social network. In this sense, the sharing of dream content with a social network is a way of demonstrating trust and emotional integration with the larger interests of that group.

The fact that the Twitter demographic is skewed allows us to spot dream trends particular to this group -- like Twitter and iPhones -- but the Twitter pool is also large enough to reveal trends in the larger realm of media culture. Dream trends like zombies or particular celebrities like Seth Rogen can be tracked over time to demonstrate that the more these archetypes appear in popular media the more they will appear in dreams. Thus, dream content is demonstrably a social media that is susceptible to popular trends in culture and topical trends in personal social networks. While this notion has been generally accepted to be true from an anecdotal perspective, searchable social media data clouds like Twitter give us the first possibility to investigate the nature and reliability of trending dream content over time.

Ongoing Research

Bulk surveys of user-generated content for research purposes are problematic in many ways. Now that the cat is out of the bag Twitter users may be alerted to the fact that their dream content is being monitored for research purposes and adjust their reporting habits. This self-awareness will undoubtedly throw a new set of data problems into the mix. Users may excitedly begin posting about every dream in the hopes that they are furthering research, others may stop tweeting their dreams because of privacy issues. Alternately, mischievous users may take a dream topic and have all their friends tweet about that topic at the same time to skew the cloud results. Spammers may employ a similar technique of posting multiple dreams about their brand in the hopes of pushing their brand name to the top of the cloud.

Because of all the various ways the Twitter data cloud can be manipulated it is reasonable to assume that any ongoing research in this area must employ human editors to monitor the dream feed for noise and irrelevant data. Future research will rely on the use of logic which can filter out irrelevant data and sort relevant data by both keyword and category headings for more sophisticated trend monitoring and tracking over time.