Advertisements
jump to navigation

Hacking Team boss: we sold to Ethiopia but ‘we’re the good guys’ July 14, 2015

Posted by OromianEconomist in Censorship.
Tags: , , , , ,
1 comment so far

???????????Hacking team hacked1Hacking team hacked

The Guardian

Hacking Team boss: we sold to Ethiopia but ‘we’re the good guys’

Attack that revealed data exposing deals with dictatorships was on a ‘governmental level’ and ‘planned for months’, says David Vincenzetti in first statement

locked laptop
Hacking Team founder speaks out about attacks that revealed company deals with dictatorships. Photograph: LJSphotography / Alamy/Alamy

The founder of cybersecurity firm Hacking Team has finally spoken out over the attack that saw 400GB of its data dumped on the internet, insisting: “We’re the good guys”.

David Vincenzetti, 47, founder of the Milan-based company, told Italian newspaper La Stampa that the cyber attack – which saw the code for companies hacking tools and its email archive published online – was not enabled by poor security or weak passwords and that it could have only been an organisation “at the governmental level”.

Vincenzetti said: “This is not an impromptu initiative: the attack was planned for months, with significant resources, the extraction of data took a long time.” But he did not explain how Hacking Team apparently failed to notice the attack while it was taking place.

Hacking Team hack casts spotlight on murky world of state surveillance

In response to concerns that Hacking Team supplied tools to repressive states which could be used to hack into and spy on almost anyone, Vincenzetti said: “We did [sell tools to Libya] when suddenly it seemed that the Libyans had become our best friends.” He also admitted providing tools to Egypt, Ethiopia, Morocco and Sudan, as exposed by the company’s email archive, though denied dealing with Syria.

But Vincenzetti said: “The geopolitical changes rapidly, and sometimes situations evolve. But we do not trade in weapons, we do not sell guns that can be used for years.” He said that without regular updates its tools are rapidly blocked by cyber security countermeasures.

In the case of the Ethiopian government, which used Hacking Team tools to spy on journalists and activists, Vincenzetti said: “We’re the good guys … when we heard that Galileo had been used to spy on a journalist in opposition of the government, we asked about this, and finally decided to stop supplying them in 2014.”

Meanwhile, the impact of the Hacking Team data dump continues to affect wider cubersecurity. A further two vulnerabilities within Adobe’s Flash plugin have been exposed and are actively being exploited as a result of the attack, Adobe has confirmed.

Warning over Adobe Flash vulnerability revealed by Hacking Team leak

Related:-

SALTED HASH-TOP SECURITY NEWS: Hacking Team hacked, attackers claim 400GB in dumped data: An email from a person linked to several domains allegedly tied to the Meles Zenawi Foundation (MZF), Ethiopia’s Prime Minister until his death in 2012, was published as part of the cache of files taken from Hacking Team

https://oromianeconomist.wordpress.com/2015/07/08/salted-hash-top-security-news-hacking-team-hacked-attackers-claim-400gb-in-dumped-data-an-email-from-a-person-linked-to-several-domains-allegedly-tied-to-the-meles-zenawi-foundation-mzf-ethiopi/

Advertisements

Obama’s Pilgrimage of National Interest.   Godaansa Fedha Sabaawaa July 14, 2015

Posted by OromianEconomist in Because I am Oromo, Ibsaa Guutamaa, US-Africa Summit.
Tags: , , , , , , , ,
add a comment

???????????Ethiopia's scores on freedom

Obama is going to tie knots with TPLF-EPRDF’s Ethiopia, the poorest country on the planet – on behalf of the richest and the most powerful country of the world. That is his right. Forget the stereotypical consolation of discussing human rights and democratic governance. The main purpose is economy and security. Again forget about the highest economic growth rates fanned by financial institutions. That, there is no growth can be seen with necked eyes. Million are hungry and destitute. This trip will try to accomplish the deregulation of remaining sectors, like banking, telecoms, land, etc. As far as the peoples of the Ethiopian empire are concerned, these were already deregulated, but monopolized by TPLF business conglomerates. International corporations want their big share. – Ibsaa Gutamaa

Obama’s Pilgrimage of National Interest

By Ibsaa Guutama* | July 2015

The oppressed and abused of Africa, and their friends and sympathizers are making their voices heard high above the globe that President Obama refrain from legitimizing dictatorship and human rights abuse in Ethiopia. This is not a casual visit, but a pre-planned trip for which arrangements were made to pave the way for the diplomatic pampering of the most brutal regime in the area; a long-time Guerrilla-friendly ambassador was appointed in addition to a visit by the U.S. Assistant Secretary for African Affairs. The rulers of Ethiopia were among those that Clinton’s Democrats branded as the “new breeds of African Democrats.” Many have now fallen out of line. These ones are also starting wavering. This visit of the President may keep them in line before they jump to another bandwagon. Obviously, it is not a pilgrimage of democracy, but that of “NATIONAL INTEREST.”

Democracy is an ideal which all human beings aspire to attend. So far, we have seen attempts, not commitment, to it. It is a principle well defined by persons like Abraham Lincoln, “Government of the people for the people …” But, in most cases, it remains in principle, not in practice. Otherwise, it is assumed that democracy is the standard of political governance – which has, at least, as ingredients equality, freedom, fairly elected officers, and freedom of speech and expression. Any of this missing, there could be no democracy. As the Assistant Secretary of U.S. State Department once said, “America believes in ‘NO DEMOCRACY NO COOPERATION.’” Ethiopia lacks even the few ingredients of democracy mentioned; however, cooperation has never been lacking for the last quarter of a century. It is alright to delay one to three years, but not to abide by one’s promise for so long, for the greatest country of the world is tantalizing. If people’s sovereignty was respected, there was no need to petition a far off power for one’s internal affairs. Their problem could be solved within the region. But, that remaining a wish, expecting fairness and impartiality from those interfering is not too much. Considering their position, they have the moral responsibility to do that.

On part of the abused people, the assumption that democracies of the world will come to their rescue should have been given up long ago. But, unfortunately, protest is made through the social media, through demonstration marches, through written materials, etc. to oppose the continued cooperation. It is all in vain; world powers are blinded by national interests. Hence, the poor and oppressed peoples of Africa are left solely to themselves to fight for their rights. The real “survival of the fittest” theory is being practiced by the world against Africa. Africans have come so long on their own; they have to continue gallantly defending their land, interests and dignity – not to perish silently. Whatever they are doing, world powers are doing knowingly and convinced that they are doing the right thing. Thousands could go hungry, rot in prison, forced to flee their country, and thousands could die and disappear, they are not worth severing friendship relations with perpetrators of genocide.

Obama is going to tie knots with TPLF-EPRDF’s Ethiopia, the poorest country on the planet – on behalf of the richest and the most powerful country of the world. That is his right. Forget the stereotypical consolation of discussing human rights and democratic governance. The main purpose is economy and security. Again forget about the highest economic growth rates fanned by financial institutions. That, there is no growth can be seen with necked eyes. Million are hungry and destitute. This trip will try to accomplish the deregulation of remaining sectors, like banking, telecoms, land, etc. As far as the peoples of the Ethiopian empire are concerned, these were already deregulated, but monopolized by TPLF business conglomerates. International corporations want their big share.

As for security, the TPLF is “the key ally of the U.S.” in hunting down terrorism. Already, many Ethiopian empire’s recruits have perished unaccounted for in Somalia. TPLF is ever ready to engage whenever the U.S. pays without any limit to geography in their deployment. The visit may encourage the continuation of this relation. As for terrorism, TPLF is manufacturing them at its convenience – killing thousands, and terrorizing and imprisoning numberless. Yes, the people have risen and are rising further against the TPLF terror. It is a rise for “liberty equality, freedom and peace” – which no body claims to know its cause and effect more than America. But since terrorism is not defined, the whole population of the empire is branded as a terrorist and is subject to persecutions. It is without consideration to redefine that the package is going to be discussed to strengthen the relation. The peoples of the empire, in particular the Oromo and journalists, are going to continue being terrorized.

Let alone a big power, the tiniest being knows no limit in defending its interests. But, for human beings, there should have been moral restraints. Here, our concern is not that for now. It is a lesson from history. During the past regimes, and under the present one, whenever there is an occasion, the destitute in towns are rounded up, beaten and taken to unknown places. There, they live in crowded enclosure without enough food and water in a deplorable hygienic environment. Many perish unreported. Now that a leader of the most powerful country is coming, and since the coming is unprecedented, unprecedented measures are certainly going to be taken. What makes this time different is that thousands were recently uprooted from their homes by the land grab, and the policy to de-Oromize and expand Finfinne (Addis Ababa). The evicted are the majority of thousands of homeless in Finfinnee. A fate worse than that of the infamous Shoolaa Camp under the emperor is awaiting them. Then very few mothers, children and the elderly were saved from typhus epidemic after university students discovered them accidentally. In addition to rounding these up much more harsh measures are to be expected to impress U.S. intelligence that certainly will be there to bolster their efforts.

Many complain that the visit amounts to recognizing the atrocities committed by the notorious dictators of the Horn of Africa. Had Africa not been ruled by autocrats pretending to be elected democrats, the visit would not have happened. Only those types can serve as partners in plundering the wealth and service of the continent during this period of the New Scramble for Africa. Whether the President visits or not, his administration had already recognized legitimacy of the illegitimate. The endeavors made to “democratize, and the free and fair election” was praised by frontline cadres months ago. Was it true? What they should complain about must have been their not been ready to defend their interests as peoples. Assuming democratic values are intrinsically universal, and no double standard for it, it would have been just if the President did not make the trip his predecessors had avoided. Healthy human and political developments could have eventually served the interests he is after better and for a longer time to come. But, the world had never been just.

If the President does not come out with a conclusion that he was dealing, not with hooligans, but legitimate rulers, the agony of the peoples of the Horn is going to be double fold; for the hooligans will be more encouraged with their brutality. We wish the President a good trip to his father’s land and back to the White House. Here inHabashaa land, his Lou people are going to be considered as Americans for his participation in the American administration, as Oromo are considered likewise for Tafarii’s participation in the Ethiopian administration. This trip will give the Wayyaanee a moral boost. We will see the leaders gleaning sitting around this powerful leader of the world to get photographed for the last time. People of the empire will wake up to another miserable day worse than before.Bon Voyage, Mr. President! Viva Oromiyaa! The struggle shall continue!

Honor and glory for the fallen heroines and heroes; liberty, equality and freedom for the living, and nagaa and araaraa for the Ayyaanaa of our forefathers!

* Ibsaa Guutama: Gubirmans.com

Godaansa Fedha Sabaawaa

Ibsaa Guutama irraa* | Adoolessa 2015

Ummati Afrikaa cunqurfamee fi itt roorrifame fi michuulee fi mararfatooti sagaleen saanii akka dhagahamutt baaqulaa ol kaasanii Presidant Obaamaan akka abbaa hirree fi mirga ilmoo namatt roorrisaa seerawaa gochuu irraa of qoqopbatu iyyaa jiru. Kun daawwii tasaa utuu hin tahin imaltu dursee karoorfame. Bulchoota godinaa keessaa nama nyaataakan tahee qanansiisa malbeekii kennuuf karaa haxaawuuf jechuun, daawwii Ittaantu Barreessituu Murnaasa Finnaatt dabalamee ambasadara michuu riphee lolaa bara hedduu muudamteett . Bulchooti Itophiyaa kanneen demokratoti Klinton “dhaloota demokratotaa haaraa” jedheen keessaa tokko turan. Hedduun amma karaa irraa maqaniiru. Jarri kunis daddaaqaa jiru. Daawwiin Presidant kun utuu isaan kan biraatt hin utaalin karaa irra isaan buusa taha. Godaansich godaansa demokraasii miti, kan “FEDHA SABAAWAATI.”

Demokraasiin fakmishoo ilmaan namaa hundi bira gahuuf hawwani. Hanga yoonaa abbala malee hojii irra ooluu saa hin agarree.Kun akeeka namoota akka Abraham Loncolniin dansatt “Mootummaa ummataa ummataaf …” jedhamee furameera. Garuu gara caaluu shaakala utuu hin tahin akeeka tahee hafe. Kanaaf malee akka jedhamutt demokraasiin hangammeessa bulcha malbulchaa kan qabeen saa yoo xinnaate “walqixxummaa, birmadummaa, qondaalota qajeeltoon kennatamanii fi bilisummaa haasaa fi of ifsuu” of keessaa qabu. Kanneen keessaa tokko hirdhannaan demokraasiin hin jiru. Akka bar tokko Ittaanaa Barreessaan Finnaa jedhett Amerikaan “DEMOKRAASIIN HIN JIRU, GAMTAAN HIN JIRU” kan jedhutt amana. Itophiyaatt qabee yartuu dhahaman keessaa hundatu hanqata. Garuu gamtaan arfacha jaarraaf hanqatee hin beeku. Waggoota tokko fi sadii tursuun hama hin tahu; yeroo dheeraa akkasiif biyyi addunyaa keessaa dagaagaan kun irbuu seene hin guutin hafuun malalchiisaa dha. Utuu moo’ummaan ummatootaa ulfeeffame dhimma keessa waliif humna fagoott iyyachuun barbaachisaa hin turre. Dhibdeen saanii godinaa keessatt furamuu dandaha ture. Garuu sun hawwa tahee hafus jara dhimmicha keessa seenan irraa qajeeltuu eeggachuun waan guddaa miti.

Sun kan fedhe haa tahu, gara itt roorrifamootaan demokraatoti addunyaa nuu birmatu taha jedhaanii eeggachuu qaata dhaabbachuutu irra ture. Garuu kan nama dhibu mormiin, gamtaan akka dhaabbatu karaa qubqabsiisii hawaasomaa fi dhiicha hiriiraan, barruulee kkf itt fufee jira. Hundi dhama’uu qofa; anga’oon addunyaa “fedha sabaawaan” jaamaniiru. Kanaaf hiyyeeyyii fi cunqurfamoon Afriikaa akka mirga ofiif lolatan of qofaatt dhiifamani. “Baraarama kan caalaa of dandahuu” dhugaan addunyaan Afriikaa irratt shaakalamaa jira. Afriikaanoti hanga yoonaa ofumaan as gahaniiruu; callisanii dhumuu irra jannumaan lafa, fedha fi ulfina saanii irraa faccisaa itt fufuu qabu. Anga’ooti addunyaa kan godhan hunda beekaa godhu, akka waan sirrii hojjetaa jiran of amansiisaniiru. Kumooti beela’uun, hidhaa keessatt samuun, biyyaa ari’amuu fi kumooti du’uu fi baduun firummaa kanneen qacefixa raawwatanii dhiisisuuf nafii hin guutani.

Obaaman ABUT-ADWUE biyya, biyyoota addunyaa hunda keessaa deegduu taatett rakoo qaluuf deema. Sun mirga saati. Jecha mararfannaan keessa deddeebi’amu waa’ee mirga ilmaan namaa fi bulcha demokratawaa jedhamu haa irraanfannu. Mummichi manaa isa baase dhimma diinagdee fi nageenyatii. Ammas waa’ee deettii guddina diinagdee dhaabota horoon afarsamus haa irraanfannu. Akka guddinni hin jirre ija qullaan ni argama. Kumkumooti beela’oo fi deegoo dha. Daawwiin kun kutaa diinagdee dambii jalaa hin bahin hafan kan baasisuu yaalu.Ummatoota Empayera Itophiyaaf kun qaata dambii jalaa bahee, garuu hidhata daldala ABUTin dhuunfatame. Korporashinooti addunyaasi qooda guddaa ofii barbaadu.

Waan nageenya ilaalutt gooltuu duukaa bu’ee adamsuuf ABUT “gartuu furtuu U.S.” Somaliyaa keessatt madaqfamtuun Empayera Itophiyaa heeduun duruu dhumanii gabasa malee hafanii. ABUT utu lafquwiin isa hin danga’in U.S. kaffallaan ergamuuf qophee dha. Daawwich akka hariiroon kun itt fufu jajjabeessa taha. Waan gooltummaa ilaalutt ABUT akka fedhett homishee, kumoota ajjesee fi shoroorkeessee kanneen lakkofsa hin qabne hidha. Dhuguma shoroorkaa ABUTiin mormuun ummatooti ka’aniiru, caalaa ka’aas jiru. Kaka’a “gaaffii mirgaa, walqixummaa, birmadummaa fi nagaaf” godhamu, kaasaa saa fi itt baha saa Ameriika caalaan beeka kan jedhu hin jirre. Gooltummaan waan maljechi saa hin himamneef dilormaan empayeritt hundi gooltuu jedhamee unkuramaa jira. Utuu sana hin maljechisini kan amma hariiroo jabeessuuf walti qabaaman mari’atamuuf deemu. Ummatooti empayeritt keessattuu Oromoo fi jornaalistoota shororkeessuun itt fufufuuf deema jechuu dha.

Aanga’aa guddaan hafee uumaa bucureenuu fedha ofii eeggachuufn hin daangahamu. Garuu ilmoo namaaf safuun madaalaa jiraachuu qaba ture. Ammaaf dhimmi keenya sana miti. Barnoota seenaati. Bulchoota duriif isa ammaa jalattis yeroo waa tokko jiraatu deegaan magaalota keessaa marfamanii tumamaa bakka hin beekamnett geeffamu. Achitt dallaa nammi walitt gooji’e keessa nyaataa fi bishaan gahaa malee, haala qulqulinni hin eegamne jala jiraatu. Heedduun dhumanii hin gabaafaman. Amma mootiin biyya hunda caalaa jabaan dhufaa jira. Dhufaatiin akkasii kanaan dura argamee waan hin beekneef tarkaanfiin fudhatamus mamii malee kan argamee hin beekne taha. Kan isa si’anaa adda tolchuu ummati kumootaan samicha lafaa fi imaammata Finfinnee Orommiteessuu fi ballisuuf lafa saanii irraa kumootaan buqqifamuu dha. Alabul kumootaan lakkaawaman Finfinnee keessaa garri caaluu buqqifamoota sana. Hiree Mooraa Shoolaa bara Nugusaa caalaa hamatutu isaan eeggata jechuu dha. Yeroo sana haadhootaa fi ilmaanii fi Manguddoo yartuutu akka tasaa barattoota Universiitiin argamanii golfaa irraa hafanii. It dabalee jara kana marsanii qabuu fi qaruutee angoo saanii ol kaasuuf Amerikaa dhufuun saanii hin hafnett of agarsiisuuf tarkaanfii caalaa hamaa tahe fudhachuun akka hin hafne hedama.

Hedduun kan komatan, daawwichi badiisa abbaa hirree gamsiisa Ganfa Afriikaan hojjetamaa jiruun hamma ishoo jechuuti jedhaniitu. Afrikaan jara abbaa hirrootaa kan ummataan kennataman fakaatanii dhihatanii utuu hin bulfamtu taatee daawwichiyyuu hin godhamu ture. Jara akkasii qofa kan saamicha qabeenyaa fi maayi baasii waldayyichaa irratt bara Hirmannaa Afriikaa Haaraa kana miltummaan tajaajiluu kan dandahan. Presidantiin daawwatee dhiisee bulchi saa qaata seerawwoo kan hin taane, seerawoo tahuu kan beekeefi. “Ifaajjee demokratawuu fi kennata qajeelaa fi bilisa hojii irra oolchuuf deemanis” dabbaloota qaraatiin ji’oota dura farsamaniiru. Kun dhugaa dhaa? Kan isaan komachuu qaban mirga ofii irraa faccifachuuf qophee tahuu dhabuu saanii tahuu qaba ture. Demokraasiin nafii barbaachisaa halle hammataa qaba yoo jedhame fi hangammeessa lakkuu hin qabu yoo tahe Presidantiin imala angafooti saa lagatan utuu hin goonee dansa ture. Misi fayyaa qabeessi ilmoo namaa fi malbulchaa, bulee fedha inni duukaa jiru caalaatti yeroo dheeraaf tajaajilu dandaha ture.

Prsesidantichi addaggoota utuu hin tahin bulchoota seerawoo waliinin dubadhe jedhee baanaan gadadoon ummatoota Gaanfaa dacha tahuuf deema; addagooti jajjabina argatanii hammeenya saanii kan dur caalaa cimsanii itt fufu. Presidantichi milla mikii qabuun gara biyya abbaa saa gahee Waayit Hawusitt akka deebi’u hawwinaaf. Akka biyya Habashaatt ummati saa Luwoon waan inni bulcha Amerikaa keessatt qooda fudhateef Amerkaaniitt fudhatamuuf deemu; akkuma Oromoon Tafariif jedhamee ilaalaman jechuu dha. Imalli kun hamilee Wayyaanee dhiitessa. Hoogganooti see, hooggana addunyaa humna guddaa kanatt marsanii yeroo dhumaaf footoo kahuuf yoo ilkaan afan ilaaluuf deemna. Ummati empayeraa bariin dammaqee guyyaa gadadoo kan duraa caalutt gad baha. Karaa Nagaa! Oromiyaa haa jiraattu! Qabsoon itt fufa!

Ulfinaa fi surraan gootota kufaniif; walabummaa, walqixxummaa fi bilisummaan kan hafaniif; nagaa fi araarri Ayyaana abboolii fi ayyoliif haa tahu!

* Ibsaa Guutama: Gubirmans.com

Data Science: Avoiding a common mistake with time series July 14, 2015

Posted by OromianEconomist in 10 best Youtube videos, 25 killer Websites that make you cleverer, Data Science.
Tags: , , , , , ,
1 comment so far

???????????

Data Science Central

Avoiding a common mistake with time series

By Tom Fawcett*

A basic mantra in statistics and data science is correlation is not causation, meaning that just because two things appear to be related to each other doesn’t mean that one causes the other. This is a lesson worth learning. If you work with data, throughout your career you’ll probably have to re-learn it several times. But you often see the principle demonstrated with a graph like this: Dow Jones vs. Jennifer Lawrence

One line is something like a stock market index, and the other is an (almost certainly) unrelated time series like “Number of times Jennifer Lawrence is mentioned in the media.” The lines look amusingly similar. There is usually a statement like: “Correlation = 0.86”.  Recall that a correlation coefficient is between +1 (a perfect linear relationship) and -1 (perfectly inversely related), with zero meaning no linear relationship at all.  0.86 is a high value, demonstrating that the statistical relationship of the two time series is strong. The correlation passes a statistical test. This is a great example of mistaking correlation for causality, right? Well, no, not really: it’s actually a time series problem analyzed poorly, and a mistake that could have been avoided. You never should have seen this correlation in the first place. The more basic problem is that the author is comparing two trended time series. The rest of this post will explain what that means, why it’s bad, and how you can avoid it fairly simply. If any of your data involves samples taken over time, and you’re exploring relationships between the series, you’ll want to read on.

Two random series

There are several ways of explaining what’s going wrong. Instead of going into the math right away, let’s look at a more intuitive visual explanation. To begin with, we’ll create two completely random time series. Each is simply a list of 100 random numbers between -1 and +1, treated as a time series. The first time is 0, then 1, etc., on up to 99. We’ll call one series Y1 (the Dow-Jones average over time) and the other Y2 (the number of Jennifer Lawrence mentions). Here they are graphed: Series Y1 Series Y2 There is no point staring at these carefully. They are random. The graphs and your intuition should tell you they are unrelated and uncorrelated. But as a test, the correlation (Pearson’s R) between Y1 and Y2 is -0.02, which is very close to zero. There is no significant relationship between them. As a second test, we do a linear regression of Y1 on Y2 to see how well Y2 can predict Y1. We get a Coefficient of Determination (R2 value) of .08 — also extremely low. Given these tests, anyone should conclude there is no relationship between them.

Adding trend

Now let’s tweak the time series by adding a slight rise to each. Specifically, to each series we simply add points from a slightly sloping line from (0,-3) to (99,+3). This is a rise of 6 across a span of 100. The sloping line looks like this: Trend line

Now we’ll add each point of the sloping line to the corresponding point of Y1 to get a slightly sloping series like this: Series Y1 Prime

We’ll add the same sloping line to Y2: Series Y2 Prime

Now let’s repeat the same tests on these new series. We get surprising results: the correlation coefficient is 0.96 — a very strong unmistakable correlation. If we regress Y on X we get a very strong R2 value of 0.92. The probability that this is due to chance is extremely low, about 1.3×10-54. These results would be enough to convince anyone that Y1 and Y2 are very strongly correlated! What’s going on? The two time series are no more related than before; we simply added a sloping line (what statisticians call trend). One trended time series regressed against another will often reveal a strong, but spurious, relationship. Put another way, we’ve introduced a mutual dependency. By introducing a trend, we’ve made Y1 dependent on X, and Y2 dependent on X as well. In a time series, X is time. Correlating Y1 and Y2 will uncover their mutual dependence — but the correlation is really just the fact that they’re both dependent on X. In many cases, as with Jennifer Lawrence’s popularity and the stock market index, what you’re really seeing is that they both increased over time in the period you’re looking at. This is sometimes called secular trend. The amount of trend determines the effect on correlation. In the example above, we needed to add only a little trend (a slope of 6/100) to change the correlation result from insignificant to highly significant. But relative to the changes in the time series itself (-1 to +1), the trend was large. A trended time series is not, of course, a bad thing. When dealing with a time series, you generally want to know whether it’s increasing or decreasing, exhibits significant periodicities or seasonalities, and so on. But in exploring relationships between two time series, you really want to know whether variations in one series are correlated with variations in another. Trend muddies these waters and should be removed.

Dealing with trend

There are many tests for detecting trend. What can you do about trend once you find it? One approach is to model the trend in each time series and use that model to remove it. So if we expected Y1 had a linear trend, we could do linear regression on it and subtract the line (in other words, replace Y1 with its residuals). Then we’d do that for Y2, then regress them against each other. There are alternative, non-parametric methods that do not require modeling. One such method for removing trend is called first differences. With first differences, you subtract from each point the point that came before it: y'(t) = y(t) – y(t-1) Another approach is called link relatives. Link relatives are similar, but they divide each point by the point that came before it: y'(t) = y(t) / y(t-1)

More examples

Once you’re aware of this effect, you’ll be surprised how often two trended time series are compared, either informally or statistically. Tyler Vigen created a web page devoted to spurious correlations, with over a dozen different graphs. Each graph shows two time series that have similar shapes but are unrelated (even comically irrelevant). The correlation coefficient is given at the bottom, and it’s usually high. How many of these relationships survive de-trending? Fortunately, Vigen provides the raw data so we can perform the tests. Some of the correlations drop considerably after de-trending. For example, here is a graph of US Crude Oil Imports from Venezuela vs Consumption of High Fructose Corn Syrup: US Crude Oil Imports vs. HFCS The correlation of these series is 0.88. Now here are the time series after first-differences de-trending: US Crude Oil Imports vs. HFCS de-trended

These time series look much less related, and indeed the correlation drops to 0.24. A recent blog post from Alex Jones, more tongue-in-cheek, attempts to link his company’s stock price with the number of days he worked at the company. Of course, the number of days worked is simply the time series: 1, 2, 3, 4, etc. It is a steadily rising line — pure trend! Since his company’s stock price also increased over time, of course he found correlation. In fact, every manipulation of the two variables he performed was simply another way of quantifying the trend in company price.

Final words

I was first introduced to this problem long ago in a job where I was investigating equipment failures as a function of weather. The data I had were taken over six months, winter into summer. The equipment failures rose over this period (that’s why I was investigating). Of course, the temperature rose as well. With two trended time series, I found strong correlation. I thought I was onto something until I started reading more about time series analysis. Trends occur in many time series. Before exploring relationships between two series, you should attempt to measure and control for trend. But de-trending is not a panacea because not all spurious correlation are caused by trends. Even after de-trending, two time series can be spuriously correlated. There can remain patterns such as seasonality, periodicity, and autocorrelation. Also, you may not want to de-trend naively with a method such as first differences if you expect lagged effects. Any good book on time series analysis should discuss these issues. My go-to text for statistical time series analysis is Quantitative Forecasting Methods by Farnum and Stanton (PWS-KENT, 1989). Chapter 4 of their book discusses regression over time series, including this issue.   *Tom Fawcett is Principal Data Scientist at Silicon Valley Data Science. Co-author of the popular book Data Science for Business, Tom has over 20 years of experience applying machine learning and data mining in practical applications. He is a veteran of companies such as Verizon and HP Labs, and an editor of the Machine Learning Journal.

Related:-

https://oromianeconomist.wordpress.com/2015/07/07/statistics-the-sexiest-job-of-the-decade/

https://oromianeconomist.wordpress.com/2015/06/22/what-is-calculus-used-for-tedx-talks/