Yes, it is pretty cool to have a job title like this on your business card, but most of the time it also requires a lot of explaining. If you want to know how we raised a pretty, little thing called Hydra, which strolls casually through our databases and servers, please keep reading.
What is this Hydra thing?
Depending on if you are a Marvel comic book fan or a Joseph Campbell type of writer, you may think that Hydra is either a secret evil organization that hungers world domination or a mythical creature with an equally exhaustive hunger of a slightly different nature. At GetResponse we thought the latter and instead of killing it, as some Greek guy did long time ago, we decided to tame Hydra, and make it work for a better tomorrow 😉
So… what hides under this ‘cute’ nickname is our in-house anti-abuse system we, as GetRespose, have been developing for quite some time.
Cool, dude, cool, but what for?!
As with many other ESPs, GetResponse is exploited by spammers and a variety of bad actors in order to send spam. During the summer of 2014 spam threats were flooding our Compliance Team and left very little place for what we think should be main role of the Compliance Dept. – helping our customers to stay in compliance with email industry standards as well as our own policies.
We were fighting waves of bad guys with baseball bats. And so, we needed something more, something that could do the heavy lifting for us, do all the leg work so that people with expert knowledge will not spend their energy on tasks that should be automated.
The Idea was simple – automate as much as you can with as small room for error as is possible. To do so we really needed our very own Hydra – in case you bypass one of the heads, another still keeps an eye out for you. Over the course of 18 months we were able to create a system that helps us assess risk related to GetResponse accounts, then launching automated operations accordingly.
First bricks, first success.
Every project needs to start somewhere, so a desire to get to the low hanging fruits drove us to conclude that we should start by mimicking processes we did by hand. At first we translated into a bunch of simple rules, what we usually did manually, to fish out malicious, free accounts that flooded us at that time.
Although we tried a few approaches finally we saw that certain IP spaces or mailbox providers can be linked to bad actors with almost a 100% certainty. That part was pretty easy, now we had a net full of accounts and we had to do something with it. What should we do to minimize false positive rates so as to not make legitimate customers life difficult, while making sure that those we need to repel do not slip away?
Here enters the spam mitigation philosophy. Big words, I know 😉 For us it translated into a simple, well known idea – increase the cost of operations for bad guys. So we did, in August 2014 we first introduced text verification for those accounts that Hydra sniffed, gazed at, and finally put label the “high risk” on. It was a huge break for our Compliance Team, over the course of a few months we were able to lower the number of abusive free accounts we had to take care of by over 4 times.
We were just getting started.
Those of you who have anything to do with security know it is a constant arms race. One side comes up with an idea how to patch a hole in the wall, then someone else finds a new one a few meters further. It is a drag. So it’s good to have something that will adjust to changing circumstances, right? Right.
(Image source: Giphy)
After the first success we developed Hydra further. With the help of our bio-engineering team we mutated Hydra’s DNA so our little pet could grow new heads and expand its brain size… OK, just kidding, we simply wrote hundreds of lines of code. From a bunch of simple rules, we evolved it into a creation that can learn from what it finds.
(Image source: Giphy)
If you ever imported subscribers into a GetResponse account, your list went through a verification process. At first it was done totally by hand, based on expert knowledge from our anti-abuse specialists. You can imagine that such a process took hours, required a lot of manpower, all of that so we could ensure the highest email deliverability possible throughout the entire GetResponse platform.
A huge step forward was made by making Hydra do the heavy lifting for us, even more, we altered the whole process to make it more thorough, as time required to review a list was no longer a factor we had to consider – Hydra devours even the biggest of lists within seconds.
At this point Hydra was no longer an internal anti-abuse system, it brought real value to our customers, lowering list import time from hours to minutes. Moreover, the whole infrastructure we created for the sole purpose of list reviewing was used to pave ground for a feature that helps marketers get higher open and click rates of their mailings – Perfect Timing. Again – the more lists Hydra goes through, the better it can score and review future lists.
How about some more mutagens?
That was what we thought – let’s go further, what else Hydra can help us with? A few months after the success of the new, automated list review process we decided that we are ready to implement the same rules for real time subscriptions. That was a bit of a challenge, though. See, when a list is uploaded for import we can look at it as one entity. It is pretty easy to find correlations with other lists, score all the email addresses, and decide whether or not hygiene of the list is good enough to be approved, when you have everything in front of you.
With a constant stream of subscriptions, they need to be considered somewhat differently. First of all, scores for a list that was sent to in the past will be different than for new subscribers that just placed their email address in the webform or closed a shopping cart (i.e. with all the syntax errors). The idea behind unleashing Hydra on the live feed was that it can help our customers who became victims of malicious scripts, bots, or any other kind of address injection. With that in place we were also able to remove the required double opt-in subscription for API and put in place a process granting single opt-in for API scripts far behind us – another win for all of our customers!
TL;DR? Ok, here is the conclusion.
Hydra’s automation helped us save thousands and thousands of work hours both for us and our customers, and it made the Compliance Team available when it is really needed – helping customers with expert knowledge. It also drastically improved experience connected to compliance and security processes that are required from whitehat ESPs for our customers.
Hydra’s decision array is made out of hundreds of single tests, all those “heads” are on the lookout for different patterns and factors. We constantly develop our little pet so it can do more tricks and adapt to new threats we come across every day. #keepcalmandhailhydra 😉
Editor’s note:
Want to help and improve your own email deliverability? Read this article on why emails go to spam and how to prevent them from doing so.