The forums suffered a cyberattack from China earlier today...

Here you might discuss basically everything.

Moderator: SC Moderators

Post Reply
User avatar
Gwyneth Llewelyn
Forum Wizard
Forum Wizard
Posts: 1183
Joined: Thu May 25, 2006 8:00 am
Contact:

The forums suffered a cyberattack from China earlier today...

Post by Gwyneth Llewelyn »

... well, if that didn't get your attention, I wonder what would :D

So, early today, the forums were down — Sudane sent me a message to let me know about it. In fact, I was getting all sorts of alarm signals from the server where the forums are currently hosted, and couldn't make any sense of them!

Basically, what happened was the following: for the past month or so, 'someone' in China has been searching very aggressively a lot of forums out there. This could simply be a new search engine being set up — search engines need to 'crawl' all sorts of websites out there to extract their content. These days, search engines (besides the 'biggies' such as Google, Bing, or China's own Baidu) are used for all sort of marketing/advertising efforts, and one thing that allegedly they do is to test the performance of the sites they 'crawl'. In other words: it's not enough to know what content is in a specific site, it's now important to know how quickly that content can be viewed because that information is allegedly important for marketers when they buy ads from the many networks...

Google and Bing (among others, of course) are 'well-behaved': you can actually tell those search engines that their content crawling is badly hurting the performance of your website. Others, of course, pretty much ignore anything you 'tell' them. This seems to be the case of the 'attack' early today.

What the 'attacker' (between quotes, because, well, the 'attack' may not have been deliberate...) did was simply to use about 8000 different servers to open connections to our forums. Each connection was, by itself, harmless — just a request to view some content, like the list of forums, or a specific page, etc. Each server, by itself, was 'moderate' in its requests — just around one request of second, which is less than a tenth of a thousandth of what the web server running these forums can handle. The problem was that these guys would make one request from each of those 8000 servers simultaneously! Suddenly the web server needed to handle 8,000 requests per second (well, on the worst-case scenario; in practice, it was not that bad). Now, this is quite a lot — but the server can deal with that amount of requests. Not easily — I'm sure the server will shake and rattle on the rack :-) — but it should be more than able to handle that. So this was not the main issue.

Where things get complicated is what happened next. When anyone accesses our forums without being registered, they automatically become 'anonymous users' or 'guests'. As you might know, such users just see a fraction of all the content in here; nevertheless, we cannot limit the number of 'guest users' that visit the forums. We can most certainly prevent them from reading 'sensitive' information — which will really just interest citizens and nobody else — but we either shut down the forums entirely to the public-at-large or we 'have' to give them some access. The forum software, in its current incarnation, does not limit the number of guest visitors that can be online simultaneously.

Now — and I'm well aware I'm becoming technical! — each time someone (a citizen, another human, or a bot) connects to our forums, it creates what is known as a 'session'. For humans, this means getting a browser cookie with a unique ID which allows the forum software to 'recognise' you. A copy of the cookie is stored in the database server — so if you log in on a later date, the forum software can look that 'session cookie' up on the database, and know who you are (in the sense of figuring out your preferences, where in the forums you were last so that it can show the same thread/post again, etc.).

For citizens — all of which are registered users — the database just stores 'one' session for each login. This makes sense because your login/password is supposed to be unique and not shared with the whole world. That means that the forum software 'remembers' your session (at least for a while — for security reasons, it will 'forget' you if you don't log in regularly) and restores the same session ID when you log in again.

Bots by Google, Bing, and a plethora of other 'legitimate' search engines work differently. Because they will also get access to content, the software internally assigns a 'virtual' session ID (so, in a sense, Google, Bing etc. are actually 'pseudo-users' in our forums!) that gets reused over and over again — for the same reason, if it's Google, it's pointless to give Google a new session when it comes crawling our content again. Google is Google and will remain Google, so... they get a 'virtual' user with a 'virtual' session ID. That does not mean that Google can 'log in' to our forums, but it does mean that Google is getting recognised as being somehow 'legitimate' (and a reference to Google appears somewhere on the list of 'current users') and does not get 'kicked out'. This is done via a simple table in the database where the forum software looks up from a list of known, legitimate web search engines. It's not a very long list, but it lists a few dozens of 'genuine' web servers.

Anonymous guests or visitors — basically, anyone or anything which is not a (registered) citizen of the CDS or a valid, legitimate, genuine web search engine — are handled quite differently. Because the forums have no way to 'know' who is behind an anonymous connection (well, that's why it is called anonymous :-) ), the only way to figure out if it's the 'same' person coming back to visit over and over again is to create a new session and 'serve' a cookie to the browser of that person, so that the next time they visit the forum, the forum software will know that it's 'this specific anonymous person and not someone else'. So, in fact, anonymous users remain anonymous — but the forum software 'knows' that there are many of them, and will not mix them up.

As you all know, you can disable cookies on your browser (meaning mostly that you'll lose a lot of functionality on practically every website you visit...). Aggressive bots (malicious or not) will also do the same, which means that when the web server 'hands them' a cookie, they will discard it silently — for several reasons, one of which is that storing cookies for literally billions of websites (or at least hundreds of millions!) would require an immense amount of storage! So what happened was that this particular 'attack' from China was requesting, say, some thousand 'sessions' per second (theoretically more, but a thousand would be more than enough for what happened next), being handed over a thousand cookies, which would be discarded... and on the next second, request a thousand more. Because the cookies were not stored, our forum software had no way to know that the requests were from the same software/server/bot, and just opened a new session and handed over a new cookie... and so forth.

One might wonder — well, if the forum software is so clever that it 'knows' when Google or Bing is knocking at its door, why didn't it do the same in this case? The main reason for that has to do with the way the Web works. When a browser requests a web page, it also sends (at least) two things with the request: the IP address and a 'handle' which identifies the kind of browser making the request (as well as the underlying operating system, the language it prefers, the size of the screen, etc.). This information is used for the web server to better handle the request — for instance, if it recognises that the request comes from a mobile device and not a personal computer, it can send back a page that adjusts itself to the smaller screen size. Or send a page in German if the browser is set to German as its main language. The Google and Bing crawlers also identify themselves as 'bots' or 'crawlers', with a specific 'handle' which is publicly known, and so they can be properly handled by the forum software as a 'special' kind of 'pseudo-user'.

This particular 'attack' from China had a clever twist: with each request, it would send a different handler, 'impersonating' all sorts of hardware (from tablets to smartphones to laptops...) and browsers. It would randomly pick one handler and just make a request for that specific IP address/handler combination. Now on our side, all we 'know' is that a bunch of requests are coming from one particular IP address (out of 8,000...) and from 'several different devices' — which is exactly what would happen if, from the comfort of your home, you'd log in to the CDS forums from your mobile, tablet, and laptop, all at the same time. A perfectly legitimate and plausible scenario: the forum software 'expects' such scenarios to be 'usual' or 'normal' and so as no reason to suspect it's being a victim of a cyberattack...

Each time a brand new session is created — for a different IP address/browser handle combination — the forum software must go to the database server, check if that particular combination is there or not, and, if it's not, create a new session for this new request, write it to the database, create a new cookie, and hand it over to what it thinks to be a 'new' visitor. The trouble is that this 'visitor' is not human and throws the cookie away, and each time it comes back, the database has to be checked... a new session needs to be created... the database gets the new session written to it... a cookie is handed back and promptly discarded... wash, rinse, repeat. A thousand times per second. Well, more or the less. You get the idea!

It didn't take long to reach a million open sessions — but the main issue was not with those. Again, the web server can handle that. However, the 'weak link' here is the database server. It can certainly handle a hundred or so requests per second if these are 'reasonable' requests — and mostly reads (and not writes). It started to get stressed when it had to deal with thousands of write requests, while at the same time also making checks on the database for existing sessions... at some point, the database simply couldn't handle all those requests in real time, and had to put some of them into a backlog... but more and more requests were coming in... and the backlog was growing and growing... to the point when, for all purposes, the database server was essentially frozen, buried under a pile of ever-growing requests.

That was when people stopped being able to log in to the forums (or, if they were already logged in, they wouldn't be able to see much).

To fix this, besides figuring out what exactly was happening (which is not always easy!), I had to identify the origin of the attack, step up the overall security (that's why some of you have seen a weird-looking page for a few seconds saying that your browser was being 'tested' — this is a very efficient counter-attack measure, since bots will not be able to do anything with that page, and so the front-end 'knows' it's a bot and not a human wanting to see the content), and finally block the whole Chinese block from where the attack originated (that takes just a few seconds). In the meanwhile, I updated the forums to the latest version. It doesn't 'fix' this kind of attack, but it should help with other security issues (or so the developers claim!).

Is this a 'permanent' fix? Well, not really. We won't be 'attacked' again (assuming it was an attack...) from that specific Chinese group of servers, but, of course, there can be others — and allegedly there are — using this exact approach to put websites such as our forums out of business, either for laughs, or deliberate reasons. At this very moment, there is no simple way to deal with it — and this is a reason why I posted on these technical support forums to see if someone comes up with an idea on how to fix this. Ironically, the fix ought to be very easy — it's just something nobody had thought of doing before!

Also, note that this kind of attack would not affect the CDS Portal at all. It's just a different type of software — it handles 'anonymous' visitors in a completely different way — therefore, an increase in visitors, even so-called 'malicious' bots, would not affect the database directly (the bots would only see previously statically generated content), and I'm assuming that the web server running the CDS Portal would be able to deal with the extra traffic. It's just that the phpBB forum software works quite differently...

Whew. Now that was a long one! Well... you know how I am!... :-)

Anyway... I'm very sorry for what happened and for the downtime! Unfortunately, there was really nothing that could have been done to deal with this particular 'attack' beforehand. I hope that I might at least have given the phpBB forum software community some incentive to work on a fix (so far, they do not seem to be very excited about that), because I'm afraid that I don't have the time to do it — even though I'm aware it's not really a very hard fix!

"I'm not building a game. I'm building a new country."
  -- Philip "Linden" Rosedale, interview to Wired, 2004-05-08

PGP Fingerprint: CE8A 6006 B611 850F 1275 72BA D93E AA3D C4B3 E1CB

User avatar
Rosie Gray
Forum Wizard
Forum Wizard
Posts: 2045
Joined: Sun Jun 06, 2010 9:47 am

Re: The forums suffered a cyberattack from China earlier today...

Post by Rosie Gray »

Wow! Thanks for the explanation, Gwyn. :shock:

"Courage, my friend, it's not too late to make the world a better place."
~ Tommy Douglas
User avatar
Sylvia Tamalyn
Master Word Wielder
Master Word Wielder
Posts: 458
Joined: Sat Dec 31, 2016 8:07 am

Re: The forums suffered a cyberattack from China earlier today...

Post by Sylvia Tamalyn »

Very interesting, Gwyn! Thanks for taking the time to spell it out in detail for us.

Over the past few weeks I have noticed some random times where there were WAY more "guests" than usual on here. I am going to assume there's a connection, like they were testing the waters in advance of the main event.

User avatar
Han Held
I need a hobby
I need a hobby
Posts: 690
Joined: Mon Feb 16, 2015 3:52 pm
Contact:

Re: The forums suffered a cyberattack from China earlier today...

Post by Han Held »

Gwyneth Llewelyn wrote: Sun Dec 02, 2018 4:35 pm

Anyway... I'm very sorry for what happened and for the downtime! Unfortunately, there was really nothing that could have been done to deal with this particular 'attack' beforehand. I hope that I might at least have given the phpBB forum software community some incentive to work on a fix (so far, they do not seem to be very excited about that), because I'm afraid that I don't have the time to do it — even though I'm aware it's not really a very hard fix!

Thank you for being on top of it, Gwyn -and for taking the time to break it down for us!

I followed your link and noticed their "meh" reaction too and thought it was rather odd (at least from my point of view).

---
"I could talk talk talk, talk myself to death
But I believe I would only waste my breath" -Roxy Music "Remake, remodel"
User avatar
Kyoko
Pundit
Pundit
Posts: 303
Joined: Thu Jun 28, 2018 7:23 pm

Re: The forums suffered a cyberattack from China earlier today...

Post by Kyoko »

Thank you so much Gwyneth.
That was magnificent sleuthing and a clear explanation.

I feel your pain as in the !990s a Unix server I was admin on was used, among MANY, to launch a Denial of Service attack on CNN. While I enjoyed working with the Canadian "Mounties", I wouldn't wish reading endless UNIX logs on anyone!

CDS Citizen
Em Warden
Passionate Protagonist
Passionate Protagonist
Posts: 136
Joined: Wed Nov 16, 2011 5:05 pm

Re: The forums suffered a cyberattack from China earlier today...

Post by Em Warden »

I join the quoire of thanks, Gwyneth!
It was a very interesting story indeed. Also in the Chinese meaning of the word "interesting" :)

When you go through hell- keep walking!

Winston Churchill
Post Reply

Return to “General Discussion”