0

So I am fairly new to pretty much anything server related. I recently pushed my project to production. It currently has very little website visitors. I am running it on a simple Digital Ocean Ubuntu 20.04 server. Now I've noticed that sometimes all of the sudden out of nowhere the site completely goes offline (the CPU usage goes from 0% to 100& in less than 10 minutes) as seen in the image below:

see image of cpu usage chart here

Now to be completely honest, I don't really know where to start looking. The first thing I checked were the server system logs.

I've noticed a tremendous amount of UFW BLOCK logs. This is an example of one of them:

kernel: [ 1355.970674] [UFW BLOCK] IN=eth0 OUT=
MAC=02:02:e9:7b:70:dd:fe:00:00:00:01:01:08:00 SRC=45.143.200.34
DST=128.199.58.144 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=27065 PROTO=TCP
SPT=56880 DPT=7361 WINDOW=1024 RES=0x00 SYN URGP=0

Looking at the timestamps, I can see that these come in every few milliseconds or so...

I've learned from another post that these are logs about traffic being blocked. When looking at the SRC and searching that IP address, most of them come from Russia? Which is strange because my website is Dutch and has 100% Dutch visitors only.

Could these traffic blocks be the reason that the CPU usage goes to 100% in minutes sometimes? Should I be worried about these blocks? And lastly, what could be other reasons that the site goes offline in minutes out of nowhere with 0 users?

I've looked at the processes that are currently using memory and CPU.

Below is a screenshot of the htop output:

Here you can see an image of the htop output

matigo
  • 20,403
  • 7
  • 43
  • 70
I'm a Koala
  • 1
  • 1
  • 3
  • 2
    Normal behavior. Anything facing the Internet gets scanned/probed by a bazillion IPs. This is normal behavior, the fact it's all being blocked is a good thing. – Thomas Ward May 20 '21 at 13:17
  • Refer to my answer at: https://askubuntu.com/questions/828223/disable-logging-of-ufw-blocks-in-the-kernel-logs/828331#828331 regarding the blocks – Thomas Ward May 20 '21 at 13:19
  • 1
    Regarding your site going offline here and there, you may be getting overloaded by requests, in which case your server is not tuned correctly. However, that will require actively digging and debugging in a way that is a bit beyond the scope of what Ask Ubuntu can provide (because that debugging process is going to be massive) – Thomas Ward May 20 '21 at 13:20
  • 1
    What processes are responsible for the CPU usage? You'll need to check using `top`, `htop`, `glances`, or another tool that you prefer. If it's the web server, then you can begin your search there. If it's the database (if you use one), then you can look for queries that may need optimisation. Just a chart is insufficient to know if it's UFW or another process that's consuming all of your processor resources. – matigo May 20 '21 at 13:20
  • @ThomasWard Thanks a lot for your answer. Atleast I know I can look elsewhere now :-) – I'm a Koala May 20 '21 at 13:36
  • @matigo Hi! I've updated the post and included a screenshot of the htop output. I don't think I can watch the output from the moment it went down, can I? Do you see anything strange in the output? Thanks in advance! – I'm a Koala May 20 '21 at 13:37
  • Your node application has consumed a remarkable amount of processing time. That's where you will need to invest your time. Check your logs, maybe do some performance testing on a lower-powered virtual machine on your local workstation. Double-check that you don't have any infinite loops in your code – matigo May 20 '21 at 13:54
  • Awsome. Thanks a lot! @matigo – I'm a Koala May 20 '21 at 13:58
  • 1
    It sounds like Denial of Service attack, which is basically what @ThomasWard mentioned earlier. – Doug Smythies May 20 '21 at 16:29

1 Answers1

2

Based on the comments stated and such, other than the UFW BLOCK issue which is explained elsewhere, it looks like your application stops responding because of some type of Denial of Service attack - some request is getting passed to Node that is causing it to eat all the CPU and freeze up. You'll need to start tracking your NodeJS logs and harden your application - it's going to be something related to your application or a NodeJS exploit that is being successfully used on your app.

Thomas Ward
  • 72,494
  • 30
  • 173
  • 237
  • What makes you think that it is a DOS attack? How can you 'see' that `some request is getting passed to Node`? – I'm a Koala May 23 '21 at 06:59
  • 1
    @I'maKoala : Your statement: `these come in every few milliseconds or so` makes us think of DOS attack. It is a best guess that it is `some request`, but there are other possibilities, such as merely overwhelming the system with garbage packets. We can only help to the limit of the information we have, which isn't much in this case. – Doug Smythies May 23 '21 at 14:44