3

Post-Solving Edit

The reason this was so hard to solve at the firewall level was that it wasn't a firewall problem. Something @tom-yan said in chat made me revisit the script which pulls out the IP Addresses. It seems that sed was pumping out the entire access_log file (rather than what I was telling it - as it did in testing). I replaced it with grep and the script is now exporting what I would expect.

Which makes this yet another example of check your assumptions.

Thank you to Tom Yan & @pillsbury-it-doughboy for your efforts. I appreciate it.

(Since Tom's is the only surviving answer, I'm going to mark his as the correct one.)

G'day,

I've got a small development server at my office with port 80 and 443 port forwarded from the modem.

To restate the title in the form of a question:

Why doesn't the firewall drop the incoming traffic from the IPv4 addresses that are listed in the rich rules.

Background: As you would expect, Bots & Baddies™ are looking for various things that a) don't exist, and b) would be bad if they they got into if they did exist. So I have a script that pulls IP addresses from Apache logs, which then end up in the firewall after they've been curated by me.

However, the firewall isn't dropping the connections. Addresses that are added on previous days are present in the new rules to be added and the logs again on subsequent days.

Details: The command that puts in the rich rules is this:

firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='165.227.87.0/24' reject"

However some rules have extra info:

rule family="ipv4" source address="162.159.244.38" log prefix="Shodan.io" reject

While some are simply:

rule family="ipv4" source address="42.224.165.0/24" reject

(those last two are rules copied from the output of firewall-cmd --list-all)

As shown below, the active zone is public, which isn't explicitly noted in the rule as I read that without it specified, it applies to the active zone.

While researching this, I realised that I might be in an unusual situation with the Modem port forwarding to the machine, rather than having it in a DMZ or hosted externally. The Apache logs show the internet facing IP addresses for the http/s client machines, and I've been assuming that these IPs are the address that presented to the firewall.

(netstat -tn does show a current connection to an external IPv4 address, but I can't establish if that's inbound or outbound.)

Warning: ALREADY_ENABLED: rule family='ipv4' source address='201.159.155.0/24' reject was a frequent response until I added logging notes into the rule, now it just happily accepts multiple reject rules for the same subnet.

Since the first time I wrote this question on serverfault.com (and was then told it wasn't ok to ask this there), I've had time to add the aforementioned logging, and have now been been to establish this:

[user@server ~]# firewall-cmd --list-rich-rules | grep 194.195.251.
rule family="ipv4" source address="194.195.251.0/24" log prefix="Sun May 23 21:23:50 UTC 2021" reject
rule family="ipv4" source address="194.195.251.0/24" log prefix="194.195.251.228 - Mon May 24 23:58:51 UTC 2021" reject
rule family="ipv4" source address="194.195.251.0/24" log prefix="194.195.251.228 - Tue May 25 21:25:27 UTC 2021" reject
rule family="ipv4" source address="194.195.251.0/24" log prefix="194.195.251.228 - Wed May 26 21:25:57 UTC 2021" reject

And this:

[user@server ~]# firewall-cmd --list-rich-rules | grep 192.241.196.
rule family="ipv4" source address="192.241.196.0/24" log prefix="192.241.196.220 - Mon May 24 23:59:25 UTC 2021" reject
rule family="ipv4" source address="192.241.196.0/24" log prefix="192.241.196.220 - Tue May 25 21:26:02 UTC 2021" reject
rule family="ipv4" source address="192.241.196.0/24" log prefix="192.241.196.220 - Wed May 26 21:26:31 UTC 2021" reject

So clearly, this isn't dropping traffic. (The date/times are the dates added to the firewall.)

In case it's helpful, here's the top part of firewall-cmd --list-all

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eno1
  sources:
  services: cockpit dhcpv6-client http https ssh
  ports: [redacted integer]/tcp [redacted integer]/tcp [redacted integer]-[redacted integer]/udp [redacted integer]-[redacted integer]/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

The rich rules then follow. And because why not, here's the active zone query:

[user@server ~]# firewall-cmd --get-active-zone
libvirt
  interfaces: virbr0
public
  interfaces: eno1

Other things to note:

  • There is only one Ethernet port on the box.
  • The firewall is reloaded after adding new rules (firewall-cmd --reload)
  • The server reboots every night
  • This server was CentOS Stream, but I converted it to CentOS 8
  • The firewall is active, enabled and functional (firewall-cmd --state returns 'running')
  • I'm aware that firewall-cmd --complete-reload exists, but it terminates all connections and that's not what I need.
  • Some things from the conf file:
    • FirewallBackend=nftables
    • IPv6_rpfilter=yes
    • RFC3964_IPv4=yes

Any ideas?

( List of Rich Rules: https://pastebin.com/qXFMBvqh )

-- EDIT -- Under the banner of "This is ridiculous", I put another local IP in the firewall (without --permanent) to test, then ran curl from the CLI. The pre-adding test worked fine, the post-adding test gave me curl: (7) Failed connect to **[redacted]**:80; Connection refused. With that behaving as expected, I cannot work out WHY these other things are not.

Have now turned on official logging (firewall-cmd --set-log-denied=all)

-- EDIT 2 -- More info on the scripts in question:

  • The first script parses the Apache logs and establishes a list of IP addresses. Upon that, it puts the firewall-cmd adding commands into a .txt file. (It also looks up Geo-location information, but that's out of scope here.)
  • The second script reads the .txt file, and executes the firewall-cmd add commands, then runs the firewall reload (--reload)

This allows the .txt file to build up over the weekend if I'm AFK. I inspect the GEO location information prior to running the second script to ensure that any IPs in the same country are more closely inspected to see whether they should be removed from the list.

-- EDIT 3 -- Logs shows this:

Jun 21 09:47:42 <SERVER_NAME> kernel: STATE_INVALID_DROP: IN=eno1 OUT= MAC=a8:a1:59:3a:03:14:20:b0:01:c0:a5:26:08:00 SRC=183.158.154.128 DST=<SERVER_INTERNAL_IP> LEN=40 TOS=0x00 PREC=0x00 TTL=44 ID=38952 DF PROTO=TCP SPT=54750 DPT=80 WINDOW=0 RES=0x00 RST URGP=0

There are four of these entries, all at the same millisecond. Interestingly, that IP address is not in the firewall rules at all.

Mark
  • 113
  • 11
  • 1
    You know about the typo in the invalid IP address you put in the `rule family="ipv4" source address="7162.159.244.38"` portion of your question, right? I need to play with Centos 8 more but do you have reject rules logging, look into that and look at the logs to see what is actually being rejected. It seems like you are listing the rules and not logs but I may not know what I'm looking at. I'm not seeing logs indicating these rules are not working in your question? Am I missing it, my apologies if so, just curious if I could help you with this question/issue. – Vomit IT - Chunky Mess Style Jun 17 '21 at 04:07
  • 1
    The rules in the firewall should work in order of precedent too, so see if you can put the deny rules before any allow rules with `firewall-cmd` as it could be as simple as that. An allow rule might be allowing something through first before the deny rule even applies, ensure the deny rule runs first precedent wise. – Vomit IT - Chunky Mess Style Jun 17 '21 at 04:10
  • I didn't see the odd IP address typo. Not sure when that was added into the question, but it doesn't exist in the firewall entries. (I've fixed it in the question now.) Re: Precedent, the `/etc/firewalld/zones/public.xml` shows `short`, `description`, `service` (x3), `port` (x4), then `rules` (from the rich rules). I'll look at the logs again in a minute. – Mark Jun 18 '21 at 03:54
  • Well, `/var/log/firewalld` shows nothing helpful but Warnings on Zone Drifting (which I've now disabled), older `WARNING: ALREADY_ENABLED` warnings. `journalctl -x -e` shows a LOT of `FINAL_REJECT`'s for eno1 from my PC's IP to x.x.x.255 (multicast?) on UDP. But it seems that the log only goes back till 1pm my time? OH. So, the server is new-ish, and I didn't have persistent storage of journal entries. :( – Mark Jun 18 '21 at 04:14
  • Have added **Edit 3**, with an entry from the journal logging. Doesn't appear helpful, but I thought I'd include it. – Mark Jun 21 '21 at 00:24
  • @Mark I still think it could be helpful if you can provide the nftables dump with `nft list ruleset`. (If you don't mind, you can even ditch all the added rules with firewall-cmd and just add back one non-working example -- that still does not work after being re-added; might make it easier for anyone that's willing to help). – Tom Yan Jun 21 '21 at 13:29
  • I've actually thinned out the duplicates, so this isn't as extensive as the previous pastes, but here it is: https://pastebin.com/0zbYVBP5 . Also, since I've done that, the system hasn't been running long enough to see the same rules being added as duplicates over more than the weekend. Kinda wish I'd left it messy now. :( – Mark Jun 22 '21 at 00:05
  • 1
    @Mark I wonder with it has something to do with the fact that, traffics of conntrack state established are accepted before they can be matched against the blocked source IP. It's common approach, but I think what you experiencing might be exactly one of the pitfalls: you need to make conntrack consider the connection to be dropped for the rules to be "in effect" (in the way as you desire/expect). – Tom Yan Jun 23 '21 at 05:26
  • 1
    @Mark So you might neither need to flush the conntrack table after the new rules are loaded (which might have some serious side effect on those that are not supposed to be blocked), or see if you can cherry-pick those connections that should be dropped. Or, do early rejection in nftables (it can be easily done if you are using nftables directly; no idea what's the right way to make firewalld have that done instead). – Tom Yan Jun 23 '21 at 05:28
  • 1
    @Mark it might be worth mentioning that, the "accept established traffics first" approach is sort of supposed to work with "only further whitelisting will be done later". IMHO any blacklisting rules should be *inserted* before it. Unfortunately, firewalld seems to be too "simple-minded", the it only precede the blacklisting rules before the whitelist rules in their own group. – Tom Yan Jun 23 '21 at 06:06

1 Answers1

1

See if the following workaround makes any difference. Run the following commands as root / with sudo (every time after reloading rules; shouldn't be necessary after adding new rules without --permanent afterwards though):

(Btw, make sure you bare in mind that rules added without --permanent will be gone after reloading, just like the following.)

nft add chain inet firewalld filter_IN_public_black
nft add rule inet firewalld filter_IN_public_black jump filter_IN_public_log
nft add rule inet firewalld filter_IN_public_black jump filter_IN_public_deny
nft insert rule inet firewalld filter_INPUT jump filter_IN_public_black

(Since typical table flush does NOT remove chains, but only rules, added to a table, you probably can't use create instead of add for the filter_IN_public_black chain to determine whether this needs to be applied again. So just check with e.g. nft list ruleset when in doubt. But of course, firewalld might enumerate all chains in its tables and delete them one by one upon reload. Do some tests if you want to know.)

P.S. This override will be applied to ALL interfaces / "zones" (with log and deny rules added, before or after the override is applied if without --permenant, for the public zone). If desired, further customize it to make it less "aggressive" (e.g. add an iifname match for the jump to filter_IN_public_black that is inserted to filter_INPUT).

Tom Yan
  • 9,075
  • 2
  • 17
  • 36
  • Hi, Thank you for following up with an answer before I'd had a chance to work through your three comments on the question. **Q:** I do have one concern before commencing. How do I revert back if the above fails / breaks something? I'm currently looking for a file that looks remotely like the output from `nft list ruleset`, but haven't found one yet. I'm not well versed in NFT, so I may require extra info/clarification in the answers using that directly. – Mark Jun 24 '21 at 01:18
  • Should I assume the above four rules: 1) create a 'pathway' (chain), 2) add two rules in order in that pathway - which tie into existing chains, & 3) tell NFT to put that pathway in the other pathway of filter_INPUT & then return to that spot post-processing? _(I looked up GOTO vs JUMP)_ – Mark Jun 24 '21 at 01:24
  • 1
    @Mark What `nft list ruleset` shows are volatile, i.e. rules that are currently loaded into the firewall engine. Since the commands manipulate the table firewalld creates, reloading rules with `firewalld` should get rid of them (except the then empty and unused `filter_IN_public_black` chain). The commands themselves do not create anything persistent. (Although there can be a system service that dump the ruleset to a file upon shutdown and reload from it upon boot, such service shouldn't be enabled if you use a frontend like firewalld.) – Tom Yan Jun 24 '21 at 02:06
  • 1
    @Mark what the commands do is essentially, create a "pathway" that leads to existing "pathways" (that consists of / in turns lead to you desired logging / rejecting rules), and make that pathway the first pathway the input traffics will go through. (Basically that means, match the traffics against those rules first, and apply what should be done if they are a match, of course.) – Tom Yan Jun 24 '21 at 02:12
  • 1
    @Mark what might help you understand better is that, there are terminating and non-terminating rules (I think those are the wordings, IIRC). Like rules with `accept`, `reject` and `drop` are the former, those keywords will determine that fate/verdict for the matched traffics, so no further matching or action will be applied on them; logging rules are on the other hand the latter, therefore you see the jump to the log chain is added before the one to the deny chain. – Tom Yan Jun 24 '21 at 02:31
  • 1
    @Mark The difference of jump and goto is not really your concern here, since neither firewalld or my commands take advantage of goto in their design. But knowing that "if there's no verdict after a jump, it will go back to where it jumped and continue" might help. – Tom Yan Jun 24 '21 at 02:36
  • Hi again Tom. The commands definitely applied. I can see the `jump` command in the `chain filter_INPUT`. (line 2, under `type filter hook input ....` (etc). I have a query though - why is it that the local IP address I tested with was rejected, but none of the external ones are? Seems like it should be the same for both? – Mark Jun 24 '21 at 04:04
  • Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/126815/discussion-between-tom-yan-and-mark). – Tom Yan Jun 24 '21 at 08:32