Re: Re: Really struggling with windows install pls help :)
By: Digital Man to Matthew C E Bamber on Mon Jan 01 2024 03:48 pm

So I have an ip.can file that is about 1.3 gigs large, and when it gets this large there is a very long delay before a user can connect because I believe the bbs is working on parsing each line and seeing if the ip matches. I would assume I would have the same issue with host.can and other .can files.

I was wondering if there's anything I can do on my end to keep these large .can files and speed things up, or if synchronet needs some internal changes.

Thanks,
---
þ Synchronet þ ::: BBSES.info - free BBS services :::
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: large *.can files make things slow
By: MRO to Digital Man on Tue Jan 02 2024 07:37 am

So I have an ip.can file that is about 1.3 gigs large, and when it gets this large there is a very long delay before a user can connect because I believe the bbs is working on parsing each line and seeing if the ip matches. I would assume I would have the same issue with host.can and other .can files.

I was wondering if there's anything I can do on my end to keep these large .can files and speed things up, or if synchronet needs some internal changes.

Do you have some kind of automated system that dumps IPs in there?

It might be worth passing that task off to something that will block those via firewall, like fail2ban or csf/lfd (for Linux). I guess there's something similar for windows (wail2ban? It hasn't been updated in forever but there are forks on github and people claim it still works)

I use csf/lfd on my linux box, much like fail2ban it can watch log files and look for specific lines/words/etc. It also catches portscans and will block an IP if there are too many connections in a certain time period.

One of these days I should sit down and clean up my scripts for it, then create a wiki page about it.

DaiTengu

...Cross-country skiing is great if you live in a small country.
---
þ Synchronet þ War Ensemble BBS - The sport is war, total war - warensemble.com
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: large *.can files make things slow
By: DaiTengu to MRO on Tue Jan 02 2024 11:00 am

So I have an ip.can file that is about 1.3 gigs large, and when it gets

Do you have some kind of automated system that dumps IPs in there?

Synchronet itself seems to automatically put IPs in there when it detects clients trying to reconnect too often? I have a lot of entries in my ip.can with comments that start with "; SMTP - TOO MANY CONSECUTIVE FAILED LOGIN ATTEMPTS". Same with SSH.

Nightfox

---
þ Synchronet þ Digital Distortion: digitaldistortionbbs.com
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: large *.can files make things slow
By: DaiTengu to MRO on Tue Jan 02 2024 11:00 am

internal changes.

Do you have some kind of automated system that dumps IPs in there?

well i have ip.can and i have a capacha thing and sometimes i move those ips into ip.can

It might be worth passing that task off to something that will block those via firewall, like fail2ban or csf/lfd (for Linux). I guess there's something similar for windows (wail2ban? It hasn't been updated in forever but there are forks on github and people claim it still works)

yeah i was scripting stuff to be blocked in windows firewall
and also i use peerblock.

but lots of ips being blocked seems to slow everything down.
---
þ Synchronet þ ::: BBSES.info - free BBS services :::
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: large *.can files make things slow
By: MRO to Digital Man on Tue Jan 02 2024 07:37 am

Re: Re: Really struggling with windows install pls help :)
By: Digital Man to Matthew C E Bamber on Mon Jan 01 2024 03:48 pm

So I have an ip.can file that is about 1.3 gigs large, and when it gets this large there is a very long delay before a user can connect because I believe the bbs is working on parsing each line and seeing if the ip matches. I would assume I would have the same issue with host.can and other .can files.

I was wondering if there's anything I can do on my end to keep these large .can files and speed things up, or if synchronet needs some internal changes.

1.3 gigs is pretty massive. How'd that happen?

A couple of my servers were under SMTP botnet attack since early December (or earlier) and I started aggressively persistent filtering them (3 failed SMTP logins as a threshold, which is very low) - since then I have over 5000 IPs blocked in my ip.can file which is at the moment 730189 bytes in size - and that's with all the extra metadata (e.g. expiration date) that's now added with auto-filtered IP addresses. Anyway, you're saying that your ip.can file is over a thousand times larger. I wonder if you have a lot of duplicates or perhaps every IP of a subnet listed, when instead you could be using CIDR notation to make the file much smaller. Or just expiring/removing old entries would eliminate much of that file size. Making the expiration of old entries easier was one of my main motivations for the recent overhaul of how auto-added .can file entries are formatted.

All that said, the Synchronet servers *could* (they don't) cache the ip.can file contents in memory (making it much faster to search through) - but 1.3 gigs is a lot of RAM to dedicate to that purpose. And each server would have its own cache of the file contents, so you'd be using several gigabytes of memory for all of SBBS to have that massive ip.can file cached. It can be done and would likely have a significant performance improvement, but at the cost of a lot of memory used (in your case).
--
digital man (rob)

Rush quote #74:
A modern-day warrior mean, mean stride, today's Tom Sawyer, mean, mean pride Norco, CA WX: 56.2øF, 63.0% humidity, 0 mph E wind, 0.00 inches rain/24hrs
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: large *.can files make things slow
By: Nightfox to DaiTengu on Tue Jan 02 2024 09:37 am

Re: large *.can files make things slow
By: DaiTengu to MRO on Tue Jan 02 2024 11:00 am

So I have an ip.can file that is about 1.3 gigs large, and when it gets

Do you have some kind of automated system that dumps IPs in there?

Synchronet itself seems to automatically put IPs in there when it detects clients trying to reconnect too often? I have a lot of entries in my ip.can with comments that start with "; SMTP - TOO MANY CONSECUTIVE FAILED LOGIN ATTEMPTS". Same with SSH.

By default, Synchronet doesn't do that (stock sbbs.ini contains LoginAttemptFilterThreshold = 0). But yeah, it can do that, though the latest code using a different format for the metadata (not a comment, like you pasted here).
--
digital man (rob)

Breaking Bad quote #46:
If I ever get anal polyps, at least I know what to name them. - Saul Goodman Norco, CA WX: 56.2øF, 63.0% humidity, 0 mph E wind, 0.00 inches rain/24hrs
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: large *.can files make things slow
By: MRO to Digital Man on Tue Jan 02 2024 01:37 pm

memory for all of SBBS to have that massive ip.can file cached. It can be done and would likely have a significant performance improvement, but at the cost of a lot of memory used (in your case).

i guess i'll just trim it down every month.

If you have the available RAM, it wouldn't be a bad option to have. In fact, when importing QWK packets, the ip.can file *is* cached (since the source IP address of each message is compared, and that'd be really slow to re-read the file each time) - so there's already some cases where your SBBS instance is allocating that much RAM for the ip.can file, but for a shorter period of time since that memory is freed after QWK packet import is complete. For a server that's listening for incoming TCP/UDP connections, the memory (for the ip.can cache) wouldn't be freed until the server was terminated.
--
digital man (rob)

Synchronet "Real Fact" #35:
The irc.synchro.net network has more servers than users
Norco, CA WX: 60.9øF, 57.0% humidity, 0 mph NE wind, 0.00 inches rain/24hrs
--- SBBSecho 3.20-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

Re: large *.can files make things slow
By: Digital Man to MRO on Tue Jan 02 2024 01:01 pm

If you have the available RAM, it wouldn't be a bad option to have. In fact, when importing QWK packets, the ip.can file *is* cached (since the source IP address of each message is compared, and that'd be really slow to re-read the file each time) - so there's already some cases where your SBBS instance is allocating that much RAM for the ip.can file, but for a shorter period of time since that memory is freed after QWK packet import is complete. For a server that's listening for incoming TCP/UDP connections, the memory (for the ip.can cache) wouldn't be freed until the server was terminated.

well i'm running 32bit windows 10 for this specific bbs, so I can only use 4 gigs of ram. I'll just go back to blocking certain countries and some bad domains instead of blocking a million ip addresses.
---
þ Synchronet þ ::: BBSES.info - free BBS services :::
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

El 2/1/24 a las 10:37, MRO escribió:

Re: Re: Really struggling with windows install pls help :)
By: Digital Man to Matthew C E Bamber on Mon Jan 01 2024 03:48 pm

So I have an ip.can file that is about 1.3 gigs large, and when it gets this large there is a very long delay before a user can connect because I believe the bbs is working on parsing each line and seeing if the ip matches. I would assume I would have the same issue with host.can and other .can files.

I was wondering if there's anything I can do on my end to keep these large .can files and speed things up, or if synchronet needs some internal changes.

Thanks,
---
� Synchronet � ::: BBSES.info - free BBS services :::

convert you gigas of ip's to subnets:

https://stackoverflow.com/questions/50472999/python-convert-a-list-of-ips-into-a-list-of-networks
---
ï¿­ Synchronet ï¿­ Dock Sud BBS TLD 24 HS - bbs.docksud.com.ar
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)