Nasty lag friday evening.

[Amdoro]

Hi Adminless four total lag on server friday evening 2019-06-14 (for 1-2 min a x 4) server all players effected on server total lagged please inquire.

Mvh/ Amdoro

[...jutuli...]

Yep, it was happened more than once...

[adminless]

ok guys thanks for report. well initially I don't know of any issue with the server, I know about some issues with the low latency south west route (mostly just affecting the old domain/web services and the spare dns rather than the route itself) and that the reliability of the balkan route could be better. but well they are alternative routes (instead of main ones) and one of them was the former host for a reason. however I don't know of any issues with the main server/route itself.

I use to work on it frequently, almost daily (these last days I haven't), and so far I haven't found any problems. furthermore it just seems to be some transitional issue as now apparently is working just fine and even the times it failed it continued to work normally by itself in some time. so I don't know it seems to me like some sorting/balancing of the network at the server company external to the server itself.

then we'll probably have to wait and see, initially I wouldn't expect this to persist beside 1 or 2 days at most I guess, then let me know in case it does. I won't proceed with some full server check as that would lead the server down for hours probably which I believe that is a much worse situation than this current issue.

the south west route, I'll probably re-work it again soon and unless some mayor catastrophe, I'm not planning on moving the server any time soon as I can guarantee you 100% that's about the best can be done. probably by 2020/2021 as new deals come I may be looking forward a new server but until then that's it.

[adminless]

ok, apparently this same thing happened now again at around 21:30, hopefully this time I was online and I could made a quick diagnosis. it's not a network/host issue or similar as I had initially thought, it's more of a server issue. apparently what's going on is that the game server under heavy pressure at main times and specially when there's plenty of clients with broken connections with mayor packet loss it overflows until a point where it enters a deadlock. in simple words that some exceptional circumstances may exceed the usual "expected" design of the server so I just increased (doubled) now this expected server design resources and I'll keep monitoring this and I hope that this fixes this. I'm deeply sorry for this absolutely unacceptable issue.

[adminless]

additionally working around this now some mayor issues at some important international transit traffic has been observed which can largely take a important part on this issue. traffic transit is typically external to both the end user isp and the own server host, is what falls in between to complete the connection when there's no a "direct" connection between the two points. then this finally seems to be more of a combination of two issues and none of them involving the server itself, for one part we have mayor external internet issues that likely affected a large group of people and on the other the server itself missed to adequately handle such exceptional circumstances.

with the adjustments made I believe that the server should now handle properly this circumstance so it's obviously still expected that some people lag badly or at least a bit more than usual, there's nothing to do about that, but at least this lag should only affect them and not the whole room/server. in my opinion I believe that greedy telecom companies oversold the internet big time, I know for a fact that they do it guessing that most people won't even reach 20 mbits with their crappy wifi at best. that's a pretty stupid and unrealistic optimistic estimation, all that it takes is just 10 customers to reach a 10 gbit load, 10 gbit is the typical entry level peering capacity of modest companies, as I see it the internet (in its entirety) is a order of magnitude far from the so called giga internet and as I foresee it stuff like the 5g is gonna end by breaking the internet beyond believe so we'll see.

[Amdoro]

Again 2 times

[adminless]

this time I don't have recorded any issue with the server (it didn't overflow/deadlock) so initially it looks like it responded correctly. I guess that it's probably saturation at some other intermediate node as I mentioned. this time it only affected some or it affected the whole room like before?

[Amdoro]

Whole server was affected can it be a Ddos attack?

Mvh/ Amdoro

[adminless]

ok, thanks for info. I don't think so, ddos were actually the thing in the 2013-2014/2015 (when that "anonymous" group broke in the scene, I have the strong believe that they relied heavily on them) but since then, not that much anymore (notice how they virtually disappeared from the scene at least from the news). ddos nowadays are more a myth than anything else, since the 2016 approx it became pretty much a "standard" setup for virtually any company to implement ddos protections that work in the billions of packets per second order or even more rendering that kind of attacks pretty much meaningless. but anyways I can tell you for a fact that the server isn't getting ddos'ed.

like I explained, in my opinion, what I see instead is overly aggressive commercial strategies by greedy telecom companies that lead to serious internet overselling. I mean, I think we can see it everyday, right the past month facebook for example was going down (not just pinging bad for a couple of mins at most, no, going completely down for hours instead) for that very same reason on a almost weekly basis or this morning for example I was my self without internet for like a hour and once people start to switch from texting to holographic communication via 5g or other non sense stuff like that then problems are going to become very real, I foreseen. what is a reality nowadays (either intentionally or even unintentionally) is internet saturation and even if I remark once again that the server is never ever under absolutely any circumstance saturated if the internet path that is in the middle is then it will have an affect.

anyways, I'll keep monitoring this and I'll also probably contact the server company to see if there's more input about this.

[Amdoro]

2 days in raw now at 19.00 total lag for all players 1-2 min.

Mvh/ Amdoro

[adminless]

well ok if things continue to break I can consider making a server replacement on the future (middle term) although you can sure trust me that this can not be done much better. I contacted the company previous time and as expected they weren't aware of any issues on their side then by now I'll continue to monitor this. thank you for report.

[Amdoro]

Again 2-3 min long, all players 20.00 ca.

Mvh/ Amdoro

[adminless]

yes I saw now, this time the cpu was overkill, it could be a software bug, I'll keep investigating.

edit: I made a few small corrections now, we'll see if it helps improve the issue.

[adminless]

apparently this took place again today around 22:30 this time finally I could identify some intentional abnormal traffic which is good news as that should rule out the software bug. consequently I'm deploying some measures now and I'll still keep investigating more and working this out better in the long run. thanks for your patience and understatement.

[adminless]

btw guys don't think that I ever forgot about this issue for a single instant. last updates on this are that I spent the last couple of weeks cleaning up the servers network traffic and by now I can tell that there's no longer any records of abnormal traffic. that being said I'm also fully aware that despite that the server keeps occasionally lagging here and there which is absolutely unacceptable no matter what, so that now probably take us to the initial point, a bizarre software deadlock/bug either at the game server itself or even at the servers operative system. I know that this being open for more than a couple of minutes at most is plainly insulting, if something like this was going to take place in the summer of 2019 I should have completely foreseen such a situation beforehand no any after the fall of 2015 or at most the winter of 2016 so a full response should be provided in mere seconds for something as serious as this, anything bellow that is just not acceptable at all on my part.

unfortunately, understand that the reality is that this is quite a obscure issue and what's worst, it's not even reproducible, so it's virtually a worst case scenario to deal with but whatever the case is make completely sure that I took and I'm taking all the proper steps to diagnose and fix this as soon as possible. once ruled out almost for a fact the network issue on the next days then I'll review some server/system code and we'll see if it gets to something, I'll keep you informed, meanwhile thank you for your support and your patience and huge apologize to anyone that got his game impacted by this issue.