? Users online on discord!

Click to join our official discord server!

What's new

MineSaga.org

Click to copy the IP address!

Loading...

...

Development Blog - 17/06/2018

System

Leadership Team
Staff member
Server Owner
Hi all,

This will be the first in a series of blog posts going into some of the technical struggles and information about MineSaga and how we run the network. It will cover anything and everything from bug fixes to infrastructure handling and detailed information about how we run our network and the support the thousands of players that we do every single day. This one will look into the past week with the recent issues with our hubs and proxies.

For ease of reading, several definitions:
- (BungeeCord, Proxy Server) A application that you connect to that allows us to route you to different minecraft servers
- (Subserver) A minecraft server instance such as a realm or hub
- (RedisBungee) A application that keeps the player counts in sync across the multiple proxy servers

Routing Players
We use a software called BungeeCord (proxy server) that is the primary point of contact from when you connect to our servers, it allows us to route you to different parts of our network such as a realm our one of our many hub servers. As MineSaga continued to grow, we didn't upgrade to a second proxy server until we few weeks ago which now means that our players are split evenly across two proxy servers.

However as we expanded to two proxies, we started running into odd issues. Both memory related, running out of RAM and the fact that they would randomly freeze, with our hub balancing software detecting our hubs as timing out and removing them (more below). This was very odd, considering we was running the same setup on some different machine as we had decided to upgrade the hardware.




An additional issue that we ran into was that our software that kept the playercounts in sync decided to randomly think a bunch of players had lagged out (every person on that proxy instance) which meant we was only displaying half the players online as we actually had while they were still connected to the proxy, and our subservers. This was very concerning and needed to be resolved instantly, which meant we had to restart the proxy and disconnect all of those one thousand players.



After some extensive testing and research we found the issue of the freezing to be that it was simply running out of connections to our communications server that allowed the two proxy instance to speak to one another so that our playercount could remain synchronised. We do apologise for the recent drops, however they should be fully resolved now.

Hub Balancing
A big issue that we continue to struggle with is one of balancing players across our hub servers. Those initial join points for players, and where several hundred players are funnelled whenever a realm reboots (of which every realm reboots at least once). We utilise software on every hub and one our proxy server to offer the best load balancing available however our machines are unable to accommodate the large amount of players and result in them dropping, and our hubs crashing.

We have attempted several methods to resolve this and have since spread our reboots out, delayed sending players to a hub by a few seconds per person to allow our hubs to correctly process them. The reboot times have moved forward quite a bit into BST time, this is because we are awake and this makes it easier for us to spot and resolve the issues.

We have also made several changes to the hubs to allow them to accommodate for you players betters and will be making some more changes in the several days, including removing players from realms from the one minute announcement mark to allow for everyone to be cleared out before the realm shuts down.

This was our machine before the changes whenever a realm with a few hundred players on it restarted.




Conclusion
Hope you understand the countless hours of work and research we are doing to fix the crashing and allow you to have a seamless playing experience. Additionally, we have outsourced to another company to assist us with the performance issues and hope to have them resolved within the coming days as we have been working with them for a while now.

I'm not sure how often I'll post those blogs, let me know what you think and how it can be improved upon such as additional information or what sort of blogs you'd like. I'll also see if any of our developers are up for writing their own ones about plugins that we run and how we overcome the issues.
 

Cptvamipergirl

Well-known member
This is really nice to see that a server is improving over time
 

48L

Member
Legend
Hi all,

This will be the first in a series of blog posts going into some of the technical struggles and information about MineSaga and how we run the network. It will cover anything and everything from bug fixes to infrastructure handling and detailed information about how we run our network and the support the thousands of players that we do every single day. This one will look into the past week with the recent issues with our hubs and proxies.

For ease of reading, several definitions:
- (BungeeCord, Proxy Server) A application that you connect to that allows us to route you to different minecraft servers
- (Subserver) A minecraft server instance such as a realm or hub
- (RedisBungee) A application that keeps the player counts in sync across the multiple proxy servers

Routing Players
We use a software called BungeeCord (proxy server) that is the primary point of contact from when you connect to our servers, it allows us to route you to different parts of our network such as a realm our one of our many hub servers. As MineSaga continued to grow, we didn't upgrade to a second proxy server until we few weeks ago which now means that our players are split evenly across two proxy servers.

However as we expanded to two proxies, we started running into odd issues. Both memory related, running out of RAM and the fact that they would randomly freeze, with our hub balancing software detecting our hubs as timing out and removing them (more below). This was very odd, considering we was running the same setup on some different machine as we had decided to upgrade the hardware.




An additional issue that we ran into was that our software that kept the playercounts in sync decided to randomly think a bunch of players had lagged out (every person on that proxy instance) which meant we was only displaying half the players online as we actually had while they were still connected to the proxy, and our subservers. This was very concerning and needed to be resolved instantly, which meant we had to restart the proxy and disconnect all of those one thousand players.



After some extensive testing and research we found the issue of the freezing to be that it was simply running out of connections to our communications server that allowed the two proxy instance to speak to one another so that our playercount could remain synchronised. We do apologise for the recent drops, however they should be fully resolved now.

Hub Balancing
A big issue that we continue to struggle with is one of balancing players across our hub servers. Those initial join points for players, and where several hundred players are funnelled whenever a realm reboots (of which every realm reboots at least once). We utilise software on every hub and one our proxy server to offer the best load balancing available however our machines are unable to accommodate the large amount of players and result in them dropping, and our hubs crashing.

We have attempted several methods to resolve this and have since spread our reboots out, delayed sending players to a hub by a few seconds per person to allow our hubs to correctly process them. The reboot times have moved forward quite a bit into BST time, this is because we are awake and this makes it easier for us to spot and resolve the issues.

We have also made several changes to the hubs to allow them to accommodate for you players betters and will be making some more changes in the several days, including removing players from realms from the one minute announcement mark to allow for everyone to be cleared out before the realm shuts down.

This was our machine before the changes whenever a realm with a few hundred players on it restarted.




Conclusion
Hope you understand the countless hours of work and research we are doing to fix the crashing and allow you to have a seamless playing experience. Additionally, we have outsourced to another company to assist us with the performance issues and hope to have them resolved within the coming days as we have been working with them for a while now.

I'm not sure how often I'll post those blogs, let me know what you think and how it can be improved upon such as additional information or what sort of blogs you'd like. I'll also see if any of our developers are up for writing their own ones about plugins that we run and how we overcome the issues.
Not gonna lie, you guys seem to know your stuff about how to manage these connection issues but I hope sooner or later this will become much less frequent.
 

System

Leadership Team
Staff member
Server Owner
Not gonna lie, you guys seem to know your stuff about how to manage these connection issues but I hope sooner or later this will become much less frequent.
Last night we deployed an update that should fix the proxy decided to untrack everyone as online, and several new hubs were setup today so they should be more or less resolved. We intend to redoing our hub setup in a more optimised way so they can handle more players along with a new and improved hub balancing setup.
 

Pax

Moderation Team
Staff member
Specialist
Triceratops
Earl
Mage
Venom
Cool post, informative as well!
 

Quig

Well-known member
Plus
Specialist
Earl
Outlaw
Velociraptor
Venom
Savage
Warlock

Pax

Moderation Team
Staff member
Specialist
Triceratops
Earl
Mage
Venom
You really are racking up those messages ain't ya!
Well I mean. Not that much :). I actually have a lot of time gaps in between my messages. Thanks for uh, tell me this though ❤
 

System

Leadership Team
Staff member
Server Owner
Stop necro posting
 
Last edited:
Top