? Users online on discord!

Click to join our official discord server!

What's new

MineSaga.org

Click to copy the IP address!

Loading...

...

Development Blog - 10/09/2018

AgentTroll

Well-known member
#1
We’re always pushing the boundaries of what is possible on MineSaga servers. Being able to fit more players on a realm is a win-win for the player, and for us as well. We have been working on and continue to work on optimizing and allowing for more concurrent players on all our realms. Some players may be familiar with a blog post that I’ve parroted wherever people ask for more slots, so if you’ve read that already, consider this to be a sort of sequel to that.

What is "Performance?"
Performance used in this context almost always refers to how quickly some task is completed by the server. This “task” can be growing cactus blocks, deciding whether or not to spawn an iron golem, processing mob AI, processing data sent by players to the server, etc. To players, performance is best represented by how smoothly the server is running. For example, when the server has poor performance, players will experience rubber banding and laggy commands. To server owners and developers, server performance is best represented by statistics and metrics: numbers such as TPS (ticks per second), CPU utilization, available RAM, etc.

For the most part, TPS is the best representation of overall server performance because it correlates very well with what the player experiences, as well as what the server is experiencing as well; CPU and RAM usage are merely side effects of TPS. So what is TPS?

The Game Loop
In many games, programmers structure the game around an abstract concept called a “game loop.” This is the same structure used to run a Minecraft server like MineSaga. What the game loop does is it will run a single task infinitely, until the server is shut down. Each time this task runs, there is a series of tasks that it must perform, such as growing plants, updating player positions, etc. as I mentioned before. Each time the task runs, the server is said to have “ticked,” sort of like the ticking of a clock as the game progresses. The maximum number of ticks that can run in a single second is 20, so when the server is at 20 TPS, then it is running the most optimally.

Since the maximum number of ticks per second is 20, that means that each tick must complete within 50 milliseconds. As a matter of fact, if the tick completes early, the server will actually pause for the remaining time left of the 50 milliseconds and not do anything. If ticks only took 25 milliseconds, then it would be possible to fit 40 ticks in a second. This would make no sense for 2 reasons: firstly, the server would “fast-forward” when it is running optimally because blocks are being grown faster, and secondly, the speed at which the server time progresses would vary too much as the server becomes laggy and stops lagging.

On the other hand, it may actually take longer for the entire tick to run than the 50 milliseconds allowed. In this case, instead of skipping the current tick and going to the next one, the server will actually cut out the next tick(s) in order to complete the current tick. So if one tick took 100 milliseconds, the server would take out 1 tick in that 1 second interval so that the 100 millisecond tick can complete. Assuming the rest of the ticks did not go over, the server TPS would then become 19. This drop in TPS is the lag experienced by players. For example, if a player types in `/spawn`, their command might not be processed until the current tick completes. If the current tick is taking a long time, then it might take a long time to get to the next tick that will process the `/spawn`, and the player ends up having to wait.

Now it may be a curious thing to force players to wait for the server lag, but bear with me here. If the server abruptly halts a tick in the middle because it has overrun its 50 milliseconds, then perhaps 1/2 the players may not receive the updates they have been waiting for. If the tick is stopped before it processes, for example a player swinging his sword, it might be unfair for that player because the server will never get around to processing their hits. Therefore, all ticks run to completion, even if that means sacrificing a few ticks to make that happen.

What Causes Lag?
As was previously discussed, lag is caused by the tick overrunning the 50 milliseconds it is given to complete. The basic principle here is simple: the more that the server has to do in the tick, the longer it will take for the entire tick to complete. As long as we have extra time, we’re good.

There are tons and tons of things that slow down a tick, much too many to list here. However, here’s a small number of them:
  • Entities. ALL entities on the server, be they dropped items, iron golems, bosses, etc. must have their AI processed, their movement communicated to players around the entity, any damage that they do must be processed, interactions with pressure plates, etc.
  • Players. ALL players on the server constantly exchange commands, movements, sword swings, block breaks, opening inventories, etc. must be processed by the server and coordinated.
  • Blocks. Plant blocks like wheat, cactus, pumpkins, etc. must grow, their light level updated when the time of day changes, and the world must be saved if necessary. There are around 36000 blocks to update on Jurassic alone in a single tick.
  • Plugins. In order to process everything, such as minion mining, generators putting items into the inventory, hoppers collecting items, void chests selling items, time must be allocated for plugins to do their tasks as well.
Again, I cannot emphasize enough that this list does not do justice to the sheer amount of tasks that need to be completed in a fraction of a second.

If you have not already read the linked blog post from the introduction, gist of it is that player count correlates very strongly with lag. When players are not on the server, the server can actually ignore the entire player as well as what the player would have loaded into the world. Each player loads hundreds, if not thousands of blocks when they are online, as well as keeping entities loaded and players themselves also generate lag because they need to be sent world information such as the terrain and where all the blocks are located in the first place. By carefully adjusting our player counts, we can minimize server lag for everyone on the server, at the cost of long queuing times for those who are not. However, that is a sacrifice that we are willing to make to improve the player experience.

How is Lag Being Addressed?
As mentioned before, we are working on finding ways to squeeze more into each tick. The more we optimize, the more time is leftover after finishing all the tasks, and the more player slots we can afford to add without the additional impact that it brings. We are cutting down on Vanilla behavior and implementing better tuned controls over entities, such as through clear lag and entity stacking. We are optimizing the amount of chunks loaded in to remove large numbers of blocks that are being unnecessarily ticked.

We have also been going through our plugins and cutting out the excesses. We are looking at unnecessary database saves and file output to prevent the server tick from being polluted with laggy calls to our database and disk. We are looking through expensive hotspots such as minions to reduce the amount of unnecessary timer calculation for block breaking and inventory management code.

Last Words
Although there have been plenty of ways that we have been reducing lag, we’ve barely scratched the surface. We will continue to streamline the server and our plugins to meet server growth and to fix existing performance issues such as those that are currently plaguing the Space realm.

Performance is a very complex topic that even I am not qualified to talk about. It is a deep rabbit hole that presents challenges to the development team as a whole. In the end, I think with the talent that we have behind us, as well as with the support and understanding of our fantastic community, that we are well equipped to tackle the technical hurdles presented by server performance.
 

flowr

New member
#3
On the other hand, it may actually take longer for the entire tick to run than the 50 milliseconds allowed. In this case, instead of skipping the current tick and going to the next one, the server will actually cut out the next tick(s) in order to complete the current tick. So if one tick took 100 milliseconds, the server would take out 1 tick in that 1 second interval so that the 100 millisecond tick can complete. Assuming the rest of the ticks did not go over, the server TPS would then become 19. This drop in TPS is the lag experienced by players. For example, if a player types in `/spawn`, their command might not be processed until the current tick completes. If the current tick is taking a long time, then it might take a long time to get to the next tick that will process the `/spawn`, and the player ends up having to wait.
I was wondering if it would be possible to avoid this problem altogether by buffering frames that take longer than 100ms to complete. I am unsure what impact it would have, but it is worth trying in my opinion. If a tick takes too long to complete, instead of waiting for a whole 100ms push the actions which are being updated into a stack and skip that tick.

Pushing these actions into a stack would effectively reserve them, and any action that took place within that tick would be saved for the future tick. It seems as though it would not solve any problem, since it would just require the next tick to take more than 50ms to complete, and the process would repeat until the latency was normalised. This could result in even higher lag, however only for certain things.

Perhaps this could be used as a filtering technique, which would allow for a better user experience. Any essential tasks such as a player attacking a mob, or a player using a command could be a priority for the server, and any other task which might be intensive and require a lot of processing but isn't as essential, such as a block growing could be pushed on the stack.

The rate of growth would not be affected, assuming the bandwidth is normalised within the time-frame of a block growing. If a block grows in 60 sec, but the task is in the reserve for longer than 60 sec, then the growth rate would be affected. However if you run the necessary check, you could prioritise tasks with a timing-condition in a way which would force them to be pulled out of the stack within the right amount of time.

This sort of technique I believe might not necessarily increase the performance of the server itself, but it would give the players the illusion that it increased, since they would not experience the lag during gameplay with things which require immediate processing.
 
S

SoulessSavage

Guest
#4
super interesting
+1 would recommend
 
#6
One thing that should be taken into consideration when discussing implementing stacks in this scenario, is this could cause the server to have unexpected issues/results. Some of which may go overlooked for a long time, causing a potentially game-breaking outcomes. It would be essential to do enough testing to ensure that none of the tasks that the server is running, are linked to one another. If task-A is pushed to a stack, but another task; task-B depends on task-A to complete, when Task-B is being processed, it could potentially cause a lot of issues. Consider as an example “Mob Spawn Rate” and “Server Time”.

What would the result be if the spawn rate of Night Mobs was pushed to a stack during the night, and Server Time is considered an Essential? As the server time progressed into daytime from night, and the stack processed the Night Mob task. Would the mobs appear and suddenly burst into flames? Would it rain zombie flesh? Or Would the mobs be re-added to the stack? What would happen if the mob’s task was delayed several in-game days? Would we have an explosion of Night Mobs spawning? These are questions that need to be considered before implementing such a system.

Unless the developers know the infrastructure of the server well, there would have to be a lot of essential testing carried out before this could be implemented. Considering these factors, and if they could be resolved, perhaps any tasks with dependencies are pushed onto the stack as well? This could work really nicely, allowing more slots in the server.
 
Last edited:

AgentTroll

Well-known member
#7
If a block grows in 60 sec, but the task is in the reserve for longer than 60 sec, then the growth rate would be affected. However if you run the necessary check, you could prioritise tasks with a timing-condition in a way which would force them to be pulled out of the stack within the right amount of time.
While it's an interesting idea, it's literally not worth the effort to put this into effect. For one, coordinating between different tasks and giving certain tasks priority is an incredible hassle because you need to know what tasks to run in the first place; not only will this be difficult to design and implement, it will also consume vast amounts of memory and saturate the server with just the housekeeping alone (we're talking of millions and millions of tasks here). Secondly, the premise that it would only be buffered for 60 or so seconds is fantastical at best, because the server is already lagging anyways. The fact is that bufferring the task will cause lag spikes every 60 seconds because they *will* run out of time, and the other option is to buffer indefinitely, but then again we're dealing with a memory problem because the millions and millions of tasks I talked about earlier will need to be held for several hours until the night, when the server isn't lagging as badly.

The issue at hand is that anything that actually matters causes lag, and we cannot afford to buffer those. The other say 10% that we can afford to buffer that don't really matter won't really do anything, or will have too great of an impact on gameplay.

The best way to handle performance is to optimize anything that is really slowing the server down and hope that each tick fits more and reduce the server time slowdown.

That being said, I still think that it is an interesting idea, perhaps if the practical problems are solved then there may be a very real possibility of it being put to trial.
 
#8
A chunk sized cactus farm has 16,384 blocks that require tick based updates, I can't be sure, but this seems like it may be a reasonable portion of your block based updates(?).

Add a new tag to the chunk called 'unchanged' every time a change is made to the chunk, set thus value to false. Every time the chunk is processed set it to True. Before processing the chunk check if unchanged is true or false. If it's false process as normal. If it's true, just leave it and give the same output as last time.

Ok that won't work because cactus has a 'stage' property right? Bugger. In reality that stage shouldn't matter because over a chunk sized farm the random variation in growth rates is negligible. This *may* matter for other crops though.

Ok ok. Give us a new block called 'Cactus farm'. We place it in any chunk with a cactus farm and then those chunks are treated as above, maybe give us a 10% increase in drops from that chunk for helping the server reduce lag as well - otherwise there's no incentive for people to do it.
 

AgentTroll

Well-known member
#9
nd then those chunks are treated as above, maybe give us a 10% increase in drops from that chunk for helping the server reduce lag
Buffing certain items doesn't have a direct relationship with reducing the frequency of its use. It also assumes that it is possible to gain a 10% speed up (it won't, it's much lower than that). If the same performance standard is held, then that means that 10% more of the buffer item may be used as well. Nothing stops players from maximizing their use of the buffed item; there's simply no good reason to limit yourself if there's a better item. Eventually, the effect of adding them item diminishes as greater quantities of that item are being used, until that extra 10% gains goes away because it is possible to support that many more.
 
#10
Buffing certain items doesn't have a direct relationship with reducing the frequency of its use. It also assumes that it is possible to gain a 10% speed up (it won't, it's much lower than that). If the same performance standard is held, then that means that 10% more of the buffer item may be used as well. Nothing stops players from maximizing their use of the buffed item; there's simply no good reason to limit yourself if there's a better item. Eventually, the effect of adding them item diminishes as greater quantities of that item are being used, until that extra 10% gains goes away because it is possible to support that many more.
I'm not sure I understand what you're getting at here. The use of cactus farms is limited by the number of accounts on an island (each account can load 1 chunks worth). Let's say you make a block called 'Cactus chunk' which can be brought from the shop for $1. When this block is placed in a chunk that is used as a cactus farm it does the following:
1. Counts the number of sand blocks with cactus on top
2. Calculate the average growth rate (drops per second) of the chunk based on (1)
3. Deletes all cactus blocks from the chunk.

Then every tick thereafter this block spits out as many cactus item entities as it calculates in (2). This is 1 block update and 1 item entity creation cycle per tick rather than 16k possible block updates. Might be worth it.

To make it easier make it so that once this cactus chunk block is placed in a chunk the updates from this block are the only block updates the chunk processes.
 
#11
We’re always pushing the boundaries of what is possible on MineSaga servers. Being able to fit more players on a realm is a win-win for the player, and for us as well. We have been working on and continue to work on optimizing and allowing for more concurrent players on all our realms. Some players may be familiar with a blog post that I’ve parroted wherever people ask for more slots, so if you’ve read that already, consider this to be a sort of sequel to that.

What is "Performance?"
Performance used in this context almost always refers to how quickly some task is completed by the server. This “task” can be growing cactus blocks, deciding whether or not to spawn an iron golem, processing mob AI, processing data sent by players to the server, etc. To players, performance is best represented by how smoothly the server is running. For example, when the server has poor performance, players will experience rubber banding and laggy commands. To server owners and developers, server performance is best represented by statistics and metrics: numbers such as TPS (ticks per second), CPU utilization, available RAM, etc.

For the most part, TPS is the best representation of overall server performance because it correlates very well with what the player experiences, as well as what the server is experiencing as well; CPU and RAM usage are merely side effects of TPS. So what is TPS?

The Game Loop
In many games, programmers structure the game around an abstract concept called a “game loop.” This is the same structure used to run a Minecraft server like MineSaga. What the game loop does is it will run a single task infinitely, until the server is shut down. Each time this task runs, there is a series of tasks that it must perform, such as growing plants, updating player positions, etc. as I mentioned before. Each time the task runs, the server is said to have “ticked,” sort of like the ticking of a clock as the game progresses. The maximum number of ticks that can run in a single second is 20, so when the server is at 20 TPS, then it is running the most optimally.

Since the maximum number of ticks per second is 20, that means that each tick must complete within 50 milliseconds. As a matter of fact, if the tick completes early, the server will actually pause for the remaining time left of the 50 milliseconds and not do anything. If ticks only took 25 milliseconds, then it would be possible to fit 40 ticks in a second. This would make no sense for 2 reasons: firstly, the server would “fast-forward” when it is running optimally because blocks are being grown faster, and secondly, the speed at which the server time progresses would vary too much as the server becomes laggy and stops lagging.

On the other hand, it may actually take longer for the entire tick to run than the 50 milliseconds allowed. In this case, instead of skipping the current tick and going to the next one, the server will actually cut out the next tick(s) in order to complete the current tick. So if one tick took 100 milliseconds, the server would take out 1 tick in that 1 second interval so that the 100 millisecond tick can complete. Assuming the rest of the ticks did not go over, the server TPS would then become 19. This drop in TPS is the lag experienced by players. For example, if a player types in `/spawn`, their command might not be processed until the current tick completes. If the current tick is taking a long time, then it might take a long time to get to the next tick that will process the `/spawn`, and the player ends up having to wait.

Now it may be a curious thing to force players to wait for the server lag, but bear with me here. If the server abruptly halts a tick in the middle because it has overrun its 50 milliseconds, then perhaps 1/2 the players may not receive the updates they have been waiting for. If the tick is stopped before it processes, for example a player swinging his sword, it might be unfair for that player because the server will never get around to processing their hits. Therefore, all ticks run to completion, even if that means sacrificing a few ticks to make that happen.

What Causes Lag?
As was previously discussed, lag is caused by the tick overrunning the 50 milliseconds it is given to complete. The basic principle here is simple: the more that the server has to do in the tick, the longer it will take for the entire tick to complete. As long as we have extra time, we’re good.

There are tons and tons of things that slow down a tick, much too many to list here. However, here’s a small number of them:
  • Entities. ALL entities on the server, be they dropped items, iron golems, bosses, etc. must have their AI processed, their movement communicated to players around the entity, any damage that they do must be processed, interactions with pressure plates, etc.
  • Players. ALL players on the server constantly exchange commands, movements, sword swings, block breaks, opening inventories, etc. must be processed by the server and coordinated.
  • Blocks. Plant blocks like wheat, cactus, pumpkins, etc. must grow, their light level updated when the time of day changes, and the world must be saved if necessary. There are around 36000 blocks to update on Jurassic alone in a single tick.
  • Plugins. In order to process everything, such as minion mining, generators putting items into the inventory, hoppers collecting items, void chests selling items, time must be allocated for plugins to do their tasks as well.
Again, I cannot emphasize enough that this list does not do justice to the sheer amount of tasks that need to be completed in a fraction of a second.

If you have not already read the linked blog post from the introduction, gist of it is that player count correlates very strongly with lag. When players are not on the server, the server can actually ignore the entire player as well as what the player would have loaded into the world. Each player loads hundreds, if not thousands of blocks when they are online, as well as keeping entities loaded and players themselves also generate lag because they need to be sent world information such as the terrain and where all the blocks are located in the first place. By carefully adjusting our player counts, we can minimize server lag for everyone on the server, at the cost of long queuing times for those who are not. However, that is a sacrifice that we are willing to make to improve the player experience.

How is Lag Being Addressed?
As mentioned before, we are working on finding ways to squeeze more into each tick. The more we optimize, the more time is leftover after finishing all the tasks, and the more player slots we can afford to add without the additional impact that it brings. We are cutting down on Vanilla behavior and implementing better tuned controls over entities, such as through clear lag and entity stacking. We are optimizing the amount of chunks loaded in to remove large numbers of blocks that are being unnecessarily ticked.

We have also been going through our plugins and cutting out the excesses. We are looking at unnecessary database saves and file output to prevent the server tick from being polluted with laggy calls to our database and disk. We are looking through expensive hotspots such as minions to reduce the amount of unnecessary timer calculation for block breaking and inventory management code.

Last Words
Although there have been plenty of ways that we have been reducing lag, we’ve barely scratched the surface. We will continue to streamline the server and our plugins to meet server growth and to fix existing performance issues such as those that are currently plaguing the Space realm.

Performance is a very complex topic that even I am not qualified to talk about. It is a deep rabbit hole that presents challenges to the development team as a whole. In the end, I think with the talent that we have behind us, as well as with the support and understanding of our fantastic community, that we are well equipped to tackle the technical hurdles presented by server performance.
clearly put a lot of time into the post, I found it very well detailed and interesting. Thanks for making it.
 

MrBWH

Active member
#13
Very informative post and the MineSaga Team (Admins/Developers/Owner) are doing an amazing job at running the servers.

Some Ideas for dealing with lag...
  1. Combine multiple plugins into on plugin and streamlining the code. This would reduce the total number of plugins and possibly reduce the overall memory usage.
  2. Disable as many broadcast messages or only broadcast the message once. This esspecially for Global broadcast messages you are sending a broadcast to upwards of 3000 clients at once
  3. Disable Entities (Mobs) from spawning in the mining world! there is nothing worse then having a mob stand right in front of you while trying to mine...
  4. Throw more CPU's and RAM at it...
 
Top